Setting 9000 MTU (Jumbo Frames) on Raspberry Pi OS

Raspberry Pi OS isn't really built to be a server OS; the main goals are stability and support for educational content. But that doesn't mean people like me don't use and abuse it to do just about anything.

In my case, I've been doing a lot of network testing lately—first with an Intel I340-T4 PCIe interface for 4.15 Gbps of networking, and more recently (yesterday, in fact!) with a Rosewill 2.5 GbE PCIe NIC.

And since the Pi's BCM2711 SoC is somewhat limited, it can't seem to pump through many Gbps of bandwidth without hitting IRQ limits, and queueing up packets.

In the case of the 2.5G NIC, I was seeing it max out around 1.92 Gpbs, and I just wouldn't accept that (at least not for a raw benchmark). Running atop, I noticed that during testing, the IRQ interrupts would max out at 99% on one CPU core—and it seems like it may be impossible to distribute interrupts across all four cores on the BCM2711.

Anyways, one quick way to get around that issue is to use 'jumbo frames'. By default, in Pi OS, the MTU ("Maximum Transmission Unit") is set to 1500, which is pretty standard on networking devices.

But for some situations, it's nice to increase it (especially if you need to ship larger volumes of data around a network, in fewer but larger packets).

Increasing MTU on my Mac

On my Mac, changing the MTU is as simple as going into a network interface's Advanced Hardware setting, and overriding in a dropdown menu:

Increase MTU to 9000 Jumbo Frame on macOS Network Interface System Preferences

Increasing MTU on a Pi

But on the Raspberry Pi, there are two ways to do it, depending on the network interface you're using:

  • For external interfaces, like the 2.5 GbE card I was testing, you can run sudo ip link set dev eth1 mtu 9000 (where eth1 is the external interface).
  • For the Pi's internal gigabit interface, you can't do that—instead, you have to patch the Pi OS kernel and recompile it.

The only way to increase the MTU on the internal gigabit interface on a Pi 4 model B, Compute Module 4, or Pi 400, is to recompile the kernel.

I have a handy guide for recompiling the kernel here: Raspberry Pi Cross Compile Environment VM.

The one thing you have to do in addition is, right after cloning the Pi kernel git repository, apply this patch to the kernel:

diff --git a/drivers/net/ethernet/broadcom/genet/bcmgenet.c b/drivers/net/ethernet/broadcom/genet/bcmgenet.c
index 62051e353278..81e3da888d1a 100644
--- a/drivers/net/ethernet/broadcom/genet/bcmgenet.c
+++ b/drivers/net/ethernet/broadcom/genet/bcmgenet.c
@@ -52,7 +52,7 @@
 #define GENET_Q16_TX_BD_CNT    \
    (TOTAL_DESC - priv->hw_params->tx_queues * priv->hw_params->tx_bds_per_q)

-#define RX_BUF_LENGTH      2048
+#define RX_BUF_LENGTH      10240
 #define SKB_ALIGNMENT      32

 /* Tx/Rx DMA register offset, skip 256 descriptors */
diff --git a/include/linux/if_vlan.h b/include/linux/if_vlan.h
index 41a518336673..28cac902cb77 100644
--- a/include/linux/if_vlan.h
+++ b/include/linux/if_vlan.h
@@ -22,8 +22,8 @@
 /*
  * According to 802.3ac, the packet can be 4 bytes longer. --Klika Jan
  */
-#define VLAN_ETH_DATA_LEN  1500    /* Max. octets in payload    */
-#define VLAN_ETH_FRAME_LEN 1518    /* Max. octets in frame sans FCS */
+#define VLAN_ETH_DATA_LEN  9000    /* Max. octets in payload    */
+#define VLAN_ETH_FRAME_LEN 9018    /* Max. octets in frame sans FCS */

 #define VLAN_MAX_DEPTH 8       /* Max. number of nested VLAN tags parsed */

diff --git a/include/uapi/linux/if_ether.h b/include/uapi/linux/if_ether.h
index d6de2b167448..78a12dd0e542 100644
--- a/include/uapi/linux/if_ether.h
+++ b/include/uapi/linux/if_ether.h
@@ -33,8 +33,8 @@
 #define ETH_TLEN   2       /* Octets in ethernet type field */
 #define ETH_HLEN   14      /* Total octets in header.   */
 #define ETH_ZLEN   60      /* Min. octets in frame sans FCS */
-#define ETH_DATA_LEN   1500        /* Max. octets in payload    */
-#define ETH_FRAME_LEN  1514        /* Max. octets in frame sans FCS */
+#define ETH_DATA_LEN   9000        /* Max. octets in payload    */
+#define ETH_FRAME_LEN  9014        /* Max. octets in frame sans FCS */
 #define ETH_FCS_LEN    4       /* Octets in the FCS         */

 #define ETH_MIN_MTU    68      /* Min IPv4 MTU per RFC791  */

Thanks to user waryishe on the Pi Forums for the suggestion.

Recompile the kernel with that patch, copy over the new kernel and modules to the Pi, and reboot, and you should have MTU 9000 available.

I don't really recommend you do this unless you (a) want to juice your benchmark numbers, or (b) know what you're doing and need to do this.

Comments

Hello Jeff
Descubrí tres cosas interesantes en el raspberry pi 4 que me gustaría que compartieras en tu canal.

Primero usando chacha20-poly1305 en el envío de datos usando ssh, scp se consigue transferencias de hasta 80MB/s, ya que al usar AES lo máximo que conseguí fue 38MB/s esto se debe a que el Rpi4 no tiene aceleración por Hardware AES.

Segundo lo mismo sucede al instalar un servidor nginx y activando el ssl. Usando el Cipher chacha20-poly1305 se consigue velocidades de hasta 100MB/s al hacer un download en el servidor que está en el Rpi4, en el caso de usar AES lo máximo fue 27MB/s.

Y tercero cuando en chromium browser cambio el user agent. Al de un equipo Mobil y entro a YouTube, ese cambia a la dirección https://m.youtube.com, al igual que el caso anterior el web browser usa chacha20-poly1305 como método de conexión ssl y la navegación de vídeos es más fluido.

Saludos desde Venezuela.

Interesting! It seems like optimizing the algorithm used for the data compression would make a huge impact with the Pi's not-super-fast CPU.

Hi Jeff,

Another way (in addition to increasing MTU) you might be able to reduce the work of the single thread receiving IRQs is by enabling Packet Steering and Flow Control and Steering.

Trying to include the details in this comment marks me as spam, I will keep trying in a separate comment.

The highest MTU i was able to achieve with the default 2048 RX_BUF_LENGTH (did not want to mess with this in case there were perf downsides) was 1964 when also using vlans (so 1964/1982 in if_vlan.h and 1964/1978 in if_ether.h). this allowed me to use baby jumbo properly, i did not want actual jumbo.