Testing 2.5 Gbps Ethernet on the Raspberry Pi CM4

Rosewill 2.5 Gbps Ethernet adapter PCIe 1x card

I got this Rosewill RC-20001 PCIe 2.5 Gbps Network Adapter working on the Raspberry Pi Compute Module 4.

Right after I got the card working, though, I tested it in an external powered PCI Express riser, and that test released the card's magic smoke. Oops.

Here's a dramatic re-enactment that's actually pretty accurate to what it looked like in real life:

PCIe card lets out magic smoke

Luckily, buying a replacment wasn't too bad, since the card is less than $20. But to get it to work on my spiffy new ten gigabit network, I also had to buy a new SFP+ transceiver that was compatible with 1, 2.5, 5, and 10 Gbps data rates, and that cost $60!

I'll get to 10 Gbps networking more in a future post, but on that topic, I didn't have any other computer with 1+ Gbps network interface, so I decided to upgrade my main MacBook Pro with a $150 OWC Thunderbolt 3 10G adapter, too.

With the hardware ready—and my wallet was a bit lighter—it's time to see how the card does!

Video for this post

I also have a companion video for this blog post, for the more visually-inclined:

Pardon the use of Comic Sans...

Only 2.5 Gbps? What about your 5 Gbps post?

"But wait a second," I hear you say, "didn't you already get 4.15 gigabits through the Intel i340 card last month?"

Well, yes. But that was in aggregate, through five separate 1 Gbps interfaces.

And while you can bond interfaces sometimes, life is simpler with a big fat pipe. And 2.5 Gbps, as I'll demonstrate later, is probably about as much as the current BCM2711 Raspberry Pi processor can handle.

So how does it work?

I plugged the card directly into the 1x slot on the IO Board and ran lspci, and the card showed up. dmesg logs showed the card didn't have any trouble allocating BAR space either, so from a hardware perspective, we're good to go!

But ip address didn't show the driver, so it was time to find a driver.

I tried the driver from Realtek's website, but I had trouble compiling it, so it was time to recompile the Linux kernel!

Since I've been cross-compiling a lot this past month, I took some time to optimize the process, and got it from around 30 minutes to 11 on my Core i9 laptop.

In menuconfig, I enabled the driver for the Realtek 8125 chip, which is inside

Device Drivers
  -> Network device support
    -> Ethernet driver support
      -> Realtek devices
        -> Realtek 8169/8168/8101/8125 ethernet support

I cross-compiled the kernel, copied it over to the Pi and rebooted, and found some encouraging new dmesg logs, so I ran ip address and found the brand new eth1 interface!

eth1 interface appears when running ip address on Linux

I plugged in a network cable, and got a good connection after a few seconds.

Getting Full Performance

I ran a speed test to my Mac over my 1 Gbps network with iperf3, and it ran at 940 megabits—the same as the internal network chip on the Pi.

MikroTik 4-port SFP+ 10g switch with Raspberry Pi Compute Module 4 at 2.5 Gbps

Next I pulled out my MikroTik 10G switch, plugged in two copper transceivers, and plugged the Pi into one port, and my Mac's new 10G adapter into the other port.

I ran iperf3, and my first tests showed just under two gigabits:

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.10 GBytes   943 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  1.10 GBytes   941 Mbits/sec                  receiver

That's not bad... but under 2 gigabits just feels like it's missing something. I learned from running atop that the Pi's little CPU just can't keep up with multiple gigabits of network traffic:

atop showing 99% IRQ utilization on Raspberry Pi CM4 CPU core

To ease the load on the processor, I set the MTU to 9000 with the command sudo ip link set dev eth1 mtu 9000.

To make a difference though, you have to make sure both ends support MTU 9000, or 'jumbo frames'. So I also edited my Mac's network settings.

Just as an aside, if you want to enable jumbo frames on the Pi's internal Ethernet jack, you can't use ip link set, you actually have to recompile the kernel with a patch. If you want to do that, I wrote up that process (and how to change a Mac's settings) in this blog post: Setting 9000 MTU (Jumbo Frames) on Raspberry Pi OS.

With the MTU set to 9000 on both ends, I ran the iperf3 test again and got:

0 bits/sec on MTU 9000 on Raspberry Pi iperf3 test

ZERO BITS PER SECOND?!

Long story short, the MikroTik switch is configured as a router out of the box, with ports limited to MTU 1500. I could've set the ports higher in the router, but instead I switched over to the MicroTik Switch OS ('SwOS').

So another networking lesson learned: if you want to use jumbo frames, you have to consider every network device in the path between your computers!

So now that Jumbo Frames were actually a thing on my little baby 10G network, I ran iperf3 again, and this time, I found a much nicer result:

2.48 Gbps on Rosewill 2.5 G NIC on Raspberry Pi CM4 in iperf3

2.48 Gbps is much better.

Overclocking the CPU to avoid clogging IRQs

But if atop showed regular-sized Ethernet frames are bottlenecked by the CPU's IRQ interrupts, wouldn't overclocking the Pi help too?

Well yes, overclocking does actually increase network throughput!

I set the MTU back to 1500, and enabled a 2.147 GHz overclock by editing /boot/config.txt and rebooting the Pi.

After the reboot, I ran the test again, and this time, I was seeing speeds around 2.3 gigabits—a 20% network speedup.

Raspberry Pi 2.3 Gbps data rate with Compute Module 4 CPU overclock

But if you just need raw throughput for large files and your network supports it, Jumbo Frames are the only way to fully saturate the network interface:

Jumbo frame performance compared to overclock and standard 2.5 Gbps performance

Even if you don't change anything, getting 2.3 Gbps with an overclock isn't bad! That's almost 300 MB/sec—a massive improvement over the built-in network interface.

Testing bidirectional throughput

The last test I ran was bidirectional throughput. I had to compile iperf3 from source to run this test since the version in the Pi OS software repository is too old. You need iperf3 3.7 or later.

To compile from source:

# Download iperf3 source.
wget https://github.com/esnet/iperf/archive/master.zip

# Unzip the download and enter source directory.
unzip master.zip && cd iperf-master/

# Run configure and make.
./configure
make

# Use the compiled version.
cd src/
./iperf3 --help

Since both the Pi and my Mac have Full Duplex network interfaces, traffic should be able to flow in both directions simultaneously.

Unfortunately, it seems bandwidth to and from the Pi isn't quite symmetrical. The receive traffic was still showing about 2.5 Gbps, but simultaneous transmit was limited to 100 Mbps.

Asymetric transfer rates aren't a big issue for most use cases, though, so I'm not too worried.

I also ran UDP tests, and the Pi could pump through 2.5 Gbps without issue. Bidirectional, though, a lot of packets were dropped if operating at full tilt, because the Pi just couldn't keep up with the firehose blasted at it from the 10G interface on my Mac.

Conclusion

You might be wondering at this point, "Why is Jeff testing all these different network cards?"

Don't worry, you're not crazy; my wife keeps asking me the same question.

If you want to keep these experiments going, please subscribe to my YouTube channel and support my work on Patreon or GitHub!

Check out other cards I'm testing on the CM4 on my Raspberry Pi PCI Express device compatibility database.

Comments

Jeff,

Not familiar with this realtek chip, but look at the ethtool tuning of tcp offload, interrupt coalescing. and ring buffers.

Also, if you're having to buy transcievers, 10G SR SFP+ and multimode fibres are cheap as chips. Don't bother with copper unless you've already got it.

Yeah; just plain old twinax DACs are cheaper for short distances. For my Mac's TB3 adapter, it's copper-only (it seems like there aren't many ThunderBolt adapters with SFP+, unfortunately), but I'll be testing a Mellanox ConnectX-2 10 Gbps SFP+ card in the Pi soon!

Watch out with any connectx 3 or earlier cards- support for them is dropped in the default compile of the mlx4 module in recent kernels.
You'll waste lot of time wondering why the card isn't being recognised.
Best is to grab the mellanox OFED long-term support stack and put that on. There's a Debian aarach64 dist, I think.

There also are USB 3.0 devices based on Realtek RTL8156 chipset that support 2.5Gbps Ethernet.
What throughput would be possible on those devices when used with a normal Raspberry Pi 4 or 400?

I was curious about this as well, and grabbed the Sabrent 2.5GbE adapter that uses this chipset. It's currently working the Raspberry Pi 4b for me, though not stable yet. I'm working with some experimental drivers that one of the devs that works on my distro compiled, and there are some Issues.

A big one is bus power. I've got a USB 3 enclosure for an NVME drive that I'm booting off of, and having that plugged into the USB3 ports, along with the adapter, causes serious instability. It doesn't seem to be able to continuously power both the drive and the adapter. I've got the NVME plugged into the USB 2 controller right now, and it's ... mostly stable, though it's still not rock solid the way it was before. I've ordered an AC-powered USB 3 hub to move the drive to.

Aside from that, the USB 3 bus on the Pi can do, I believe, 4.1Gbps, given the limitations of the processor. I'm not sure if this is aggregate or per-port, but in either case, it seems unlikely we'll ever get near 2.5Gbps on the Pi 4 so long as there's anything else on the USB 3 bus.

My NVME drive, for instance, operates at 300 MBps read, which is ... ~2.5Gbps. Jeff's testing in another article showed similar results, so I'm guessing each port can only do about 2.5Gbps on its own, but I have no way of being sure about that.

Right now, with 1500 MTU, a 2.0 GHz overclock, and all default settings on the Pi; the iPerf3 server running off my 10G Mac mini, and flow control DISABLED on my switch, I'm seeing:

~]$ iperf3 -c $IT.ME
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 219 MBytes 1.84 Gbits/sec 0 660 KBytes
[ 5] 1.00-2.00 sec 216 MBytes 1.81 Gbits/sec 0 660 KBytes
[ 5] 2.00-3.00 sec 224 MBytes 1.88 Gbits/sec 0 696 KBytes
[ 5] 3.00-4.00 sec 216 MBytes 1.82 Gbits/sec 0 795 KBytes
[ 5] 4.00-5.00 sec 218 MBytes 1.82 Gbits/sec 0 795 KBytes
[ 5] 5.00-6.00 sec 215 MBytes 1.80 Gbits/sec 0 795 KBytes
[ 5] 6.00-7.00 sec 218 MBytes 1.82 Gbits/sec 0 795 KBytes
[ 5] 7.00-8.00 sec 222 MBytes 1.87 Gbits/sec 0 891 KBytes
[ 5] 8.00-9.00 sec 216 MBytes 1.81 Gbits/sec 0 939 KBytes
[ 5] 9.00-10.00 sec 218 MBytes 1.82 Gbits/sec 0 996 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.13 GBytes 1.83 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 2.13 GBytes 1.83 Gbits/sec receiver

Given that I was getting about 867 Mbps average on the builtin 1Gbps ethernet, I can't complain at all about these results. Something like a 2.07x increase.

Jeff now that you have this have you tested this as a full blown router with iperf3 tests? If so can I expect to see the results here or on your YT? Would this make a half decent budget firewall or router?

I will be doing some testing with a couple more dedicated Raspberry Pi CM4 router boards, one which I just received this week, the other... soon!

Hi Jeff,

Dunno if you are aware of that, but Mikrotik advises against having two RJ45 Transceivers side by side on passive-cooled switches. These things get hooooot! :D

90-95°C when side by side :)

I'm considering hacking a fan somewhere on the case to suck some more air across those transceivers.

I ordered a fan last week, though, truth be told, it has been running for more than 6 months without any issue.
But Jeff, one thing didn't get much clear to me: when you enabled jumbo frames, what was the CPU usage?
I would like to build a Kubernetes cluster with this setup, and I would like to have a (faster) network storage, but it won't make sense if the ethernet card steals all the cpu... :D

Do you mean, it was only high under full load, or it went down only when iperf3 was running? I know the question is weird, I just want to be sure I got it.
Thanks for the great video, I will see if I dare to go 2.5gbe for this cluster, or if I just settle with 1gbs.

It was only high (and not in terms of absolute numbers, just interrupts) while iperf3 was running on both interfaces. It was fine otherwise.

Awesome! Then I will try to build my cluster this way. With a bit of luck, this might make the whole thing future proof when newer CM are released… :)

Jeff, if you have the time, could you tell me if you think that PoE would be enough to power that carrier board you used with this nic you tested? I am considering to buy this (a Waveshare carrier board with PoE:
ompute-module-4-poe-board-b
But now it occurred to me that PoE might not be enough to power the PCI device...

Hi Jeff,

So, I ran across this article while setting up my RPI4b 4GB and I can indeed testify it works. Really grateful to the author, TomCore, on that one. Would you be willing to retest with the modifications. I think this would fix the bidirectional issue.

https://tomcore.io/docs/articles/RaspberryPI/raspberry-pi-4b-network-pe…

My final setup will be a pi that hopefully gets line speed, but we will see. For now it runs as unifi server, plex, resilio sync. I would like to get amazon deep glacier working but that is for later.

Thank you I really enjoy your videos.