Since the Raspberry Pi Compute Module 4 was introduced last year, I've been testing a variety of PCI Express NICs with it. One of the main types of NIC I'm interested in is cheap 2.5 Gigabit Ethernet adapters.
2.5 Gigabits is about the highest reasonable bandwidth you can get through the PCI Express Gen 2.0 x1 lane on the Raspberry Pi, and it's also a lot more accessible than 10 Gigabit networking, especially for home users who might already have Cat5e runs that they are loathe to swap out for Cat6 or better cabling.
In my testing, besides discovering that not all 10 Gbps SFP+ transceivers are created equal, I found out that when it comes to performance, the Linux driver you're using matters—a lot.
And as a stark illustration of this point, I recently pitted an ASUSTOR Drivestor 4 Pro NAS against a Raspberry Pi 'Taco' NAS. Both of these devices use the popular Realtek RTL8125B chip, and both of them were reporting underwhelming iperf3 bandwidth results:
The maximum throughput was eerily similar, just under 1.9 Gbps. And worse, I noticed when using devices through a PCI Express switch on the Pi, like a SATA RAID array, the network throughput took a hit, down to 1.51 Gbps on the Pi.
I wondered if the driver might've seen some improvements in later revisions of the Linux kernel, so I recompiled the kernel using the 5.15.x source:
That made no difference—in fact, the results started to have more jitter in them, so I decided that avenue wasn't worth pursuing further.
So next, I downloaded the driver directly from Realtek's website, and installed version 9.007.01:
$ sudo apt-get install -y raspberrypi-kernel-headers
$ tar vjxf r8125-9.007.01.tar.bz2
$ cd r8125-9.007.01/
$ sudo ./autorun.sh
I re-ran my benchmarks, and wouldn't you know? I'm able to get the full 2.35 Gbps of real-world throughput through the card—regardless of other traffic on the PCIe bus!
So the moral of the story seems to be: if you're not seeing the performance you expect, see if the vendor's driver is better than what's in the kernel tree.
Many vendors take an upstream-first approach, where they make sure any driver optimizations make their way to the kernel quickly... but not all. I'm not sure how the discrepancy is so big, though, in this case. It looks like 8125B support was only added a year and a half ago, and only a few chip-specific commits have been added since.
I have to wonder if maybe since far fewer Linux users have 2.5G-capable networks, these kinds of performance issues aren't discussed as much?
I confirmed with ASUSTOR that their current Drivestor 4 Pro software is running the driver from the kernel, and not the Realtek vendor driver. No word yet on if that could change—but if so, it may be able to unlock more performance from their value-lineup of 2.5G NASes!
For all my debugging details, check out my GitHub issue NAS Comparison - ASUSTOR Drivestor 4 Pro vs Pi CM4.
Comments
Interesting! The RTL 8125B chip seems to be quite common on PC mainboards as well - I just noticed that I'm using the same chip in my recently build SFF PC (Asus ROG Strix B560I). The firmware is really quite old: rtl8125b-2_0.0.2 07/13/20 (kernel 5.15.8)
However, all my other devices have 1GbE or less, so it doesn't matter anyway (at least for the forseeable future, I'm confident that the driver will be updated eventually).
I got an AMD 5 mini computer with dual 2g and 5G WiFi. The 5G connection came up and choosing it is much faster. Same cable modem from cable company install 2 yes ago...is it possible it has 5G on it or does 2G run faster to a 5G receiver?
Wrong place to ask, but yes, the cable modem may have 5G wifi. Just make sure that's actually your modem you're connecting to. And no, 2G won't be any faster, it'll still use 2G.
I'm using a pcie NIC with that r8125 controller on it and after installing the driver you mentioned I get 2.5Gbps speeds but only in ONE direction! Only when sending FROM that Ubuntu machine to another machine. It's driving me crazy but whenever I try to send from another machine to that Ubuntu machine it's capped at 1.0Gbps speeds. Any ideas what else I might try? I've posted a question on askubuntu and sent a mail to Reaktek's support but no luck so far.
I'm having the same issue, I opened this bug report today: https://bugzilla.kernel.org/show_bug.cgi?id=217362
The only difference is I'm not capped at 1Gb/s, I get closer to 1.5Gb/s of throughput.
f
Very ironic that the Realtek 2.5Gb chipset has been supported (not well, typically) for a while but the Intel I225 is *barely* making it into the kernel. IIRC kernel 5.15 is the cutoff. What I've read is that v3 of the Intel chip broke the older driver, argghh.
Just about to buy a new Intel m/b so I'm interested ....
Comment comes quite late, but:
On my test system I can get the full 2.5Gbps also with r8169 driver. Your system may not be able to cope with the interrupt load.
Try the following (software interrupt coalescing) to reduce the interrupt load and see whether performance improves.
echo 20000 > /sys/class/net//gro_flush_timeout
echo 1 > /sys/class/net//napi_defer_hard_irqs
I noticed something interesting about the drivers on Windows vs Linux. On Windows max MTU selectable is 16128 bytes, on Linux it seems to be the second highest listed in Windows, 9014 bytes.
Realtek doesn't provide chip datasheets. Therefore the only information source is the r8125 vendor driver, and it's limited to 9K too.