Forcing PCI Express Gen 3.0 speeds on the Pi 5

The Raspberry Pi 5 includes 5 active PCI Express lanes—4 go to the new RP1 chip for I/O like USB, Ethernet, MIPI Camera and Display, and GPIO, and 1 goes to a new external PCIe connector:

Raspberry Pi 5 PCIe connector

By default, all PCIe lanes operate at Gen 2.0 speeds, or about 5 GT/sec per lane. Currently there's no way to change that default for the RP1 chip's 'internal' lanes, but on the external connector, you can add the following lines inside /boot/firmware/config.txt (and reboot) to upgrade the connection to Gen 3.0 (8 GT/sec, almost double the speed):

dtparam=pciex1
dtparam=pciex1_gen=3

And yes, you can also downgrade the connection to Gen 1.0 speeds (2.5 GT/sec) if you like.

Why default to PCIe Gen 2.0?

Why is it defaulted to Gen 2.0? Because that's the speed at which the board could be certified for PCI Express. Even older standards like 2.0 and 3.0 are considered 'high speed' interconnects. And with any connection on a board, interference and signal issues can cause problems with higher bandwidth.

On expensive motherboards, PCIe Gen 5 and even Gen 4 have issues with some configurations, especially if people use risers for things like vertically-mounted GPUs.

Raspberry Pi 5 PCIe external FPC Connector - Flat cable

But even on the tiny Pi 5, things like the insertion loss from the flat FPC connection can cause issues at higher speeds. I encountered some link errors from time to time, and they were compounded on certain devices which don't handle them as gracefully.

One such device was the Google Coral TPU, which seemed to like resetting its connection to the Pi 5 constantly, to the point I can't get it working yet (under Gen 1, 2, or 3!). You can follow the saga here, but other devices do see a massive benefit with Gen 3.0 speeds:

It'd be really interesting to see if we can hack a Pi 5 board to expose all five lanes of PCIe from the BCM2712, and uprate all of them to Gen 3. You could conceivably run a 10 Gbps NAS or multi-port 2.5 Gbps router or firewall pretty easily with that bandwidth, sucking down 2-3W at idle.

Where to go for more

I've been tracking all of my experiences with PCI Express devices on Raspberry Pis for years now, and will continue to do so for the Pi 5, over on my Raspberry Pi PCIe Database.

Also see my initial post on Testing PCIe on the Raspberry Pi 5.

Comments

It'll be really interesting to see what a Compute Module 5 will look like. Maybe the RP1 will be optional or offloaded onto the dev board and all five lanes will be exposed.

This makes me excited for the Compute Module version, 5 lanes of PCI-E v3.0!

According to BCM2712's description (device-tree) the SoC has *six* PCIe Gen3 lanes in total. But you won't get more than 2 for general purpose usage on a potential CM5 since 4 of those available lanes are dedicated to be paired with RP1.

If a CM5 will not be accompanied by RP1 then this will be a rather limited thing, just having HDMI out and USB2 and all the 'typical' interfaces gone.

BCM2712 though does have some internal low-speed interfaces such as SPI, I2C and so on but I doubt Raspberry Ltd. to maintain different sets of device-tree definitions to keep different sets of available GPIO interfaces (at different addresses and so on) between RPi 5 and a potential CM5.

Also the only high-speed interface on current BroadCom VideoCore SoCs is PCIe as such be prepared to have four lanes always 'wasted' for RP1.

Love your work Jeff.

The Pi 5 really is taking things in a great direction but for me a thing I really struggle with is security. Specifically encrypting the card so if it is separated from the Pi there's little to fear with regard to local caches and copies of passwords, data, etc. What do you do to account for this?

Wouldn't encryption be handled at a partition level, like any other Linux computer? I am thinking among the likes of LUKS and dm-crypt to encrypt the root partition (not the boot partition).

> It'd be really interesting to see if we can hack a Pi 5 board to expose all five lanes of PCIe from the BCM2712, and uprate all of them to Gen 3.

Yes, this sounds very interesting! I wonder what you would have to give up that the lane was previously providing, though...?

Hello, I found your article to be really helpful with setting up NVME boot on Geekworm X1001 hat. Unfortunately I cannot get PCI-E 3.0 speeds despite adding both dtparm settings in /boot/config.txt. I am using high quality HP NVME drive so that should not be an issue.

Are you using the cable they included? If so it might not be impedance-matched (meaning some of the traces are different than others, to make the signaling better...), and that can definitely make speeds worse :(

I got the X1003 and with help of this site I have 800 MB/s however with or without the modification I notice these messages in dmesg.

pcieport 0000:00:00.0: AER: Corrected error received: 0000:00:00.0
pcieport 0000:00:00.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Transmitter ID)
pcieport 0000:00:00.0: device [14e4:2712] error status/mask=00001000/00002000
pcieport 0000:00:00.0: [12] Timeout

What did I miss ??

Those are fairly common if the link isn't 100% rock solid. I noticed those HATs come with the white non-impedance-controlled cables, so the cabling might lead to more link errors.

Hi Jeff,

I have exactly the same problem. Does it make sense to buy another cable to avoid these errors? If so, could you recommend one.

Thanks in advance

Hi, where could I buy that pcie data cable? I bought a pimoroni NVMe hat for the pi 5 but I need a longer cable, thanks!

Hi Jeff,
Great article, I have a pi 5 with Pimoroni nvme base with a crucial p3 ssd drive. I have set it up with Ubuntu, 23.10 I think, and it runs great but it is running PCIe Gen 2, is it possible to force gen 3 whilst running Ubuntu? And if so how is the best way of doing it?
Many thanks in advance
Andrew

Sorry Jeff, I have now sorted it out - found the Config.txt file and edited it adding the 2 lines in your article. (for some reason it was giving an error when i tried to edit through the command line). So now running great at Gen 3.
Kind regards,
Andrew

Hi Jeff,
Found the config.txt in Ubuntu 23.10 and forced gen 3 and speedtest for my Crucial P3 plus is 809.78 (read speed)
I use the Argon Neo 5 with NVME and is working fine.
thanx.

Awesome stuff as Always ! I would love to try this on my RPi 5 or even CM4. Is there an adapter yet for the RPi5 PCIe ribbon cable to PCIe X adapter ?

Thank you Jeff !