The Raspberry Pi 5 includes 5 active PCI Express lanes—4 go to the new RP1 chip for I/O like USB, Ethernet, MIPI Camera and Display, and GPIO, and 1 goes to a new external PCIe connector:
By default, all PCIe lanes operate at Gen 2.0 speeds, or about 5 GT/sec per lane. Currently there's no way to change that default for the RP1 chip's 'internal' lanes, but on the external connector, you can add the following lines inside /boot/firmware/config.txt (and reboot) to upgrade the connection to Gen 3.0 (8 GT/sec, almost double the speed):
dtparam=pciex1
dtparam=pciex1_gen=3
And yes, you can also downgrade the connection to Gen 1.0 speeds (2.5 GT/sec) if you like.
Why default to PCIe Gen 2.0?
Why is it defaulted to Gen 2.0? Because that's the speed at which the board could be certified for PCI Express. Even older standards like 2.0 and 3.0 are considered 'high speed' interconnects. And with any connection on a board, interference and signal issues can cause problems with higher bandwidth.
On expensive motherboards, PCIe Gen 5 and even Gen 4 have issues with some configurations, especially if people use risers for things like vertically-mounted GPUs.
But even on the tiny Pi 5, things like the insertion loss from the flat FPC connection can cause issues at higher speeds. I encountered some link errors from time to time, and they were compounded on certain devices which don't handle them as gracefully.
One such device was the Google Coral TPU, which seemed to like resetting its connection to the Pi 5 constantly, to the point I can't get it working yet (under Gen 1, 2, or 3!). You can follow the saga here, but other devices do see a massive benefit with Gen 3.0 speeds:
- My 10G ASUS Network Card now gets 6 Gbps (instead of about 3.5) with the Pi 5.
- My Kioxa XG8 NVMe SSD now gets 900 MB/sec reads versus 450 MB/sec under Gen 2.0.
It'd be really interesting to see if we can hack a Pi 5 board to expose all five lanes of PCIe from the BCM2712, and uprate all of them to Gen 3. You could conceivably run a 10 Gbps NAS or multi-port 2.5 Gbps router or firewall pretty easily with that bandwidth, sucking down 2-3W at idle.
Where to go for more
I've been tracking all of my experiences with PCI Express devices on Raspberry Pis for years now, and will continue to do so for the Pi 5, over on my Raspberry Pi PCIe Database.
Also see my initial post on Testing PCIe on the Raspberry Pi 5.
Comments
It'll be really interesting to see what a Compute Module 5 will look like. Maybe the RP1 will be optional or offloaded onto the dev board and all five lanes will be exposed.
This makes me excited for the Compute Module version, 5 lanes of PCI-E v3.0!
According to BCM2712's description (device-tree) the SoC has *six* PCIe Gen3 lanes in total. But you won't get more than 2 for general purpose usage on a potential CM5 since 4 of those available lanes are dedicated to be paired with RP1.
If a CM5 will not be accompanied by RP1 then this will be a rather limited thing, just having HDMI out and USB2 and all the 'typical' interfaces gone.
BCM2712 though does have some internal low-speed interfaces such as SPI, I2C and so on but I doubt Raspberry Ltd. to maintain different sets of device-tree definitions to keep different sets of available GPIO interfaces (at different addresses and so on) between RPi 5 and a potential CM5.
Also the only high-speed interface on current BroadCom VideoCore SoCs is PCIe as such be prepared to have four lanes always 'wasted' for RP1.
Love your work Jeff.
The Pi 5 really is taking things in a great direction but for me a thing I really struggle with is security. Specifically encrypting the card so if it is separated from the Pi there's little to fear with regard to local caches and copies of passwords, data, etc. What do you do to account for this?
Wouldn't encryption be handled at a partition level, like any other Linux computer? I am thinking among the likes of LUKS and dm-crypt to encrypt the root partition (not the boot partition).
> It'd be really interesting to see if we can hack a Pi 5 board to expose all five lanes of PCIe from the BCM2712, and uprate all of them to Gen 3.
Yes, this sounds very interesting! I wonder what you would have to give up that the lane was previously providing, though...?
Hello, I found your article to be really helpful with setting up NVME boot on Geekworm X1001 hat. Unfortunately I cannot get PCI-E 3.0 speeds despite adding both dtparm settings in /boot/config.txt. I am using high quality HP NVME drive so that should not be an issue.
Are you using the cable they included? If so it might not be impedance-matched (meaning some of the traces are different than others, to make the signaling better...), and that can definitely make speeds worse :(
I got the X1003 and with help of this site I have 800 MB/s however with or without the modification I notice these messages in dmesg.
pcieport 0000:00:00.0: AER: Corrected error received: 0000:00:00.0
pcieport 0000:00:00.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Transmitter ID)
pcieport 0000:00:00.0: device [14e4:2712] error status/mask=00001000/00002000
pcieport 0000:00:00.0: [12] Timeout
What did I miss ??
Those are fairly common if the link isn't 100% rock solid. I noticed those HATs come with the white non-impedance-controlled cables, so the cabling might lead to more link errors.
Hi Jeff,
I have exactly the same problem. Does it make sense to buy another cable to avoid these errors? If so, could you recommend one.
Thanks in advance
I also would like to know exactly what cable is needed; this does amaze me that you can even OC a pcie for the pi 5.
would love to know in the future of the CM5 would natively support pcie 3.0 for multiple lanes!
Hi, where could I buy that pcie data cable? I bought a pimoroni NVMe hat for the pi 5 but I need a longer cable, thanks!
Hi Jeff,
Great article, I have a pi 5 with Pimoroni nvme base with a crucial p3 ssd drive. I have set it up with Ubuntu, 23.10 I think, and it runs great but it is running PCIe Gen 2, is it possible to force gen 3 whilst running Ubuntu? And if so how is the best way of doing it?
Many thanks in advance
Andrew
Sorry Jeff, I have now sorted it out - found the Config.txt file and edited it adding the 2 lines in your article. (for some reason it was giving an error when i tried to edit through the command line). So now running great at Gen 3.
Kind regards,
Andrew
Hi Jeff,
Found the config.txt in Ubuntu 23.10 and forced gen 3 and speedtest for my Crucial P3 plus is 809.78 (read speed)
I use the Argon Neo 5 with NVME and is working fine.
thanx.
Awesome stuff as Always ! I would love to try this on my RPi 5 or even CM4. Is there an adapter yet for the RPi5 PCIe ribbon cable to PCIe X adapter ?
Thank you Jeff !
Searching also for making an old pci-x Intel quad card to work.
Love your tuts!
For my old hw (rpi4b) seems to only mod?
Hey Jeff,
I just wanted to drop a couple of quick things:
1.) I really like the content you create. I've been subbed to your youtube for quite a while now and just wanted to say keep up the good work.
2.) I also have crohn's and am a developer. Unfortunate, but cool connections.
3.) Have you attempted this with the CM4 on your compute blade? Is that even possible? I'm thinking I might give it a try when I get my first blade (I ordered several), but I'd be interested to know if that's possible before playing around with things on my own blade. If it IS possible, I think it'd make a cool video (I'm thinking that with enough cooling + overclocking the pi 4, the SOC may be able to support PCIe 3.0 since the differences in the Pi4 and Pi5 aren't THAT huge, right?)
Unfortunately the PCIe controller on the CM4/Pi 4 is not capable of PCIe Gen 3 speeds no matter what, at least not with the firmware that Raspberry Pi ships.
Thanks Jeff! Good to know. I was kind of afraid of that. Crossing my fingers that:
a.) CM5 has the same features as pi5 (or better)
b.) the CM5 will run on the compute blade and
c.) the compute blade will support running at 3.0 speeds with a CM5