NVMe SSD boot with the Raspberry Pi 5

Pi 5 PCIe NVMe Kioxia XG8 SSD

In my video about the brand new Raspberry Pi 5, I mentioned the new external PCIe port makes it possible to boot the standard Pi 5 model B directly off NVMe storage—an option which is much faster and more reliable than standard microSD storage (even with industrial-rated cards!).

Enabling NVMe boot is pretty easy, you add a line to /boot/config.txt, modify the BOOT_ORDER in the bootloader configuration, and reboot!

Of course, you'll also need to get Pi OS onto the NVMe, and there are a few ways to do that—I'll walk you through my favorite method below.

Note: Raspberry Pi announced they are developing an official Raspberry Pi M.2 HAT, but there is no word on a launch date for it yet. I am also tracking other PCIe HATs for Raspberry Pi 5 here.

Enable the external PCI Express port

First, enable the external PCIe port on the Raspberry Pi 5. Edit /boot/config.txt and add the following at the bottom:

# Add to bottom of /boot/config.txt
dtparam=pciex1

# Note: You could also just add the following (it is an alias to the above line)
# dtparam=nvme

# Optionally, you can control the PCIe lane speed using this parameter
# dtparam=pciex1_gen=3

I have the pciex1_gen=3 part commented out above because Raspberry Pi allows you to tweak the bus speed (you can choose Gen 1 for 2.5 GS/s, Gen 2 for 5 GS/s, and Gen 3 for 8 GS/s), but the port is only rated for up to PCIe Gen 2 speeds.

In practice, I have been able to run multiple NVMe SSDs at Gen 3.0 speed (getting up to 900 MB/sec) on my alpha Pi 5, but YMMV—PCIe can be very fickle, depending on the quality of the FFC cable and connections on your own setup.

Set NVMe early in the boot order

The PCIe connection should work after a reboot, but your Pi won't try booting off an NVMe SSD yet. For that, you need to change the BOOT_ORDER in the Raspberry Pi's bootloader configuration:

# Edit the EEPROM on the Raspberry Pi 5.
sudo rpi-eeprom-config --edit

# Change the BOOT_ORDER line to the following:
BOOT_ORDER=0xf416

# Press Ctrl-O, then enter, to write the change to the file.
# Press Ctrl-X to exit nano (the editor).

Read Raspberry Pi's documentation on BOOT_ORDER for all the details. For now, the pertinent bit is the 6 at the end: that is what tells the Pi to attempt NVMe boot first!

Reboot your Raspberry Pi 5 to make the change take effect.

NVMe boot won't work unless you have the external PCI Express port enabled, and there's a working NVMe drive with a valid boot partition! If you don't have that (e.g. you used Raspberry Pi Imager with an external USB NVMe adapter to flash Pi OS to an NVMe drive from another computer), then follow the steps in the next section to clone your existing Pi OS install to an NVMe SSD.

Flash the SSD with Raspberry Pi Imager

To get the NVMe SSD to boot your Pi, it needs to have an OS. One option would be to clone an existing installation to it using rpi-clone or some other tool (see below), but my preferred option is to flash a fresh Pi OS install using Raspberry Pi Imager.

  1. Install Pi Imager and open it
  2. Plug your NVMe SSD into your computer using a USB to NVMe adapter
  3. Choose an OS to install
  4. Choose the drive (connected through your adapter) to flash
  5. Click write (and set any options you'd like)

Once flashing is complete, pull the NVMe drive, attach it to your Pi 5, and it should boot off it (with or without a microSD card inserted)—assuming you have the bootloader up to date and set the BOOT_ORDER appropriately!

Clone your microSD boot volume to an NVMe SSD

NOTE: This process may not be as reliable as flashing Raspberry Pi OS directly to an NVMe drive (see above). It has worked sometimes in my testing, but not always. I recommend flashing a fresh Pi OS image to an NVMe drive if at all possible.

Assuming you already have Raspberry Pi OS on a microSD card that is booting your Raspberry Pi 5 internally, and the NVMe SSD is connected and visible (check if you see a device /dev/nvme0n1 after running lsblk), you can use rpi-clone to clone the internal microSD boot volumes to your NVMe SSD:

# Install rpi-clone.
git clone -b 123-nvme https://github.com/geerlingguy/rpi-clone.git
cd rpi-clone
sudo cp rpi-clone rpi-clone-setup /usr/local/sbin

# Clone to the NVMe drive (usually nvme0n1, but check with `lsblk`).
sudo rpi-clone nvme0n1

Note: I'm using my fork of rpi-clone, because the official version has not merged NVMe support yet.

Note 2: You may want to wipe all disk partitions before cloning:

sudo umount /dev/nvme0n1p?
sudo wipefs --all --force /dev/nvme0n1p?
sudo wipefs --all --force /dev/nvme0n1
sudo dd if=/dev/zero of=/dev/nvme0n1 bs=1024 count=1

NVMe behind a PCIe bridge / switch

Currently the Raspberry Pi 5 only exposes one PCIe lane externally—though there are four more lanes taken up by the RP1 chip. Typical PC motherboards have a number of lanes to play with, so you often find two, three, or even four M.2 NVMe slots on high-end motherboards.

Even there, some motherboards have PCI Express switches (or 'bridges') which allow multiple PCIe devices to share the same lane or lanes, in a similar way an Ethernet switch can allow multiple computers to share a single network connection.

NVMe boot behind PCIe switch on Raspberry Pi

On the Compute Module 4, bootloader space constraints prevented NVMe boot if you used a switch, but I wonder if that restriction is lifted on the Raspberry Pi 5—and if so, is it already implemented?

As of now, no. I can see and use an NVMe SSD through a PCIe switch, but I am not able to boot the Raspberry Pi 5 from it, unless it is directly connected (as the lone PCIe device on the bus).

I've opened an issue to ask about this feature in the Raspberry Pi firmware repo: Can't boot Pi 5 via NVMe behind PCIe switch / bridge.

Comments

Well done Jeff, I don't know where you find the time! Hopefully we will see some decent Pi5 nvme NAS hardware if PCIe switching is reliable.

My Pi5 4GB arrived yesterday, so retail has started shipping - at least in the UK.

I am seeing some spurious nvme boot disk issues (the same nvme to USB adapter and SSD from my Pi4). I'm investigating root cause and will open an issue if I can't fix it.

Nice to hear your Pi5 has turned up. My 8GB is still sitting at “waiting to be picked” at Pimoroni. I was hoping it would at least ship on the 23rd.

I wonder if it is possible to bridge two raspberry pi 5s together via their pcie buses?
The purpose would be to have one running Android TV and the other running Hyperbian? That might avoid having to connect an external video decoder/encoder for ambient style lighting for the LED strip around the rim of the TV.

Hi Jeff,
can you suggest me, please, what I have to buy to attach an SSD NVMe to the Raspberry Pi (and boot the OS from the NVMe)? What kind of device are using you?
Regards
Massimiliano

I think Jeff is using a dev board from the Pi foundation. There will be actual PCIe HATs available in the future though, the foundation had a picture of one of their M.2 PCIe HATs in one of their blog posts about the Pi5

Nice one!

I'm currently using an SSK M.2 NVME SATA Adapter plugged into the USB 3.0 port with a WD Blue SA510 2TB M.2 SATA SSD (Prime day sale purchase). It was not happy at first, but after a little trial and error it is working perfectly. The USB speeds are amazing, copying files of my Pi 4 takes hours to a USB memory stick, then literally seconds or minutes to copy from the drive to my Pi 5.

I can't wait for an official PCIe M.2 adapter, hopefully they are wallet friendly. Currently transferring some of my old Docker containers from the Pi 4 to the Pi 5 (mainly for the extra storage and speed for Jellyfin and NextCloud, my Pi 4 has a 500GB M.2 SSD in the Argon ONE M.2).

If anyone intersted, you can run the rpi5 at max performance without a PD psu, only a +- cable and 5v 5A+ psu is needed and this row in the eeprom config file:
PSU_MAX_CURRENT=5000

Fresh and crusty info from rpi support.

Seems like this would preclude the use of the fan for active cooling of the BCM chip, no?

Supposedly the official M.2 HAT would be high enough the active cooler could still run and get enough airflow underneath. Supposedly

Any reason not to use an M,2 to USB adapter card in one of the USB3 connectors while waiting for the official M.2 HAT? It may not be as fast, but this might save a few steps down the road.

Similar to microSD, around 6-8 seconds with Pi OS. The boot time is highly optimized, and the NVMe's speed advantage isn't as great there as other storage-related tasks (like file copies, installs, launching heavy apps, that sort of thing).

Hi Jeff
Would be grateful if you can give some advise on what types of NVMe M.2 SSD drives would be compatible. Would Gen4 x 4 PCIe work, or do I need to stick with Gen3 x 4 PCIe?

PCIe devices are usually very good about backwards compatibility / slower speeds—so you can generally take any drive (even PCIe Gen 5) and drop it into even PCIe Gen 1 x1 and it will work, just a bit slower.

Thanks very much Jeff. Would you generally say go for PCIe Gen5 SSD over earlier gen version for raspberry Pi 5, or would there generally be no benefit given it will downgrade?

I would go for a Gen 3 or Gen 4 SSD for now—Gen 3 especially is getting cheaper as more people move on to Gen 4, so for example, TeamGroup seems to be doing a lot of sales on their older drive versions, which are still very fast on Gen 3 systems.

Hi Jeff,

This is great stuff, thanks for sharing! For booting off a switched NVME, would it work to have /boot on an SD card, and / on the NVME? That way the RPi boot loader isn't hampered by the switch, but by the time Grub is running off the SD, it should be able to mount and load the rest of the file system from the NVME. Would obviously need to run a full installer instead of flashing an image, but that's not the most awful process.

Thanks for this post, Jeff! Instead of waiting for the hats and adapters, I use NFSv4 to share ZFS pools and file systems on the Pi with an HP Proliant Microserver which has 4 SATA bus bays. Speed over a wired intranet connection isn’t that much of an issue for me, I’m mainly interested in mirrored redundancy, and capacity. I run Ubuntu 23.10 on the HP, and Bookworm-based RPiOS on my Pi. Both with ZFS installed.

Hello Jeff,

First off I love your videos and blog. They are so helpful and full of knowledge. I was wondering what the board is called that you are connecting the flex pci cable to in your pictures that allows you to use the pcie port? Thanks in advance

That is a board Raspberry Pi designed for testing and evaluation of the PCIe port on the Pi 5. Unfortunately they are not yet selling it or anything like it. Hopefully a third party will!

Jeff,
I have always been using USB3 to NVMe adapters on my Pi4's and Pi3's without issue. I tried the same while waiting for the Pineberry board with my Pi5, and it will occasionally try to boot. Once it does, its just a matter of minutes before it comes up with a write error and "file system is read-only" error. booting from SD card and using the NVMe as a storage drive, it eventually does the same. Have tried different NVMe drives in the adapter, same issues. these same drives/Adapter work fine with other Pi's with Bookworm. Any ideas?

Which power supply are you using? If it's not the official 5A 5V power supply, it could be supplying inadequate power to the USB ports.