The Raspberry Pi can boot off NVMe SSDs now

When the Compute Module 4 was released (see my CM4 review here), I asked the Pi Foundation engineers when we might be able to boot off NVMe storage, since it was trivially easy to use with the exposed PCIe x1 lane on the CM4 IO Board.

The initial response in October 2020 was "we'll see". Luckily, after more people started asking about it, beta support was added for direct NVMe boot just a couple weeks ago.

MirkoPC with SN750 WD_BLACK NVMe SSD and Raspberry Pi Compute Module 4

And in a stroke of perfect timing, I also received a hand-made MirkoPC board from Mirek in Poland last week, which among other features includes a full-size M.2 M-key slot, meaning I could test out NVMe boot on it with a Compute Module 4 and a WD_black SN750 500GB SSD I use for testing.

Video version of this post

I also posted a video about booting from NVMe on my YouTube channel; it's embedded below:

Setting up NVMe Boot

The setup process is a little involved right now, since you have to set the correct boot order, update the Pi's bootloader, update the Pi's firmware, but it's not too crazy.

Because I had some trouble with the MirkoPC's built-in microSD card slot, I chose to use an eMMC-powered Compute Module 4, so I flashed Pi OS to it, then set the Pi to USB mass storage mode.

September 2022 Update: NVMe boot is no longer in beta: see the new NVMe boot documentation for more details. I am currently using boot order BOOT_ORDER=0xf25416. See all available BOOT_ORDER options here.

The new process for updating the boot order is to:

  1. Clone rpiboot: git clone --depth=1 https://github.com/raspberrypi/usbboot
  2. Build rpiboot: cd usbboot && make
  3. Open the recovery/boot.conf file and update the BOOT_ORDER variable so it reads: BOOT_ORDER=0xf25416
  4. Run ./update-pieeprom.sh to update the pieeprom.bin file with the new settings
  5. Connect your CM4 in USB boot mode (see detailed guide) and run: ../rpiboot -d .

After a few seconds it should successfully update the EEPROM and the new boot order should be ready. The activity LED should start blinking rapidly once complete.

I followed the beta instructions to change the BOOT_ORDER, but I had to modify the sed command on my Mac, since the arguments are different for the non-GNU version of sed shipped with macOS:

sed -i '' 's/\(BOOT_ORDER=.*\)6\(.*\)/\1\26/' boot.conf

(Note the addition of the ''.)

For the firmware update, I used the built-in SD Card Copier utility to copy all data from the eMMC storage over to the NVMe SSD, and made sure to check the "New Partition UUIDs" option when doing so.

After that, I rebooted the Pi, and used lsblk to verify the NVMe was mounted as / and /boot, and then got on to benchmarking.

Cloning Raspberry Pi OS from microSD to NVMe

The easiest way to migrate from microSD to an NVMe drive (or, really, any other boot volume) on the Raspberry Pi is to use rpi-clone. (The commands below use my fork of rpi-clone instead of the original, since that version is no longer maintained.)

All you have to do is install rpi-clone, then clone the disk:

# Install rpi-clone.
git clone https://github.com/geerlingguy/rpi-clone.git
cd rpi-clone
sudo cp rpi-clone rpi-clone-setup /usr/local/sbin

# Clone to the NVMe drive (usually nvme0n1, but check with `lsblk`).
sudo rpi-clone nvme0n1

Confirm that you would like to format the NVMe drive, then wait for the clone to complete. Assuming you have the BOOT_ORDER configured with NVMe taking priority, your Raspberry Pi should boot off the NVMe.

Benchmarking NVMe vs microSD, eMMC, and USB adapters

The two main things I wanted to test were boot times and app launch/use times, to see what kind of impact the NVMe storage would have.

I already know from my World's tiniest NVMe RAID video that NVMe storage is, on average, 12x faster on the Pi for normal file operations (copies, random read/write), but how much does it affect everyday performance?

Well, boot times are hardly affected by disk IO, it seems; after realizing I had to disable Bluetooth in /boot/config.txt and disable the hciuart service, since my CM4 didn't have a wireless module, I got pretty much the exact same boot time across NVMe, eMMC, and microSD bootups:

Boot time benchmarks for NVMe, eMMC, and microSD on Pi

But the next test reveals more of the typical speedup you'd encounter on a Pi, day-to-day. I wrote a Node.js benchmark script for Chromium to test how long it takes to launch Chromium, load a webpage, and quit last year, for some other testing.

I ran that same benchmark on each storage device (three times, restarting the Pi between each test, with < 2% standard deviation in the results), and got the following times:

Chromium usage benchmarks for NVMe, eMMC, and microSD on the Pi

In this test, NVMe storage is 44% faster than microSD, and around 20% faster than eMMC. The bigger surprise was how much better eMMC fares than microSD; the 8-bit configuration on the CM4 (vs. 4-bit on the CM3+ and earlier) makes a big difference, especially for random IO.

But 44% is a pretty significant difference. You will definitely notice the speedup from NVMe boot on the Pi, and that's not even considering the fact that you can get 4 TB (or larger) NVMe SSDs, and they have much better write longevity compared to all but the most expensive microSD cards.

And I know people will ask about it, so I also compared the exact same NVMe drive in a USB 3.0 to NVMe adapter, on the exact same Pi, using the exact same VL805 USB 3.0 chipset, to see what overhead there is when running NVMe storage through USB 3.0 on a Pi 4 model B:

Chromium benchmark with NVMe native vs USB 3.0 to NVMe adapter with UASP

So yes, there is a measurable difference—as I've found across many tests these past few months. Using native NVMe (vs. through USB adapters) is about 10% faster.

Anyway, I thought the MirkoPC would be a great little board to test NVMe boot on, though it works on any CM4 board (look at the list of Compute Module 4 project boards I am maintaining here), as long as you can adapt an M.2 NVMe SSD to the board's PCI Express connection (many boards have M.2 slots built-in!).

Check out the YouTube video for more details about my tests, and stay tuned (maybe subscribe to the blog's RSS!), as I'll continue exploring more storage options for the Pi, and I'll also be posting more about the MirkoPC soon!

Note: While booting directly off an NVMe SSD works, I and others found that if you use a PCIe bridge/switch, it doesn't seem to work (e.g. if you want to plug both an NVMe SSD and NIC into the Pi's PCIe bus). I opened up this issue on the bootloader repo to track progress: Can't boot CM4 via NVMe behind PCIe switch / bridge.

Comments

I'd love to see various USB options in the benchmarks (USB3 attached NVMe, USB3 attached SATA, etc)

Oh shoot! I completely forgot to add that benchmark in—I just updated the article with the benchmark comparing the same NVMe drive in the same Pi, but connected directly (first result — 6.04s) vs connected through the best USB 3.0 to NVMe adapter (with UASP support) that I've tested (out of four I have), and the result is the same as I've seen with many different tests—using adapted drives through USB 3.0 results in about a 10% performance reduction.

Check out the last benchmark graph in the post—I just updated it, so it could be 5-10 minutes before it appears.

One thing you may want to test out with the NVMe drives (especially with the limited power budget of the MirkoPC board) is forcing the NVMe drive to a lower power state and holding it there rather than letting it run at full wattage. The lower power states generally will mean lower performance, but given the limit the pi already has, it may not translate to real-world impact for this application. I do this with my Pinebook Pro and it's Intel 660p to keep consumption down, details on how we are limiting here: https://wiki.pine64.org/index.php?title=Pinebook_Pro#Post_NVMe_install_…

There is also a list of user tested drives and their power states in the wiki as well, not specifically with a Raspberry Pi, but should at least have good information: https://wiki.pine64.org/wiki/Pinebook_Pro_Hardware_Accessory_Compatibil…

I've kept my eye on that Wiki page but never thought about the power state changes—definitely a good idea on the MirkoPC since it can only supply 15W with the circuit that it has.

It will also lower heat, which will be a factor, but on the CM3, and the SSD itself. I agree that throttling the SSD is a win-win in this setup.

Did you get a diff file for the change of the boot.conf file?
A unified diff is good to show the changes made.

Why not put the /boot partition on sd and / on ssd? Boot speed is the same, root is fast and one does not have to mess with the firmware.

>> (look at the list of Compute Module 4 project boards I am maintaining here),

I love you.

Hi Jeff
Great article as always but try as I may - two new out of the box CM4s with eMMC, two new 128GB NVMe SSD's and a Compute Module 4 IO board - I simply cannot get this update to work. Convinced myself it's fat finger trouble at my end I've gone through the RPi official documentation and your video many times but no joy, the CM4's still boot off eMMC. I have checked and double checked that the boot order starts with 6 and it does so I'm wondering, have you tried this update on a CM4 IO board? And if so, were there any gotchas?

I did boot off the NVMe on the IO Board and didn't have an issue. Unfortunately my setup is a bit of a mess doing some other things right now, so I can't dig and and see anything else that might help. Maybe ask about it in the thread on the Raspberry Pi Forums?

I have now managed to get one new virgin pair of CM4 with eMMC + NVMe SSD working together with native boot but have tried so many different procedures that I am not sure exactly how I ended up with a working system! I think my problem was down to NOT using the latest usbboot. I had an earlier version installed and did not realise the significance at the time. It wasn't until I checked the repo that I saw the usbboot util had been updated a week or so ago and once I updated to the latest version from then on I was able to build a working system. However it was with a different procedure to the one you described. Installing the necessary updated kernel version (>#1403) seemed to be my sticking point. Got it in the end though. I am going to step by step document the procedure tomorrow to make sure it a) makes sense and b) is repeatable!

Step (b) is lost on far too many people. Good on you for doing that :)

And yeah, I forgot to mention that if you already have usbboot installed, make sure it's latest, because the nvme stuff was just added.

It's now a repeatable process albeit with a better understanding of what needs to be done to achieve the nvme boot. Next challenge for you Jeff - nvme boot on the far side of a PCIe switch. Try as I might I can't figure out why this doesn't work given that the switch I am using (ASM1182e) is supposed to be a transparent bridge with one lane in and two out. There is a post on the forum from me in reply to peterharperuck's post about this subject - with my findings, repeated here for convenience...
peterharperuk says ...
If you boot from the SD but want rootfs mounted from nvme then you'll either have to upgrade the kernel on there or use some other trick to get the driver loaded. I think it would be nice for us to get PCIe bridges/switches working at some point.

peterstrong replies
Yes it would be a big help for those of us who would like to use a CM4 Lite and boot directly from an nvme drive on the far side of a switch such as ASM1182e or ASM1184e. At the moment with a CM4 with eMMC on a CM4 IO board, with the latest kernel and usbboot fixes, I can boot from an nvme drive attached to the CM4 PCIe slot with both the boot / and /rootfs on the nvme. With the same HW setup I can also boot initially from / on eMMC and then hand over to /rootfs on nvme to complete the boot process and load the filesystem. AND with a switch (ASM1182e) in the CM4 IO board PCIe slot and the nvme ssd attached to one of the two PCIe switch output ports I can also boot from / on eMMC and hand over to /rootfs on the nvme to complete the boot process and load the filesystem BUT I cannot boot from / on the nvme when it is behind the switch, the boot partition needs to be on eMMC. I feel sure this could be done with some configuration changes but that is above my pay grade so hopefully some brighter spark than I will figure it out (Jeff Geerling perhaps?). However if it needs a modification to closed source binaries or the kernel then perhaps a kind soul at Pi will tell us so and I for one can stop trying to find a way to do this and wait patiently for an update.

Unfortunately the issue right now is that the bootloader would need to be able to work with NVMe through a switch, and it currently doesn't :(

(For reference.)

Hopefully one of the Pi engineers on the forum will put us out of our misery by telling us we need a mod to the kernel or closed source binaries before this problem will be fixed. But it would be very helpful as a first step to understand why it does not work !! Or better still telling us what they think the reason is it doesn't work, so we can put some effort into fixing it.

+1 on this front ... I have a PCIe riser which I bought from AliExpress as per below which is ASM1102e based: https://www.aliexpress.com/item/33031021554.html

It works just fine with both an NVMe and SATA controller within, alas I can't boot from the NVMe (and presumably the SATA controller too) when they're on the far side of the riser.

Given I'm running a CM4 lite, I guess my only option currently is to try and boot from the SD card, then hand off the rootfs to the NVMe.

I'd love a fix for this if anyone could help out!

Hi Jeff,

I am new in this raspberry pi world...
i want to have hardware that can run Home assistant for my smart home.
i read about raspberry pi 4 vs home assistant blue and couldn't decide witch one of them better, i search the internet and saw yours video about the mirkopc and tofu with raspberry pi module 4.
i have a few question please:
1. can you recommend for my needs witch one of them will be good?
2. how can i get the mirkopc/tofu? where can i buy them?
3. do u have list or links for the hardware, i mean nvme m.2, power supply, cases, fan etc...

i tried to find links but no luck..
thanks alot
yossi

Hello Jeff
,
I'm considering building CM4 based DIY portable laptop (that could do very basic tasks)
In my mind, it'll be something like rpi400 like. (with old thinkpad keyboard, or at least with trackpoint).
Planning to design something like MirkoPC board with NVME.
Being fan of Arch, I really want arch on that system.

Have you seen any information about arch on CM4 with NVME boot.

Love your work, your videos gave me this idea in the first place.

Garid.

The only distros so far that I've seen keeping up with some of the latest CM4 features are Pi OS and Ubuntu for Pi. Haven't kept up much with Arch's Pi support, unfortunately—you'd have to ask elsewhere.

Just another data point for anyone interested. I bought a Western Digital 500GB Blue SN550 NVMe along with a USB 3.1 Gen 2 enclosure with UASP and trim support (both linked below), I enabled UASP and trim and my Raspberry Pi 4 Model B would lock up every 2 or 3 days. After a lot of troubleshooting I eventually discovered the Raspberry Pi would lose its connection to the drive controller, i.e. listing usb devices via lsusb would no longer show the drive controller. I tried powering the drive with a powered USB hub, I disabled UASP and trim, and I also tried booting from SD and simply mounted the drive as a non-boot partition, but the problem persisted. So I believe there's just an incompatibility with this hardware combination so others might want to avoid it:

WD drive: https://www.amazon.com/gp/product/B07YFF3JCN
USB enclosure: https://www.amazon.com/gp/product/B08C2THR25

Hi,

So 6 months went by, is this feature still in beta, or is it stable yet? Do I still have to perform the actions described?

Thanks!
Regards

Hi Jeff.

Is the procedure for booting an CM4 from NVMe when the CM4 is *without* eMMC any different ? Have just had a quick glance at the documentation, but seems to be rather complicated so hoping a NON-eMMC CM4 would be simpler :)

regs,

Per Arne Munthe
Oslo, Norway

I did a whole bunch of things before getting the CM4 to boot from the M.2 NVME stick, but I think worked for me was to put the M.2 NVME drive into a USB adapter and image the drive. I then just stuck the M.2 NVME stick into the slot on the TOFU and that seemed to work. My CM4 is very new, so it could have a newer firmware that makes this work much more easily.

2 months ago I bought 3 Pi 4s. With those I also bought 3 nvme drives. They very often became unresponsive. Way too often. So, yesterday I replaced them all with SATA III drives. The Pis have worked flawlessly since then. Something just isn't working with the m.2s.

They likely use too much power when in the default power mode that is typically used for high speed and that is what causes the issues? Set the power use down with nvme set-feature /dev/nvme0 -f 2 -v 2 -s assuming you are not running off USB M.2 drives. Could run in P1 instead of P2 but do not run it in P0 unless the USB port has enough power available. The RP4 does not.

Do you think it is possible to have it run on PCI 3x2 or PCI 3x4 ?
I searched online but found no answer