The Raspberry Pi can boot off NVMe SSDs now

When the Compute Module 4 was released (see my CM4 review here), I asked the Pi Foundation engineers when we might be able to boot off NVMe storage, since it was trivially easy to use with the exposed PCIe x1 lane on the CM4 IO Board.

The initial response in October 2020 was "we'll see". Luckily, after more people started asking about it, beta support was added for direct NVMe boot just a couple weeks ago.

MirkoPC with SN750 WD_BLACK NVMe SSD and Raspberry Pi Compute Module 4

Building the World's Tiniest NVMe RAID Array

Just posting to the blog for reference; I posted this video on YouTube recently, in which I built (what I believe to be) the world's tiniest NVMe SSD RAID array, using the Raspberry Pi Compute Module 4 and three diminutive WD SN520 NVMe drives (which are M.2 2230 size, which makes them each about the size of a quarter):

.embed-container { position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden; max-width: 100%; } .embed-container iframe, .embed-container object, .embed-container embed { position: absolute; top: 0; left: 0; width: 100%; height: 100%; }

I ran some benchmarks in RAID 5 and RAID 0, as well as one drive by itself, and found one surprising thing: the Pi's overall IO bandwidth is already saturated by just one drive, so putting NVMe disks in RAID doesn't really help with performance, like it does with slower spinning hard drives.

Pi Day 2021 - Livestream of 16 drives on a Raspberry Pi (2nd attempt)

For Pi Day, I'm going to livestream my second attempt at getting 16 hard drives (well, 12 hard drives and 4 SSDs) recognized by a Raspberry Pi.

The first attempt went decently well... but I wound up running into power supply issues.

.embed-container { position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden; max-width: 100%; } .embed-container iframe, .embed-container object, .embed-container embed { position: absolute; top: 0; left: 0; width: 100%; height: 100%; }

This time around, I will hopefully have those issues solved, and also we may have a little fun building a software-RAID-on-hardware-RAID (depending on how crazy we want to get). It probably won't work like I expect, but that's what makes it fun!.

M.2 on a Raspberry Pi - the TOFU Compute Module 4 Carrier Board

Ever since the Pi 2 model B went to a 4-core processor, disk IO has often been the primary bottleneck for my Pi projects.

You can use microSD cards, which aren't horrible, but... well, nevermind, they're pretty bad as a primary disk. Or you can plug in a USB 3.0 SSD and get decent speed, but you end up with a cabling mess and lose bandwidth and latency to a USB-to-SATA or USB-to-NVMe adapter.

The Pi 4 actually has an x1 PCI Express gen 2.0 lane, but the USB 3.0 controller chip populates that bus on the model B. The Compute Module 4, however doesn't presume anything—it exposes the PCIe lane directly to any card it plugs into.

TOFU board by Oratek - Raspberry Pi Compute Module 4 Carrier with M.2 slot

And in the case of Oratek's TOFU, it's exposed through an M.2 slot, making this board the first one I've used that can accept native NVMe storage, directly under the Pi:

Hardware RAID on the Raspberry Pi CM4

A few months ago, I posted a video titled Enterprise SAS RAID on the Raspberry Pi... but I never actually showed a SAS drive in it. And soon after, I posted another video, The Fastest SATA RAID on a Raspberry Pi.

Broadcom MegaRAID SAS storage controller HBA with HP 10K drives and Raspberry Pi Compute Module 4

Well now I have actual enterprise SAS drives running on a hardware RAID controller on a Raspberry Pi, and it's faster than the 'fastest' SATA RAID array I set up in that other video.

A Broadcom engineer named Josh watched my earlier videos and realized the ancient LSI card I was testing would not likely work with the ARM processor in the Pi, so he was able to send two pieces of kit my way:

Argon One M.2 Raspberry Pi SSD Case Review

I'm a fan of Raspberry Pi cases that keep my Pi cool. And the cases made by Argon Forty have great cooling, which is one reason they're a popular choice. Their latest Argon One M.2 case also adds a built-in high speed SSD drive slot!

Argon One M.2 Case for Raspberry Pi

A few months ago, someone from Argon Forty reached out and asked if I'd like to review the Argon ONE M.2, after they watched my video on booting a Pi 4 from an external SSD, and I accepted.

Unlike most Pi cases, this one actually adds features through it's design, like putting all the ports on the back, and adding a mostly-internal SSD, and so I decided to put it through its paces and see what I liked, and what I didn't like.

Video Review

I also posted a video version of this blog post on YouTube:

Building the fastest Raspberry Pi NAS, with SATA RAID

Since the day I received a pre-production Raspberry Pi Compute Module 4 and IO Board, I've been testing a variety of PCI Express cards with the Pi, and documenting everything I've learned.

The first card I tested after completing my initial review was the IO Crest 4-port SATA card pictured with my homegrown Pi NAS setup below:

Raspberry Pi Compute Module 4 with IOCrest 4-port SATA card and four Kingston SSDs

But it's been a long time testing, as I wanted to get a feel for how the Raspberry Pi handled a variety of storage situations, including single hard drives and SSD and RAID arrays built with mdadm.

I also wanted to measure thermal performance and energy efficiency, since the end goal is to build a compact Raspberry-Pi based NAS that is competitive with any other budget NAS on the market.

Raspberry Pi USB Boot - UASP, TRIM, and performance

In the past few weeks, I reviewed USB drive performance on the Raspberry Pi 4, and the importance of UASP support for USB drive performance.

Both posts generated great discussion, and there were three things I wanted to cover in this follow-up, namely:

  1. Which drives support UASP
  2. Real-world performance benchmarks
  3. TRIM support

For reference, here are all the products I'm testing in this post (product links are to their Amazon product page, starting from top middle, clockwise):

USB Performance testing - SATA SSD, NVMe, and Flash drives

Enabling TRIM on an external SSD on a Raspberry Pi

I've been doing a lot of benchmarking and testing with the Raspberry Pi 4 and SSDs connected via USB. I explored UASP Support, which USB SSDs are the fastest, and I'm now booting my Pis from USB SSDs.

Anyways, one thing that I have wondered about—and some people have asked me about—is TRIM support.

I'm working on a new video for my YouTube channel that will go into some more detail on which of the drives I tested support TRIM, but while I was researching for that video, I also found that TRIM support in Linux is not as simple as it seems at first glance—it's definitely not plug-and-play, in my experience.

While internal microSD cards seem to support TRIM out of the box, none of the external USB drives I tested supported it out of the box. They all needed a little help!

The fastest USB storage options for Raspberry Pi

For years, I've been maintaining benchmarks for microSD cards on the Raspberry Pi, but I only spent a little time testing external USB storage, due to historic limitations with the Pi's USB 2.0 bus.

But the Pi 4 cleared away the limitations with a full-speed USB 3.0 bus offering much better performance, so I've done a lot of testing with USB boot, and with all the USB SSDs I had at my disposal. You can see some of those results in this blog post and video on booting a Pi 4 via USB.

After posting my tests concerning UASP support in USB SATA adapters, I got an email from Rob Logan mentioning the performance of some other types of drives he had with him. And he even offered to ship a few drives to me for comparisons!

There's also a video that accompanies this blog post, for the more visually-inclined: