ssd

HTGWA: Use bcache for SSD caching on a Raspberry Pi

This is a simple guide, part of a series I'll call 'How-To Guide Without Ads'. In it, I'm going to document how I set up bcache on a Raspberry Pi, so I could use an SSD as a cache in front of a RAID array.

Getting bcache

bcache is sometimes used on Linux devices to allow a more efficient SSD cache to run in front of a single or multiple slower hard drives—typically in a storage array.

In my case, I have three SATA hard drives: /dev/sda, /dev/sdb, and /dev/sdc. And I have one NVMe SSD: /dev/nvme0n1.

I created a RAID5 array with mdadm for the three hard drives, and had the raid device /dev/md0.

I then installed bcache-tools:

$ sudo apt-get install bcache-tools

And used make-bcache to create the backing and cache devices:

Kubesail's PiBox mini 2 - 16 TB of SSD storage on a Pi

Kubesail Raspberry PiBox mini 2 front side exposed

Many months ago, when I was first testing different SATA cards on the Raspberry Pi Compute Module 4, I started hearing from GitHub user PastuDan about his experiences testing a few different SATA interface chips on the CM4.

As it turns out, he was working on the design for the PiBox mini 2, a small two-drive NAS unit powered by a Compute Module 4 with 2 native SATA ports (providing data and power), 1 Gbps Ethernet, HDMI, USB 2, and a front-panel LCD for information display.

The Hardware

The PiBox mini 2 is powered by the Compute Module 4 on this interesting carrier board:

PiBox mini carrier board with Raspberry Pi Compute Module 4

Raspberry Pi OS now has SATA support built-in

After months of testing various SATA cards on the Raspberry Pi Compute Module 4, the default Raspberry Pi OS kernel now includes SATA support out of the box.

SATA card and Samsung SSD with Raspberry Pi Compute Module 4 IO Board

In the past, if you wanted to use SATA hard drives or SSDs and get native SATA speeds, and be able to RAID them together for redundancy or performance, you'd have to recompile the Linux kernel with SATA and AHCI.

Sure you could always use hard drives and SSDs with SATA to USB adapters, but you sacrifice 10-20% of the performance, and can't RAID them together, at least not without some hacks.

There's a video version of this post: SATA support is now built into Raspberry Pi OS!

Trying KIOXIA CM6 and PM6 Enterprise SSDs on a Raspberry Pi

Late last year, an engineer at Broadcom sent me some hardware and offered some help getting Broadcom's MegaRAID card working on the Raspberry Pi. It took some time, but eventually we were able to get the card and a demonstrator 'UBM' backplane working on the Pi, and it culminated in my posting about Hardware RAID on the Pi, and on a livestream, getting 16 hard drives working on a Pi.

The one thing I couldn't test in those earlier videos was the backplane and storage card's 'Tri-mode' support, allowing PCI Express NVMe drives—like KIOXIA's CM6—to work in the same slot as the SATA and SAS drives I was used to testing.

So after some conversation with reps at KIOXIA, I was able to get a PM6 and three CM6 drives on loan to test them:

KIOXIA CM6 and PM6 SSD with Raspberry Pi Compute Module 4

The Raspberry Pi can boot off NVMe SSDs now

When the Compute Module 4 was released (see my CM4 review here), I asked the Pi Foundation engineers when we might be able to boot off NVMe storage, since it was trivially easy to use with the exposed PCIe x1 lane on the CM4 IO Board.

The initial response in October 2020 was "we'll see". Luckily, after more people started asking about it, beta support was added for direct NVMe boot just a couple weeks ago.

MirkoPC with SN750 WD_BLACK NVMe SSD and Raspberry Pi Compute Module 4

Building the World's Tiniest NVMe RAID Array

Just posting to the blog for reference; I posted this video on YouTube recently, in which I built (what I believe to be) the world's tiniest NVMe SSD RAID array, using the Raspberry Pi Compute Module 4 and three diminutive WD SN520 NVMe drives (which are M.2 2230 size, which makes them each about the size of a quarter):

.embed-container { position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden; max-width: 100%; } .embed-container iframe, .embed-container object, .embed-container embed { position: absolute; top: 0; left: 0; width: 100%; height: 100%; }

I ran some benchmarks in RAID 5 and RAID 0, as well as one drive by itself, and found one surprising thing: the Pi's overall IO bandwidth is already saturated by just one drive, so putting NVMe disks in RAID doesn't really help with performance, like it does with slower spinning hard drives.

Pi Day 2021 - Livestream of 16 drives on a Raspberry Pi (2nd attempt)

For Pi Day, I'm going to livestream my second attempt at getting 16 hard drives (well, 12 hard drives and 4 SSDs) recognized by a Raspberry Pi.

The first attempt went decently well... but I wound up running into power supply issues.

.embed-container { position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden; max-width: 100%; } .embed-container iframe, .embed-container object, .embed-container embed { position: absolute; top: 0; left: 0; width: 100%; height: 100%; }

This time around, I will hopefully have those issues solved, and also we may have a little fun building a software-RAID-on-hardware-RAID (depending on how crazy we want to get). It probably won't work like I expect, but that's what makes it fun!.

M.2 on a Raspberry Pi - the TOFU Compute Module 4 Carrier Board

Ever since the Pi 2 model B went to a 4-core processor, disk IO has often been the primary bottleneck for my Pi projects.

You can use microSD cards, which aren't horrible, but... well, nevermind, they're pretty bad as a primary disk. Or you can plug in a USB 3.0 SSD and get decent speed, but you end up with a cabling mess and lose bandwidth and latency to a USB-to-SATA or USB-to-NVMe adapter.

The Pi 4 actually has an x1 PCI Express gen 2.0 lane, but the USB 3.0 controller chip populates that bus on the model B. The Compute Module 4, however doesn't presume anything—it exposes the PCIe lane directly to any card it plugs into.

TOFU board by Oratek - Raspberry Pi Compute Module 4 Carrier with M.2 slot

And in the case of Oratek's TOFU, it's exposed through an M.2 slot, making this board the first one I've used that can accept native NVMe storage, directly under the Pi:

Hardware RAID on the Raspberry Pi CM4

A few months ago, I posted a video titled Enterprise SAS RAID on the Raspberry Pi... but I never actually showed a SAS drive in it. And soon after, I posted another video, The Fastest SATA RAID on a Raspberry Pi.

Broadcom MegaRAID SAS storage controller HBA with HP 10K drives and Raspberry Pi Compute Module 4

Well now I have actual enterprise SAS drives running on a hardware RAID controller on a Raspberry Pi, and it's faster than the 'fastest' SATA RAID array I set up in that other video.

A Broadcom engineer named Josh watched my earlier videos and realized the ancient LSI card I was testing would not likely work with the ARM processor in the Pi, so he was able to send two pieces of kit my way:

Argon One M.2 Raspberry Pi SSD Case Review

I'm a fan of Raspberry Pi cases that keep my Pi cool. And the cases made by Argon Forty have great cooling, which is one reason they're a popular choice. Their latest Argon One M.2 case also adds a built-in high speed SSD drive slot!

Argon One M.2 Case for Raspberry Pi

A few months ago, someone from Argon Forty reached out and asked if I'd like to review the Argon ONE M.2, after they watched my video on booting a Pi 4 from an external SSD, and I accepted.

Unlike most Pi cases, this one actually adds features through it's design, like putting all the ports on the back, and adding a mostly-internal SSD, and so I decided to put it through its paces and see what I liked, and what I didn't like.

Video Review

I also posted a video version of this blog post on YouTube: