I built a $5,000 Raspberry Pi server (yes, it's ridiculous)

When I heard about Radxa's Taco—a Raspberry Pi Compute Module 4-powered NAS/router-in-a-box—I knew what must be done.

Load it up with as much SSD storage as I can afford, and see what it can do.

Raspberry Pi CM4 Taco NAS with 48 TB of SSD storage

And after installing five Samsung 870 QVO 8TB SSDs and one Sabrent Rocket Q NVMe SSD—loading up every drive slot on the Taco to the tune of 48TB raw storage—I found out it can actually do a lot! Just... not very fast. At least not compared to a modern desktop.

Special thanks to Lambda for sponsoring this project—I was originally going to put a bunch of the cheapest SSDs I had on hand on the Taco and call it a day, but with Lambda's help I was able to buy the 8TB SSDs to make this the most overpowered Pi storage project ever!

What can it do?

Well, for starters, ZFS on the Pi is now easy. I wrote an entire ZFS-on-Pi guide here; you install the Pi kernel headers, then apt install zfs-dkms zfsutils-linux and you're on your way.

And with the Compute Module 4's exposed PCI Express Gen 2.0 lane, you can use a decent SATA controller (the Taco uses JMicron's JMB585) and put as many SATA III drives as you want on the bus.

Sadly, the Pi's bus being only Gen 2 and x1 means you're quite limited in terms of bandwidth, though. All RAID levels (including RAIDZ1) basically maxed out the Pi's bus on sequential reads:

RAID results on Raspberry Pi CM4 Radxa Taco

And of course, that Sabrent NVMe drive is quite bottlenecked, also only seeing a few hundred megabytes per second of throughput in the best case.

I was pleasantly surprised with how well ZFS performed, though—I initially thought a RAIDZ1 would be slower than a typical mdadm-based RAID5 array, but it actually performed better in many cases.

But better's still not amazing, since the Pi's (relatively) anemic CPU throttles pretty much everything from the past decade, since PCI Express Gen 3 was a thing.

Networking too

The Taco has another trick up it's sleeve, though (two, in fact)—since they are already using an ASMedia PCIe switch chip to split traffic between an M.2 NVMe slot and the SATA ports, they also placed a Realtek 8125b 2.5 Gbps Ethernet NIC onboard—so you get a 2.5G Ethernet port (in addition to the Pi CM4's built-in 1 Gbps NIC).

And to top that off, they'll also include a second M.2 slot—this time E-key—so you could add a WiFi 6 chip, or a machine learning accelerator, like Google's Coral TPU (note: drivers for the latter aren't working on any Pi yet, but new CM4-compatible boards may have a better shot!).

Anyways, I had to install Realtek's driver to get it running, but I found out the latest Pi OS kernel and firmware actually support Realtek chipsets out of the box now—no word yet on when that'll trickle down to a stable build (right now you have to run rpi-update to get it).

iperf3 network benchmark on 2.5G network on Radxa Taco CM4 board

I tested the 2.5G network throughput and didn't have any trouble saturating my home network at 2.35 Gbps both ways to my other high-speed devices.

Having 2.5G networking and a bunch of SATA drives, I also tested Samba file copy performance on the various configurations:

Samba SMB network copy performance benchmark results with CM4 Radxa Taco Raspberry Pi NAS

And here's where the Pi's story falls apart a little—and why it's probably best to throw higher-speed storage at a processor able to keep up. Because the CPU is bottlenecking the RAID parity calculations already, the interrupts that hit when network traffic goes up results in a lot lower performance than you might expect.

I think the ideal setup for a Pi-based storage device would be low-end SSDs (or even 3.5" HDDs—they work with the Taco too, though you will need SATA/power extension cables), and RAID 1 or RAID 10 (or ZFS stripe+mirror). That way the Pi's CPU will be free when it comes to putting through more network traffic.

Over gigabit networks, the Pi is perfectly adequate, but I like to go big or go home, so I target 2.5G (minimum) for all my new gear.

It remains to be seen whether any of the higher-end cards I've been testing might be able to transfer data direct from storage to the network (and vice-versa). Supposedly this is A Thing™ with some Mellanox and Intel network cards and NVMe storage, but it's not clear if that's something that requires deeper PCI Express system support that's not implemented on the Pi.

Taco Availability and the CM3

Unlike many projects I've looked at this year, the Taco (board only) will be available for sale soon—by the end of this year—for under $100. And a full kit with the board, a nice metal case, and a CM4 will be available early next year for $200.

And in other interesting news, Radxa also sent me one of their CM3 boards with the Taco. It's basically a drop-in replacement for the CM4 that uses a higher-clocked Rockchip RK3566, and might solve some of the PCIe trouble I've had with the CM4.

Check out my video for a full review of the Taco and some more tidbits that I left out of this post for brevity's sake:

Comments

The exaggerated face at the bottom of this page is repulsive and an affront to your readers.

Let's not go full ad-hominem here—I understand that 'youtube face' / clickbaity thumbnails aren't desirable, but they do work, and that's why I use it (see two good videos on the phenomenon: Veritasium and LTT).

But that's also why I embedded the video at the bottom of the post. Just stop scrolling after the blog post written content if you don't want to see it ;)

Jeff, it still amazes me to see these sorts of shills out here with nothing better to say. What they fail to realize is what it really says about themselves... So, keep doing you!

I do work a (good) day job doing many of the things you're doing for fun, and even after 12-14 hour days I still look forward to learning from you and your research. It's truly invaluable to me. Moreover, I always find myself wondering what RSJ is up to today :)

Take care, buddy!

Fellow reader here, I was neither disgusted nor insulted by the still image from Mr Geerling’s YouTube video. I would not imply in any way that my feelings and or opinions in anyway reflect those of other readers. However I will express my disappointment when other readers feel they have a platform and right to speak for all other readers.

The number of homelab YouTubers I've seen doing this in crazy, initially I reached out to many and highlighted, their viewers are looking for competent presenters and informative videos. Gaumless photos really don't inspire confidence and actually damages their viewing figures. Initially I reached out to quite a few, most realised, but alas there are to many these days, there must be a Dummies Guide to be being a "Youtuber" that promotes it! :shrug:

Jeff if you read this, its just a suggestion about the silly photos and all, actual content is good :)

The difference is really 'narrow audience' versus 'wide audience with some of the potential narrow audience turned off by the thumbnail'. And for me, since I'm trying to make a living off educational content + open source, I have to choose the latter or go back to spending 90% of my time working a desk job.

It also limits you from having a top tier respectable channel. If I was a YouTuber I would be looking for other solutions for getting the numbers up that didn't involve monetizing my likeness in an exaggerated unlikeable way. I wonder if that's why many of the B-tier YouTubers look like soyjacks - because only those type of people without shame would sell their likeness out for little to nothing with no meaningful control over it other than to pull it.

There's a ton of content and quality channels out there that gets disregarded because the creator had monetization at the forefront and did things like that to maintain relevancy. If your content can't stand on merit and honest effort alone, your content or the delivery is likely inadequate.

Agreed. "Soy face" is obnoxious, and it's seriously annoying that it "works" to some extent. Of course, I remain unconvinced it actually increases repeat viewership; it probably attracts more clicks by idiots who don't care about the content as much, but I doubt many of them subscribe.

Ah, it's actually not a typo, I completely forgot to add a couple lines about Radxa's CM3, a CM4-form-factor SBC that Radxa is now producing (there's also Pine64's SOQuartz... waiting for that to ship).

> I think the ideal setup for a Pi-based storage device would be low-end SSDs
I thought I'd share something else I discovered WRT low-end SSDs. They use less power. I have found that the Micro Center house brand SSDs (Inland) work on a Pi 4 on the USB-3 port and using the UAS driver. No performance robbing work-arounds needed. I've tested the 120 and 240GB models with a Pi 4/4GB and Pi 4/8GB using a variety of disk benchmarks and no low voltage events were reported. I should probably buy a few more for the day they switch to a new manufacturer and power requirements exceed the capability of the Pi 4B USB ports.
Thanks for your support of the Raspberry Pi community.

Nice review Jeff !

How would this setup compare to a set with a NanoPi M4V2 and the 4xsata hat ?

Greetings from The Netherlands!

I've now tested the Inland 520GB SSD on a Pi 4B/8GB and so far it is looking good. I've only run one benchmark so far (`dd`) but it ran w/out any under-voltage warnings.

The thumbnail is obnoxious, repulsive, and NOT funny or amusing.

That is a total waste of 5 really good Samsung solid state drives. Buy another drive so that you have 6 drives, obtain the latest Avago/Broadcom/LSI RAID controller card and run as RAID 10 with an effective storage space of 24 TB. The torture of running RAID 5 on such an inadequate system is stunning.

So first off, whatever you're doing is working as I give zero crud about useless pi servers that can easily be out powered by a Celeron. Yet this was recommended to me.

Second, your "narrow" audience seems to be rude and entitled and seem to be extremely wealthy since none of them understand sometimes you have to do what you don't like to make money.

Third, nice flex on the 5gs of storage.

I wonder if the 2.5G Ethernet really performs better than the 1G one on a sustained load involving the disks. The 1G one is on another bus right?

Bro, the fck. If you had used ceph and multiple pi's for this money you would have a much more scalable and available system with more possible protocols, and storage capacity. Congrats to that kind of wasted money.

Hey Jeff. You are even more famous now - see "TheRegister" dot com's article from November 18th.

Wow. I wondered what you had spend your $5K on, then I see you have 48Tb of storage. How awesome is this.
Kind regards, Richard Mullins

Ever considered using btrfs in combination with SSD raid on SBC's? Instead of zfs.
I ask because i red somewhere that the best option to run multiple ssd's with btrfs was data=single and mdata=raid1. Or even mdata=raid1c3 or raid1c4 for more copies of mdata spread over the drives for more reliability.
If you use compression you can squeeze even more space out of it and should compensate the loss of space due to mdata in raid1(c3). And maybe more speed. I imagine that you can have the full 48TB of space.
The idea was that ssd's are pretty reliable and start to fail after a certain amount of writes. So the key is in saving writes. No raid means saving writes.
Btrfs is SSD and multiple disk aware, but let you decide which raid level or not.
The metadata (mdata) should be in raid1 so that it can reassemble data when it's corrupt.

Did you ever try the Radxa CM3? I currently have a XU4+Cloudshell2 as a NAS and am in need of an upgrade, especially for storage but also for speed, and that I think would be perfect. Of course the CM4 is probably still an upgrade, but why settle?