Raspberry Pi 2.5 Gbps 16 TB OMV NAS - Setup and Performance

Raspberry Pi CM4 2.5 Gbps NAS build

Last week, I posted about the Pi NAS hardware build, and compared setting up an off-the-shelf ASUSTOR Lockerstor 4 to the same thing, but with a Raspberry Pi Compute Module 4 an IO Board.

Some people wondered why not just use a Raspberry Pi 4 model B with USB hard drives, and there are a few reasons:

  1. The model B's USB 3.0 ports add about a 10% overhead to SATA operations, which is more apparent over small file operations.
  2. You can't RAID together USB 3.0 drives using OMV or other NAS-specific software, since RAID over USB is not very reliable.
  3. The Compute Module 4 IO Board's PCI Express slot lets me plug in multiple cards, like a 2.5 Gbps Rosewill NIC and a 5-port IOCrest SATA controller.

Video

There is a video to go along with this blog post, that goes into much more depth on the full setup process on the ASUSTOR Lockerstor 4 and OMV on the Raspberry Pi:

Open Media Vault

With that out of the way, and the hardware set up and running, I went to install Open Media Vault, and found the process to be pretty painless; I just had to run one command to run their install script for Pi OS:

wget -O - https://github.com/OpenMediaVault-Plugin-Developers/installScript/raw/master/install | sudo bash

After that completed, I used sudo nmap -sn 10.0.100.0/24 | grep pi to find the Pi's new IP address (OMV usually changes the IP as part of its setup process), and then logged into the web management UI in my browser (default credentials admin/openmediavault):

Open Media Vault browser login screen

The UI is straightforward and utilitarian. After having set up the Lockerstor 4, which uses ADM, a custom Linux distro built on top of Busybox, it felt a little less polished having to do things like manage filesystems and click through 3-5 different screens to set up a new network share in OMV.

But it's definitely manageable, and a one-time task; the only real annoyance was the fact that after I clicked 'OK' for a settings change, there would be a bar that appeared at the top of the screen 3-5 seconds later telling me I also had to apply the change for it to take effect. I'm not sure why it wouldn't just apply it when I click 'OK' or 'Save' in the first place:

OMV Do you really want to apply configuration changes?

Performance

I decided to stick with Samba (SMB) shares and Windows 10 for the performance testing, mostly because I have a PC tied into my 10 Gbps network with a DAC sitting in my office, and its easier to run uninterrupted benchmarks on it with nothing else running than it is to do so on my main Mac workstation.

File copy in Windows 10 from PC to Raspberry Pi NAS

Copies to the Pi NAS, with three Seagate IronWolf 8TB NAS drives in RAID 5, measured around 95 MB/sec. Copies from the Pi clocked in around 193 MB/sec.

But I noticed two things when I ran atop during the copy:

  1. When writing to the RAID 5 array, after the RAM cache would fill up (after about 0.5 GB of data copied), the single-threaded smb process would be around 70% CPU, one core would hit around 50% interrupts (network packets seem to only be able to go through one CPU core on the current Pi generation), and the CPU in general would run between 40-70% CPU.
  2. When reading from the array, interrupts on the single core would hit 99% and the copy would top out around 1.7 Gbps (about 200 MB/sec).

Using Jumbo Frames (9000 MTU) could mitigate the network packet overhead issue, but I also remember from previous testing that overclocking the CPU also produced a noticeable speed gain.

Overclocked to 2.0 GHz, the Raspberry Pi was able to put through about 100 MB/sec write speeds, and 200 MB/sec read speeds. Not too bad, but also a bit less than the ASUSTOR, which has a faster Intel CPU inside, and much more PCI Express bandwidth to go around. Even without SSD caching, the ASUSTOR wrote to the drives more than twice as fast as the Pi:

Samba network file copy performance - ASUSTOR Lockerstor 4 vs Raspberry Pi NAS 2.5 Gbps

I also tried a RAID 0 array on the Pi, to take out the overhead associated with RAID parity calculations. This seemed to go well, as a write to the Pi was solid at 230 MB/sec... but then the Pi locked up partway through the copy. This happened every time I tried copying to and from the RAID 0 array, and I'm still trying to figure out why.

I didn't have four similar drives to test out RAID 10, but I may test that out when I get a chance.

Conclusion

As I mentioned in the video, I think for most people who need network storage and don't need the extreme customization (or performance, if building a server with a better processor/ECC RAM), an out-of-the-box NAS like the Lockerstor (or a Synology, QNAP, etc.) is the best option.

Modern NASes like the Lockerstor I tested are performant, well-supported, and easy to set up and configure.

If you don't need the faster networking, though, there is one Pi-based option I'm keeping my eye on, the Wiretrustee SATA, a CM4-based board that will incorporate a SATA controller and power for the drives.

Wiretrustee SATA inside case 2.5 in hard drives

Using it with 2.5" SATA drives would make for a very compact, fast-enough 1 Gbps NAS.

But if you want more performance, want ECC memory, or if you need something like FreeNAS or ZFS, neither of which run that well on a Raspberry Pi, you'll have to stick to a custom build or higher end server for now.

Comments

Why no price comparison? Because it would cost 4-5 times more than a x86 diy nas with omv. A factory solution would be still cheaper at least two time.

Thanks for the video! I've been using omv on a pi and a x86 home built pc and always been curious what hw and cost it would be to have a proper pi raid config. Good to finally know. It seems if you don't use usb hdds the diy x86 pc is cheapest by a wide margin.

OMV suck!!!, i have Raspberry 4 b 4GB NAS, using SAMBA and miniDNLA , and im using 4x WD 2T RED drives with usb to sata (and usb3.1 hub), and it works great!!!!! (im using single power supply for all of that)

I had hoped to see "an apple to apple" comparison to a NAS with ARM processor (i.e. QNAP TS-231P3-4G).

With an upcoming new release 6 of OMV, may I suggest to replace all references of "OMV" with "OMV5"?
Same with "Open Media Vault" to "openmediavault"

OMV release 6 is not yet available but planned for Q2/2021.
The UI will see a major overhaul, hence flagging the version reviewed with "OMV5" will avoid confusions :)

This seems redundant. My exos drive is capable of 2.5gbit itself. Why use raid when you could just use the 2.5gbit adapter?

I'm fully saturating my gigabit line before even touching half the drive spec.

RAID can be helpful for other aspects, like redundancy, or in some cases even latency improvement for either reads or writes (depending on RAID level) vs. a single drive. It's not just for increasing throughput (and indeed, in the case of RAID 5 on the Pi at least, it was slower than a single disk!).

Hey Jeff!

Regarding needing more than a pi for a ZFS solution, I have been running a pi 4 as a ZFS NAS + backup solution for a year with my computers and servers backing up to it (zfs send) daily and it replicating to another pi at my mom's place every night. (I am in fact in the process of writing an extensive blog post about it; until then, feel free to check out the replication script I have been maintaining: https://gbyte.dev/blog/simple-zfs-snapshotting-replicating-backup-rotat…).

Sure, USB is not the best interface for that, but I have never had any performance issues and both the 4GB and the 8GB pi versions are capable of handling ZFS well.

For some context, we met in my Drupal module's (simple_sitemap) queue a couple of times. I enjoy your content and it inspired me once or twice for my self hosting projects.

Good to know! I am especially interested in more testing/experience with the 8GB Pi for ZFS, seems like it may be an adequate solution for all but more advanced use cases.

Greetings Jeff, thank for all the videos, great source of information.

I have a Pi4 8GB and installed OMV, I planned running anything/everything I needed in contairnes with it, right now I'm isntalling piHole as of the instructions in this site; https://www.wundertech.net/how-to-set-up-pi-hole-on-openmediavault/ But I'm very keen of running your internet pi setup. Any way that can be done through containers?