For years, I've been maintaining benchmarks for microSD cards on the Raspberry Pi, but I only spent a little time testing external USB storage, due to historic limitations with the Pi's USB 2.0 bus.
But the Pi 4 cleared away the limitations with a full-speed USB 3.0 bus offering much better performance, so I've done a lot of testing with USB boot, and with all the USB SSDs I had at my disposal. You can see some of those results in this blog post and video on booting a Pi 4 via USB.
After posting my tests concerning UASP support in USB SATA adapters, I got an email from Rob Logan mentioning the performance of some other types of drives he had with him. And he even offered to ship a few drives to me for comparisons!
There's also a video that accompanies this blog post, for the more visually-inclined:
So I took Rob up on the offer, and he sent me an XPG NVMe drive in a TDBT enclosure and an Arcanite AK58 USB 3.1 flash drive, touted by some as 'one of the fastest USB flash drives available'. I added a couple other drives to the test, pictured below:
Clockwise, from the Inateck case in the top middle:
- Inatech SATA enclosure w/ UASP and Kingston 120 GB SSD
- TDBT M.2 NVMe Enclosure and XPG SX6000 Lite 128GB
- Corsair Flash Voyager GTX 128GB flash drive
- Arcanite 128GB USB 3.1 flash drive
- SanDisk Ultra Flair 16GB USB 3.0 flash drive
- SanDisk Ultra Fit 128GB USB 3.0 flash drive
- (Inside the Raspberry Pi) Samsung Evo Plus 32GB microSD card
I ran benchmarks on every one of these drives, testing their sequential read and write performance with
dd (to test large file operations), as well as their random 4K read and write performance with
iozone (to test more general usage patterns when booting or running apps on a Pi).
So how did all these drives perform? I ran all the tests booting the Pi from the device that was being tested, and I also ran the same benchmarks on my fastest microSD card, a Samsung Evo Plus.
Sequential I/O performance
The sequential results show a huge gap between the SSDs and fast new USB flash drives and the cheaper older-generation flash drives and the microSD card.
From what I've found, it seems like most USB flash drives perform the same as a microSD card in a USB adapter:
The Arcanite does well here, but it does lag a little bit behind the SSDs and even the Corsair GTX.
Random I/O Performance
Random IO performance paints a more complex picture. The performance of the older flash drives remains abysmal, with even the microSD card trouncing them in random 4K write performance.
But the Arcanite also falls off quite a bit in comparison to the SSDs and the Corsair. And the XPG NMVe drive is at least twice as fast as every other option when it comes to random write performance.
So overall, it looks like a decent quality NVMe drive and USB enclosure is going to give the best overall performance. And the Corsair GTX is by far the fastest USB flash drive I've ever tested.
But there's one other test I wanted to do before closing the book on performance, and that's a 10 GB file copy over the network.
This test doesn't sustain the maximum sequential throughput for the drive, but it does take a long time and tests how well the different devices handle heat from constant write activity.
Large file copy over network
NOTE: This graph is a little misleading. I believe that the scale should be 'Mbps', but I need to go back to my data and verify what exactly was measured, and re-generate the graph below. The relative scale is correct, however:
This shows some interesting results. The Arcanite and the SanDisk Ultra Fit perform much worse for long-duration file copies than all the rest.
The Arcanite was only a tiny bit slower than the SSDs and the Corsair in the quick sequential tests, and the Ultra Fit was actually slightly faster than the Ultra Flair.
Why do they perform so much worse in this benchmark?
Well, I pulled out my Seek thermal camera and took a reading on the Ultra Fit:
I put some thermal tape on the tiny bit of metal that was exposed when it's plugged in, and the temperature measured over 60°C (140°F)!
The Arcanite's plastic body didn't measure quite so hot, but that's just the problem—plastic is a good thermal insulator, and that's why you often see it used in coolers. Metal, on the other hand, is good at dispersing heat, but you have to have enough area for the metal to disperse the heat, or the drive is going to get really hot.
The Arcanite's plastic body traps the heat inside, which leads to overheating, while the Ultra Fit's tiny profile doesn't leave enough room to dissipate heat.
Compare that to the much beefier Corsair GTX, which is larger (lots more surface area) and made of solid metal. Even under heavy write load, the Corsair kept its cool at 36°C (<100°F):
When you look at the benchmarks on the packaging, or even benchmarks posted to Amazon reviews from CrystalDiskMark, remember most of them don't reflect the true performance of these drives tested under real-world conditions.
SSDs and NVMe drives typically have a lot more surface area for heat dispersion, so they tend to perform more consistently since they can avoid overheating issues.
Most USB flash drives are designed more for compactness and convenience, and performance with general computing tasks or over long periods of time is usually more of an afterthought.
Usually, that is, unless we're talking about the Corsair—it's the first USB flash drive I've tested that compares favorably to USB SSDs!
But what about price?
What drive gives the most value? After all, the Corsair is $55, while the cheapest option, the SanDisk USB Fit, is about $7. When you go to buy a USB drive for your Pi, you want the best overall value, and you might want to sacrifice a little performance for a lot in savings!
So comparing all these drives (using the price for the 128 GB version), I came up with these two graphs:
This graph compares how many dollars you have to spend per MB/sec on a large file copy. The Arcanite gives the best bang for your buck (assuming you're not constantly writing to it all day), with double the value of the SSDs or the Corsair GTX. The Kingston SSD comes in second... while the microSD and older SanDisk flash drives are a pretty poor choice when it comes to value for sequential access.
In this graph, showing value for random IO activity, the microSD card fares much better, but the older SanDisk flash drives are still a terrible value.
But the XPG NVMe (with an enclosure) becomes the best value, with the Kingston SSD and Corsair GTX in close pursuit. The Arcanite is a bit of a laggard, but it's still respectable with performance similar to the microSD drive.
What do these values mean? Should you get the XPG drive or the Corsair for the best raw performance? Or should you stick with the Arcanite, which gives the best bang for the buck for sequential performance (in some conditions)?
Well, that's impossible for me to answer. If you're going to store large files on the drive, and use it as a media server or NAS, then an Arcanite might be the best option. If you want to run applications or use the Pi as a desktop, the NVMe and an enclosure is probably the best option.
Or, if you want the most portable Pi possible, using the least amount of space and energy, you might be willing to sacrifice a little more performance and stick with a reliable microSD card.
Or if you need even more performance, you might want to look at a different single board computer that offers built-in SATA or NVMe support.
In the end, it's really up to how you want to use your Raspberry Pi, I just hope this helps make your decision a little easier.
I put all the raw performance data and benchmarks used in this issue on the Raspberry Pi Dramble issue tracker.
Also, Rob sent me many more results for other drives he tested, and was generous enough to allow me to share some of those results. They are summarized in the table below:
|Drive||hdparm - sequential||4k rand read||4k rand write|
|Inland Premium 256GB NVMe 3.0 x4 SSD||308.46||19.41||31.80|
|SanDisk 128GB Extreme Pro USB 3.1 Flash Drive||221.08||10.96||11.51|
|eMMC 5.1 module in USB 3.1 adapter||113.11||8.28||4.84|
|Netac Z8 250GB SM2258XT||299.05||20.48||31.47|
|Inland Pro 120GB PS3111-S11||313.05||18.59||24.66|
|eVtran 128GB ASM1153E||334.49||19.14||29.40|
And he sent over a few pictures of some of the drives he tested, one of which I'll share here, for your reference:
Hi Jeff, great analysis as ever - thank you. For some time now I have been intrigued by the possibility of creating the best value computation platform by booting from a very cheap source of memory (microSD) and then running an application or even the OS from a RAM disk on one of the more RAM-equipped Pi 4 models. I have seen two approaches to it so far. Firstly there's the more involved image generation using something like Alistair Buxton's rpi-ramdisk project (https://github.com/ali1234/rpi-ramdisk). The second, and perhaps more seductive, approach would be more like the one taken by HBFS Robotics where they run Chromium from a RAM disk (https://hbfsrobotics.com/blog/accelerating-raspberry-pi-4-applications-…). The trouble I have is that I can't see what would be involved in running another application (say, Octave) from a RAM disk because it doesn't have the kinds of files or installation that HBFS Robotics take advantage of in their example. I think this could be a real step forward in the value for money proposition of distributed computing but I'd love to know what you think.
Do you expect that the performance of the USB drives would improve if there was some active cooling around the 'stick'? In particular the Ultra Fit that reached some 60 degrees, might benefit from some indirect fan airflow.
Performance probably wouldn't improve, but consistency would (e.g. the Fit would work at it's highest speed for a huge file copy and not slow down a lot after a few minutes).
No, the Ultra Fit and Ultra Flair I have, do not get especially hot on the exterior, yet still slow down to a pathetic <15MB/s on large sequential writes during extended writes where they heat up internally, and in some cases even hiccup and freeze for a bit before continuing.
Some other people have observed their exterior getting hotter than mine from same generation so I can only assume the drive have some sort of linear regulator and the heat difference depends on the voltage of the USB port it is used on, despite the theory that they should all be dead on 5.0V, which we know is not always the case in reality.
At one point I considered modifying a hub to put a diode in series on the USB supply, to see if it would tolerate the lower voltage and if running cooler made it keep higher sustained read speeds, but since these are budget priced and nothing is going to make them perform like an Arcanite, or Sandisk Extreme Pro, I never bothered testing it.
Very interesting to see how different USB Drives compare. I don't understand the data for the Network transfer test. 232 MB/s is a lot to transfer over the Raspberry Pi's Gigabit Ethernet port. Are you sure it's MB and not Megabit? If so the numbers make sense.
With my own WD Green Internal SSD, Green, 240GB I measured 174 Mbit/s. No heating issue. Just Luke warm in a ORICO NGFF case.
Oh, yikes! I got that benchmark graphic all kinds of messed up. I'm not even sure what I did, but I had measured the time in seconds and the average rate in MB/s, so somewhere I must've screwed up the two or accidentally added a conversion to Mbps, as you say.
I will try to figure out what happened and get that benchmark updated.
Agreed some of those are too high to be mega*bytes* over GigE. But some seem too low for it to be mega*bits*. They look a little like expected mega*bytes* per second, but without network.
Would you be willing to switch to fio (https://github.com/axboe/fio ) for your testing (but note it is MUCH more complex)? Using dd can have serious drawbacks (for example it is common to use if=/dev/zero and writing zero data can lead to unrealistic speeds due to zero detection/compression by something in your storage stack, see https://github.com/axboe/fio/issues/711 )...
you should give Samsungs T5 or T7 a try, hdparm tests giving 400+ MB read performance, I discovered those last year and have several, appears to be NVMe in a box, crazy fast storage. HTH
I just read a few Amazon reviews of the Corsair stick. They say the write performance comes from a 1.5 GB write cache, and data compression. Once the cache is exhausted, writing data becomes much slower. Especially when the data cannot be compressed much.
Thanks for the extensive comparison! I'm only getting started with SBCs and found this very helpful.
Hi Jeff, I have both the RPi 4 and my computer connected to a Gigabit router and I'm uploading a 4GB test file from the computer to the RPi' SSD via SSH at roughly 12 MB/s (byte). The SSD is connected to the USB 3 via a StarTech USB3S2SAT3CB cable.
A friend of mine has a very similar issue getting a little better, roughly 15 MB/s. Same USB/SATA cable different SSD (Corsais MX500 1TB). Thoughts?
Any chance you might share the exact dd/hdparm/iozone commands you used to benchmark? I'm testing a few of my own and would like to compare results to yours. Happy to post my results. Thank you.
See the benchmarks here (https://github.com/geerlingguy/raspberry-pi-dramble/tree/master/setup/benchmarks).
Awesome, thanks. Sent you a PR to fix a missing script path.
Initial result with Western Digital 1TB WD Blue 3D NAND Internal PC SSD - WDS100T2B0B. Used with Argon ONE M.2 Case for Raspberry Pi 4 and the arm64 version of Raspberry Pi OS, kernel 5.10.7-v8+.
Running hdparm test...
Timing buffered disk reads: 256 MB in 0.90 seconds = 285.21 MB/sec
Running dd test...
51200+0 records in
51200+0 records out
419430400 bytes (419 MB, 400 MiB) copied, 2.98124 s, 141 MB/s
Not too bad on hdparm but a good clip slower than you saw for the dd test.
Here's another test, this one run against a Samsung SSD 860 EVO M.2 500GB drive. Tad slower than the Western Digital but not far off.
Timing buffered disk reads: 256 MB in 0.91 seconds = 282.20 MB/sec
Running dd test...
51200+0 records in
51200+0 records out
419430400 bytes (419 MB, 400 MiB) copied, 3.06317 s, 137 MB/s
I have a PI 4 acting as a NAS with two WD passport drives. I plugged a power hub into the top USB 3.0 socket and both drives into that. It works fine and happy, what I do not understand is whether I should have two power hubs and run each disk drive off each USB sockets the PI.
Thanks for the speed test! I am using my Raspberry Pi 4B with Docker now, and am running about 14 containers. It is slow to start up all the containers when I reboot (load is at 17+ after a few min, then drops to under 1 after about 15 minutes). So I am looking for a faster storage medium than the Samsung EVO microSD card. Going to try booting from a USB thumb drive, and if that doesn't help, will try a SSD in a USB 3.0 enclosure. :)
Replying to my own post! Thanks Jeff; great guide. I used some of the guide at Tom's Hardware to prepare my pi 4B to boot from USB: https://www.tomshardware.com/how-to/boot-raspberry-pi-4-usb. I have invested way too much time into my pi config to start over from scratch, so using the guide at Tom's Hardware gave me the necessary steps to update my existing Pi OS (I only followed the steps "sudo apt update" and "sudo apt full-upgrade" as I didn't want to risk running "sudo rpi-update" and breaking firmware, if it wasn't needed; fortunately, I didn't need to update firmware for this to work on my pi). I didn't follow any of the remaining steps there, LOL! Next I powered off my pi, then used win32diskimager on my Windows 10 PC to take an image of my pi's microSD card.
Then I used your guide here (picking a fast USB storage option) & James A. Chambers's blog (https://jamesachambers.com/raspberry-pi-4-usb-boot-config-guide-for-ssd…) to confirm the USB 3.0->SATA enclosure/cable I bought would work with the pi4 (ironically, I had already ordered a Sabrent UC-UASP, which on the bad list on James's blog, so I was able to cancel that order before it even shipped!). I decided to use the drive & cable he recommended there (StarTech 2.5" SATA to USB adapter & Kingston A400 SSD/128GB). Your excellent blog article also confirmed the Kingston A400 is a solid performer, so I ordered those confidently yesterday.
While waiting for that, as a test I wrote my 64GB microSD card's pi image to a PNY 128GB Turbo USB3.0 thumb drive (https://www.amazon.com/PNY-Turbo-128GB-Flash-Drive/dp/B00FE2N1WS). I saw a good improvement in speed, so was looking forward to the SSD. When it came earlier today, I wrote the 64GB microSD card image to the Kingston 128GB SSD, hooked it up and expanded the root filesystem (using "sudo raspi-config", then advanced options, you'll find it in there). Rebooted the pi to confirm all space was now allocated for the SSD, and then followed your great guide on setting up TRIM (https://www.jeffgeerling.com/blog/2020/enabling-trim-on-external-ssd-on…) so the hard drive will be maintained well. Thanks a bunch.
To help others considering this project, I'll close my comments with some speed comparisons. For reference, I'm running a pi 4b, 4GB. I use docker heavily and am running 15 containers now! Based on this I hope others are encouraged to move to SSD boot for their pi. The pi feels a lot faster and it is so much more responsive than it was on a microSD card. Thanks for all the great info!
Load Leveling: How long after boot-up for system to have a load level <1.00?
- 15 minutes on microSD
- 10 minutes on USB thumb drive
- 4 minutes on SSD
Docker Performance (running "docker pull linuxserver/ombi", ~100MB [already had other linuxserver images so didn't have to download more than 100MB]
- 3 minutes on microSD
- 1 minute 30 seconds on USB thumb drive
- 27 seconds on SSD!!!!! WOW!
Speed test ("hdparm -t --direct [DEVICE]")
- microSD: 40.03 MB/sec
- USB: 99.92 MB/sec
- SSD: 311.69 MB/sec
Thanks all for the great info.
I'm new at Raspberry Pi and I have got it running for Plex and SteamLink.
I'm looking at a SSD to put my media files on to share over the network. Not knowing I will connect it to a network router/switch or to the Raspberry. Knowing that I could install Raspbian on the SSD in the second case, what would you recommend ?
Thanks for taking the time and sharing all this information, I really appreciate it!
I picked up the Arcanite 128GB USB 3.1 flash drive to test as a boot or storage drive on my Pi 4. I am beginning to think they have changed the product as my speed tests are not good. I have it mounted, but not booting from it. The partition is ext4.
My machine is fully updated, running Debian 10 distro from Raspberry Pi. I am not running lite. This is better than my Sandisk Fit 3.2 and way better than the micro SD... but not what I expected to see.
Do you have it plugged into one of the blue USB 3.0 ports on the Pi, or one of the black USB 2.0 ports? The USB 2.0 ports will max out around 35 MB/sec.
Also, even if plugged into USB 3.0, there could be problems on the bus—try running
lsusb -tto see if the drive is mounting at 5000 Mbps ('superspeed') or just 400 Mbps (USB 2.0).
So, yes I was in one of the blue ports. The drive was really fast on Windows 10. So, what is a bus problem? I have an ssd and Argon 1 M.2 bottom coming in and wanted to do a little testing before hand.
lsusb -t shows the drive in question is running 480M, but my Sandisk Fit is running 5000M. If I am reading the output correctly. I have also tried removing the UPS USB2.0 connection and rebooting... still the same.
I reversed the drives and rebooted. Problem followed:
I have the same issue with Arcanite 128GB bought from Amazon.es. No matter what I do, it's always in 480M. Also running debian 10 64 bit on my raspberry pi 4.
I have a similar problem. I have 2 similar Orico 2-drive SATA-USB adapters (JMS551 chipset, also known as Sharkoon SATA QuickPort Duo; one of them has RAID option, but I don't use it), connected to the 2 USB3 ports of the RPi4. One of them gets detected as SuperSpeed (USB3 speeds: 5000M), and the other one as high-speed (USB2 speeds: 480M).
It's like the RPi4 thinks an USB2 device is attached to the fourth USB2 port (shared with the second USB3 port).
i've red some of your blogs about SSD USB Boot for raspberry pi 4. Based on your advices i've order a sandisk extreme pro SSD flash drive with 128gb of storage. I can't seem to get USB boot to work.
I run raspbian (latest version) headless. I've done sudo rpi-update and when i check bootloader version it has the latest/current version. In raspi-config I've set boot order to first use USB.
But i get this error:
"usb-MSD boot requires the latest software. Get the latest software from https://www.raspberrypi.org/downloads
When I search the internet about solutions it only gives me solutions of things I have already tried. Wondering if you have any suggestions.
BTW: the Sandiks is flashed using Balena etcher with a DD full backup zip image from my current SD card (which i run automatically every week, and yes I've made another backup after doing all the things like update rpi).
Nevermind, i found the problem. I was running NOOBS and that really bit me in the a*s. But worked around it with this scripting: https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=196778&hilit=op…
Incredible points. Outstanding arguments. Keep up the amazing work.
Did you have to enable anything to get the nvme disk to work over USB? I've been trying to get this to work for a while now. Tried multiple drives and multiple adapters and power supplies but all of them end up with the USB device being recognized but unusable. I "think" dmesg is logging the issue but I've not found anything to fix it.
Hah! Same problem here! I tried two enclosures (Orico M2PV-C3 RTL9210b and SAN ZANG DMP2-10G RTL9210b) with a XPG SX6000 Lite and none is detecting it, while a Samsung 970 EVO works fine. I tried various firmware versions without success. The XPG NVMe works fine in a JMS583 case, though.
Orico claims the SX6000 Lite works on their website, even though they have never tested it....
Buffalo sells an external USB 3 HDD enclosure with a 1GB cache.
A blog post testing one, with both a HDD and with the HDD replaced with a SATA SSD would make interesting reading..!?
(It would be nice if there was a way to sign up for reply notifications and relavant new posts here..?)