Raspberry Pi USB Boot - UASP, TRIM, and performance

In the past few weeks, I reviewed USB drive performance on the Raspberry Pi 4, and the importance of UASP support for USB drive performance.

Both posts generated great discussion, and there were three things I wanted to cover in this follow-up, namely:

  1. Which drives support UASP
  2. Real-world performance benchmarks
  3. TRIM support

For reference, here are all the products I'm testing in this post (product links are to their Amazon product page, starting from top middle, clockwise):

USB Performance testing - SATA SSD, NVMe, and Flash drives

Video Version

There is also a video version of this blog post: view video on YouTube.

UASP support

First, in the last post, I completely forgot to discuss which of the USB drives I tested supported UASP, and which ones didn't.

For a refresher, UASP lets the Raspberry Pi communicate with the drive using the SCSI protocol, which is up to twice as fast for file copies and disk performance as the older 'USB mass storage' protocol.

You can check if your own drive supports UASP with the command lsusb -t. If the output shows 'uas', it supports it out of the box. If it shows 'usb-storage', it doesn't.

So here's a quick graph showing which drives support UASP.

UASP Device
Inatech enclosure with Kingston 120GB SSD
TDBT M.2 enclosure with XPG SX6000 128GB NVMe
Corsair Flash Voyager GTX 128GB
Arcanite 128GB USB 3.1 flash drive
SanDisk Ultra Flair 16GB USB 3.0 flash drive
SanDisk Ultra Fit 128GB USB 3.0 flash drive
Samsung Evo+ 32GB microSD

It looks like all the fastest drives I benchmarked support it, while all the slowest ones don't. No huge surprise there; the faster drives use better chipsets that are built for SSD performance.

The Arcanite is an outlier, though. It doesn't support UASP, but it does perform very well for its price. And it sometimes behaves a little like an SSD—I'll talk about that in a bit.

Performance

For the second thing I wanted to cover, I was prompted by none other than Gordon Hollingworth, the Director of Engineering at Raspberry Pi, in his Twitter post. He said:

First, I'll test boot time performance.

Boot Performance

The most important thing when measuring boot time is to find a way to compare different devices using an objective measure. That is, you don't want to sit around with a stopwatch and try to time 'from plug-in to desktop appearing' boot times. That could be helpful sometimes, but it's not very objective.

Instead, I use a built-in tool in Raspberry Pi OS, called systemd-analyze. It's a tool that helps you analyze the system manager, and by default, if you just run that command, it will output the boot time. How easy is that?!

So I did that, three times for each drive, and averaged the results:

USB boot performance for Raspberry Pi 4

All the drives performed pretty well, though the non-UASP drives did tend to be slower, with the strange exception of the SanDisk Ultra Flair, which punched above its weight class. The slowest by far was the SanDisk Ultra Fit, which I mentioned in the previous post has a tendency to overheat and slow way down.

But there are two important caveats to these boot time numbers:

First, I booted with the August 2020 version of Rasbperry Pi OS, and followed the directions in this blog post to configure the USB drive to be able to boot the Pi.

Second, it seems like the Pi already optimized its boot performance really well. The first boot was always a bit slower, but subsequent boots took around 15-17 seconds, on all the USB drives I tested.

The biggest difference was that first boot was much faster on the faster SSDs and NVMe drive, and a bit slower on the cheap flash drives and the microSD card.

Browser launch time

The other thing Gordon mentioned in his tweet was testing the web browser launch time from the command line.

It actually took a bit of doing, figuring out a way to launch Chromium from the command line, load a web page, and then quit it, and get an accurate time measurement of the process. Just using chromium-browser [URL here] didn't cut it, because that launched the browser, and the process wouldn't exit until I manually quit the browser.

After asking on Twitter about what others might do, I found a neat Node.js utility called 'puppeteer', which I could use to do it all automatically, and then I used the time utility in Linux to benchmark the process three times for each drive. I described how I did this benchmark in detail in this post: Testing how long it takes Chromium to open, load a web page, and quit on Debian.

Here are the results:

USB drive web browser performance for Raspberry Pi 4

The difference really isn't that big. Definitely not as big as I thought it would be. The faster drives still open Chromium a tiny bit faster—especially on first launch—but only a tiny bit!

I tested all of these Pis over a VNC connection, with the resolution set in raspi-config to 1280x720. And just like the boot times, the first launch of the browser after a reboot would take a bit longer than the 2nd, 3rd, 4th, or 5th launches.

I think this just means the caching mechanisms in Linux are good at normalizing performance for even very slow boot volumes, as long as you have enough system memory.

Once booted for the second time and after quitting and restarting Chrome, the difference in common UI tasks between the slowest drive and the NVMe drive was almost imperceptible.

There are some things, especially when you're doing upgrades, installing software, writing files, or working on large projects (like Drupal websites I maintain) where the difference is more apparent.

To test that, I installed php7.3-cli on each of the drives, and checked how long that took:

USB drive PHP installation performance on Raspberry Pi 4

And... as with the other performance tests, this one is not the most consistent. I ran it a couple times on some of the drives, re-flashing the entire drive between runs, and the standard deviation—the variance between runs—was usually around 20%, so take these results with a grain of salt.

Generally speaking, the faster drives did do better, but it was hard to get exact numbers when benchmarking real-world workloads.

TRIM support

So finally, in a Hacker News thread, user Legogris asked about TRIM, saying:

Are you aware if the GTX and Arcanite support TRIM? That definitely makes a difference when considering OS storage.

And why would Legogris be interested in TRIM support? Well, the short answer is with SSDs, when little bits of data are deleted, and new data needs to be written to where those old deleted bits were, the drive can slow down and also do more work than it should have to.

Defragmenting a Windows 95 Hard Drive

This is a really simple answer, but basically think of it like 'automatic defragmentation' for an SSD. I don't know if you've ever had the honor of sitting in front of an old Windows computer watching it defragment your 80 MB IDE drive for hours on end, but it's kind of like that, at warp speed. TRIM doesn't do defragmentation, technically, but it's similar in that it lets your SSD perform its best through some automatic cleanup processes.

The hard thing is, you have to have TRIM support in both your operating system—in our case, Raspberry Pi OS (which does support TRIM)—and in the drive controller's firmware.

There are a few ways to check for TRIM support, like running the fstrim command:

sudo fstrim -v /

If it says the discard operation is not supported, then TRIM isn't currently working for your drive.

You can also run the lsblk command:

lsblk -D

If the DISC-MAX value is 0B, then, again, TRIM isn't currently working for your drive.

Many adapters will work with TRIM after you follow a special process to change their 'provisioning mode'.

I have a separate blog post with details on how to enable TRIM support if your firmware supports it but it's not enabled by default: Enabling TRIM on an external SSD on a Raspberry Pi.

Some drive controllers may also need a firmware update to enable TRIM support, so check on your drive manufacturer's website.

Here are the results for all the drives I tested:

Device TRIM OOTB TRIM support in Firmware
Inatech enclosure with Kingston 120GB SSD
TDBT M.2 enclosure with XPG SX6000 128GB NVMe
Corsair Flash Voyager GTX 128GB
Arcanite 128GB USB 3.1 flash drive ⁉️
SanDisk Ultra Flair 16GB USB 3.0 flash drive
SanDisk Ultra Fit 128GB USB 3.0 flash drive
Samsung Evo+ 32GB microSD

Surprisingly, the Inatech enclosure didn't seem to have any TRIM support, while the TDBT NVMe enclosure and the Corsair did.

Also, and this is something I never thought about—the Raspberry Pi actually supports TRIM out of the box for internal microSD cards!

But the most alarming result is that the Arcanite firmware indicated TRIM support, but when I followed the process to change the provisioning mode and ran fstrim, the drive failed spectacularly, and now I can't even mount or initialize the thing on any computer!

So... the Arcanite firmware may be from an SSD, but the flash memory itself seemed to not take well to the fstrim command. Either that, or I had one defective unit!

Summary

In the end, I found there are a lot of different traits, positive and negative, on all the drives I tested. If you just need a drive to store large files, I still think the Arcanite is the best overall value, even though it doesn't support UASP, and it kind of explodes if you try enabling TRIM.

And if you're after raw performance, an NVMe inside an enclosure is going to give the best bang for your buck, as well as—at least in the case of the TDBT enclosure I tested—full TRIM and UASP support.

In the end, if you have any USB 3.0 drive, outside of cheap flash drives, it's probably going to perform as well as or better than a microSD card in a Raspberry Pi 4.

Comments

It would be cool if the benchmark covers throughput, latency and CPU usage on heavy io usage.

I’m new to this and bought a system from Geekworm to connect an HDD through usb 3. The HDD is a barracuda 4tb. Will this work? If so could you tell me how to configure it.

You forgot to mention some important specs of your hardware, so we have no way to tell you whether or not it will work.

First of all, what kind of computer are you plugging this hard drive into? A Raspberry Pi? If so, what model? This matters because different models support different USB standards (usb 2.0, usb 3.0) The USB standards dictate how much current (amount of electricity) is allowed to pass through the cable.

Second of all, what kind of hard drive is it? is it 2.5 inches wide (like a laptop harddrive) or 3.5 inches wide (like a desktop harddrive). This also matters because it determines how much current the hard drive requires.

Finally, does the "system from Geekworm to connect an HDD through usb 3" have its own power supply, or does it take power from the Pi? This also matters because it specifies how much current the USB to SATA adapter is able to supply to the harddrive.

Now, since I don't know any of these specs for your hardware, I will have to answer your question for all possible combinations of specs.

Raspberry Pi 3 with a USB to SATA adapter that does NOT have its own power supply: It won't work, the Pi 3 cannot supply enough current for a HDD on its own regardless of the size of the HDD.

Raspberry Pi 3 OR Raspberry Pi 4 with a USB to SATA adapter that DOES have its own power supply outside of the Pi: It should work just fine with any hard drive, 2.5" or 3.5".

Raspberry Pi 4 with a USB to SATA adapter that does NOT have its own power supply: It will work, but only with 2.5" harddrives. The Pi 4 supports USB 3.0 which allows it to provide more current to the harddrive. But still not enough for a 3.5" drive.

Hey jeff do you think TRIM support depends more on the drive being used (for sata drives) or the adapter? Maybe I will do some of my own tests and report back

Hi Jeff,

I just got a Pi 4 2GB, updated everything, even the EEPROM. I also got an XPG SX8200 256GB and and XPG SX6000 128GB to play with in my TDBT case. (I was inspired by you!) :)

I just wanted to mention, that with the SX8200, I needed a powered USB 3.0 hub in order for the Pi to recognize it. However, for the SX6000, the hub was not necessary. Not sure why this is.

The really weird bit is that while the SX6000 was pulling about 0.5 amp, the SX8200 was only pulling about 0.4. The only thing I can think of is if there is some kind of power-on current spike that the Pi does not like with the SX8200.

Anyway, just wanted to share my finding in case you or others are thinking "XPG but newer/bigger!" :)

Hope it helps,
Krys

First, hello from Spain, and thank you for yours instructables.
I connected a usb 3.0 Samsung, then lsusb - t, and:
'Driver=usbhid'
I xpected the magic uas.
What should I think about it?
Thank you again.

Hi Jeff,

Watched your video and buyed Arcanite usb drive, but i have problems with speed. Can get more when 35 mbs . Its same speed on usb2 and usb 3 socket. Tryed it on other pc with fedora 33 distro , showing 400mbs. Also tryed quirks change, but same speeed after restar. Dont know what to do next. I have connect HDD Seagate Barracuda Pro 1TB 2.5" 7200RPM SATAIII ST1000LM049 and its working perfect about 150-180mbs(it is NAs drive), but i want run sytem from Arcanite usb drive, its even slower then my sd card samsung evo 32gb, which get about 45mbs.

For curious people, UAS can run on USB 2 controllers too if driver/firmware supports it. It's not the case for RPI3 because https://github.com/raspberrypi/linux/issues/875 (TL:DR at [1])

[root@rpi-aarch64-fedora ~]# dmesg | egrep '(dwc2_hsotg|UAS|Machine\ model)'
[ 0.000000] Machine model: Raspberry Pi 3 Model B Rev 1.2
[ 5.988315] usb usb1: Manufacturer: Linux 5.8.15-301.fc33.aarch64 dwc2_hsotg
[ 2475.356205] usb 1-1.4: The driver for the USB controller dwc2_hsotg does not support scatter-gather which is
[ 2475.356218] usb 1-1.4: required by the UAS driver. Please try an other USB controller if you wish to use UAS.
[root@rpi-aarch64-fedora ~]#

[1]: Comment below on bug #875
"The limitation is that there is a 1:1 mapping between a USB transfer and a single, contiguous DMA transfer in hardware. To implement scatter-gather, you'd have to rewrite significant chunks of the driver to split out a SG URB's individual transfer fragments into separate QTDs, then have another method of joining up all the completions once the hardware's done the sub-transfers.

This would a) require significant software engineering time and b) add a ton of overhead to an already bloat-heavy driver. In short, we've not got a case where SG is an absolute requirement therefore a driver rewrite is not going to happen.

UAS is now re-enabled because it adds a bunch of quirks for newer usb-storage devices. The module is loaded but as there's no SG support, UAS devices are accessed as mass-storage class devices."

Just wanted to mention that the StarTech USB3S2SAT3CB adapter supports UASP and TRIM out of the box on the Raspberry Pi 4 (confirmed via your suggested commands).

Which SATA 2.5" hard drive was used? Does it stay mounted 24/7?

I have a USB/SATA that mounts and works OK -but after a few hours an error occurs (dmesg log) and the drive unmounts and is no longer active.

Does the type of SSD drive matter for UASP support?
Should any SATA 2.5" SSD work in the Inatech enclosure - or only the Kingston?

My problem was with the USB hub powering the SSD.
I created a breakout cable to power SSD with separate +5V supply and ran USB signals direct.
Now is stable and not dropping data.

Wait, I'm confused now, I thought UASP was on the chip of the usb-2-sata adapter, rather than the drive itself? Or is it both now too?

I was going to buy a star-tech as my current adapter only does USB storage mode, but if my spare SSDs need to be UASP too then I'll have to throw that idea out of the window as they're too old (pre-2015) to take advantage of it, its a Crucial RealSSD C300 and a Samsung 850 Pro. However googling for UASP supported SSDs doesn't return anything relevant....

Very interesting, thanks for shraing your information.
I struggle with an old raspberry pi 1b and lots of io-wait on USB IO.
Interestingly that differs a lot with the USB-stick I used.
So if you do read/write performance tests can you have an eye on CPU usage, especially io-wait when stressing throughput.
Feels like the old ide vs scsi debate I had with friend 30 ears ago. Throughput was about the same put CPU usage was way better with scsi.

Your Arconite drive may not be as dead as you thought.

In Linux you can use the hdparm command to interrogate the drive - and if it responds with sensible data - do a formal erase on it.

Viz: https://grok.lsu.edu/article.aspx?articleid=16716
"Erasing SATA Drives by using the Linux hdparm Utility"

I have had drives that - due to various stupidities on my part - were rendered unusable. Running a firmware-level security erase basically returned the drives to the out-of-box condition, ready for partitioning and formatting.

It's worth a try.

I tried to enable TRIM on my Samsung 2 Tb T5 SSD.
When I do this:
echo 2147450880 > /sys/block/sda/queue/discard_max_bytes
I get this error message:

bash: echo: write error: Invalid argument