nas

Last year, after I started a search for a good out-of-the-box all-flash-storage setup for a video editing NAS, I floated the idea of an all-M.2 NVMe NAS to ASUSTOR. I am not the first person with the idea, nor is ASUSTOR the first prebuilt NAS company to build one (that honor goes QNAP, with their TBS-453DX).

But I do think the concept can be executed to suit different needs—like in my case, video editing over a 10 Gbps network with minimal latency for at least one concurrent user with multiple 4K streams and sometimes complex edits, without lower-bitrate transcoded media (e.g. ProRes RAW).

Do you remember when Netflix first started their movie streaming service, back in 2007?

Physical media was still the preferred way to consume media. Besides sports content, and some TV shows that were cable-exclusive for a time, most people would run by Blockbuster and pick up a movie.

Netflix started with mail-order DVDs, then switched to streaming. The absence of ads (which were rampant on cable channels) and the convenience of not having to drive to a physical store (Blockbuster et all) made Netflix a no-brainer, especially considering the depth of their initial library.

I edit videos non-stop nowadays. In a former life, I had a 2 TB backup volume and that stored my entire digital life—all my photos, family video clips, and every bit of code and text I'd ever written.

Video is a different beast, entirely.

Every minute of 4K ProRes LT footage (which is a very lightweight format, compared to RAW) is 3 GB of space. A typical video I produce has between 30-60 minutes of raw footage (which brings the total project size up to around 100-200 GB).

A few weeks ago, I posted a video about the Petabyte Pi Project—an experiment to see if a single Raspberry Pi Compute Module 4 could directly address sixty 20TB hard drives, totaling 1.2 Petabytes.

And in that video, it did, but with a caveat: RAID was unstable. For some reason, after writing 2 or 3 GB of data at a time, one of the HBAs I was using would flake out and reset itself, due to PCI Express bus errors.

I haven't had time to write up the details yet, but I wanted to share a project that's been many months in the making: The Petabyte Pi Project on YouTube.

I'm still doing follow-up testing based on feedback from Broadcom storage engineers, and will put out a much more in-depth blog post later, but the gist is:

Can a single Raspberry Pi cosplay as an 'enterprise' storage server, directly addressing 1 PB of storage?

Now... caveats abound here. What does 'enterprise' mean? And what does 'directly addressing' mean? Those things are all answered in the video linked above.

But to give a tl;dr: The Pi does not perform swimmingly. But... I did get a single array of 60 hard drives—20TB Exos HDDs to be exact—working in a 45Drives Storinator XL60 chassis, controlled only through a single Raspberry Pi Compute Module 4. Of course I had to rip out the Xeon guts and replace them with said Pi:

It's been more than a decade since I wrote Ripping Movies from Blu-Ray, HD-DVD and DVD, Getting them onto Apple TV, iPad, iPhone, etc.. Heck, back then I didn't write everything as a 'blog post'—that was labeled as an 'article' :P

In a surprising twist of fate, we went from a somewhat more centralized online media situation back then (basically, Netflix) to a hellscape of dozens of streaming services today. And in many cases, older movies can only be found as used and/or pirated DVDs on eBay!

Thus, I'm writing a fresh guide to how I rip DVDs and Blu-Ray discs into my Mac, then transcode them with Handbrake. Heck, some people who are deeper into the r/datahoarder rabbit hole even have dedicated transcoding servers so they can generate optimal archival copies in 4K, 1080p, etc. akin to how YouTube and other online platforms set up their files!

But for me, the basic process goes:

Over on the Geerling Engineering YouTube channel, my Dad and I just posted a video where we installed the ASUSTOR Lockerstor 4RS - AS6504RS at his radio station, to increase their raw network storage capacity from 4 to 16 TB:

In the video, we focused on the installation, though I highlighted the unit's top-line features at the beginning.

In this blog post, I'll quickly recap the main features, then give more impressions of the unit from our experience setting it up, and my Dad's use of it at the station since we recorded the video.

A few months ago, I wrote up a post covering my backup plan. In it, I talk about the 3-2-1 backup strategy:

• 3 copies of all your important data
• 2 different media
• 1 offsite

In that post, I mentioned I back everything up with two local copies (two separate NAS units), and a third offsite copy on Amazon Glacier Deep Archive.

I've had a number of people ask about my backup strategy—how I ensure the 6 TB of video project files and a few TB of other files stays intact over time.

Over the past year, since I got more serious about my growing YouTube channel's success, I decided to document and automate as much of my backups as possible, following a 3-2-1 backup plan:

• 3 Copies of all my data
• 2 Copies on different storage media
• 1 Offsite copy

The culmination of that work is this GitHub repository: my-backup-plan.

The first thing I needed to do was take a data inventory—all the files important enough for me to worry about fell into six main categories:

I was recently working on some backup scripts to make sure I could clone all my GitHub repositories to my NAS, which I have mounted to a Raspberry Pi that handles all my backups.

I'm using gickup to run through all my GitHub repos and clone them locally, and I configured it to clone each repo directly into my NAS share, which is mounted over CIFS using something like:

sudo mount -t cifs -o uid=pi,username=myuser,password=mypass //my-nas-server/Backups /Volumes/Backups

Most repositories cloned correctly, but a few had symlinks inside, and when git was cloning them, the process would error out with: