Raspberry Pi Cluster Episode 2 - Setting up the Cluster

This post is based on one of the videos in my series on Raspberry Pi Clustering, and I'm posting the video + transcript to my blog so you can follow along even if you don't enjoy sitting through a video :)

In the first episode, I talked about how and why I build Raspberry Pi clusters.

I mentioned my Raspberry Pi Dramble cluster, and how it's evolved over the past five years.

In this episode, I'm gonna show you how to use the Turing Pi to build an even better Raspberry Pi-based cluster. With the Turing Pi, you don't have to buy a network switch, a bunch of network cables, a bunch of micro USB cables, a multi-port USB power supply, and a bunch of microSD cards to build your cluster.

And then you don't have to spend an hour wiring everything together and building a case to hold everything, like I did with my Dramble!

The Turing Pi is, basically, a "cluster on a board". This one board has seven slots, for seven Raspberry Pi Compute Modules.

A Raspberry Pi Compute Module is basically a fast Raspberry Pi model B, but without any built in IO connections. It's on a little chip the same size as standard computer RAM. A stick of RAM, a Raspberry Pi Zero, and a Compute Module are similar in size, but very different in what they can do!

The Turing Pi includes dedicated I/O connections for the first slot, so you can manage the entire cluster through the Pi in slot 1, or you can manage the cluster externally using another computer. At a minimum, you just need to plug in power and a keyboard and mouse, or power and a network cable, and you're off to the races!

Before I talk about setting up the Turing Pi, I thought I'd show you how I built my current Raspberry Pi cluster, with four Raspberry Pi 4 model B computers.

Building a Pi Cluster

Raspberry Pi Dramble Cluster in Episode 2

I have a full parts list for my current Dramble cluster on the pidramble.com wiki. You need to buy:

All of this will set you back around $300.

Once you have everything, it's time to start assembling the parts!

First, if you're using the Power over Ethernet HAT like I am, install the HAT onto each Pi board. Since one of the headers on my PoE boards failed last time I re-assembled it, I already have them installed and won't be reinstalling them in this video.

Then you'll want to put all the Pis into your stackable cases (or into some other creative chassis you build).

Then you can wire up all the network connections, one cable to each Pi. Plug the other end of each cable into your network switch.

In my case, because I'm using Power over Ethernet, my Pis get their power straight from the PoE network switch, but if you're not using a PoE switch, you should plug in the USB power connections, with one USB cable for each Pi. But don't plug the power adapter into the wall yet, because the Pis don't have any operating system to boot!

Loading an OS onto the microSD Cards

Before we load an OS onto the Raspberry Pis, it's important to think about what you want to do with your cluster. If you want a general purpose cluster running the fully-supported Raspbian OS from the Raspberry Pi Foundation, you can download Raspbian Lite, which doesn't include a GUI or interface, just a command line or remote access interface that lets you manage the Pis via SSH.

If you want to run certain software that requires 64-bit compatibility, you might need to consider an OS that supports 64-bit on ARM processors, like Ubuntu's 64-bit ARM distribution.

If you want to have an OS meant for container workloads and that is easy to configure 'headless' (like we'll do with the Turing Pi in a bit), you might want to consider HypriotOS.

For my Dramble, I want a generic cluster running a fully supported distribution built just for the Raspberry Pi, so I'm sticking with Raspbian.

So for each Pi, plug a microSD card into a card reader attached to another computer (if you have a separate working Raspberry Pi, you could use it to flash cards!). Then follow the Raspberry Pi Foundation's instructions for flashing the Raspbian disk image to each microSD card.

Their instructions recommend using the Raspberry Pi Imager app, and there are other GUI alternatives like Etcher, but in my case, I use Terminal on my Mac. Depending on your computer, these instructions might be a little different, but here's what I do:

  1. Insert the microSD card.
  2. Run diskutil list in the Terminal to verify which disk device the card is using.
  3. Run diskutil unmountDisk /dev/disk3 to unmount the card. You can run this command even if the card isn't mounted on your Desktop. You just need to make sure the disk is unmounted or the next command will fail.
  4. Run pv [disk image name] | sudo dd bs=1m of=/dev/rdisk3 and enter your admin password to start the flashing process. This command uses the dd utility to directly write the contents of a disk image to the microSD card, using a block size of one megabyte. I use the pv or 'progress viewer' utility so I can monitor the progress of the copy, but you could drop that and just use dd instead.
  5. Wait for the flashing process to complete. It could take a few minutes. When it's done you'll see a boot volume mounted on your desktop.
  6. Run touch /Volumes/boot/ssh to create a file on the boot volume that tells the Raspberry Pi to enable SSH access when it first boots up.
  7. Unmount the card one more time using diskutil unmountDisk /dev/disk3.
  8. Remove the microSD card.
  9. Realize it's gonna be a while because you have to do all of that three more times!

Booting up the cluster the first time

After all four cards are flashed, insert them into each of the Raspberry Pis.

Now plug in your cluster's power adapter, and after a few minutes, all the Pis should be booted and ready ... but for what?

Well, the next step is finding them all on your network, connecting to them, and managing them. For now, I'm going to skip that part because it's hard [insert sarcasm here]—actually, because I want to show you how to do the same setup, but using the Turing Pi!

Building a Turing Pi Cluster

Turing Pi Cluster in Episode 2

The Turing Pi builds the power distribution and networking directly into the main board, so you don't even have to worry about purchasing all the extra cables, USB multi-port power supply, or network switch.

Instead, you'll need:

  • The Turing Pi
  • A 12V Power supply compatible with the Turing Pi
  • 7 Raspberry Pi Compute Modules (I recommend the 3+, or when it someday becomes available, a newer version)

And that's it!

You plug all the Compute Modules into the Turing Pi, and you're good to go! Except, you may be wondering, what about the operating system? Oh, yeah... about that.

Getting the right Compute Modules

Raspberry Pi Compute Module 3+ with eMMC and without eMMC

There are two types of Raspberry Pi Compute Modules:

  • Ones with built-in eMMC memory (allowing you to boot the Compute Module without an external microSD card)
  • Ones without built-in eMMC memory (meaning you'd need to buy an additional microSD card for each node)

The Turing Pi works with any Compute Module, and has a microSD card slot for each Pi, so you can choose whichever type of Compute Module fits your needs.

I recommend buying the ones with eMMC, though, because it's easier to set up and manage in a cluster.

And you don't have to fill up all the slots on the Turing Pi—you can run it with any number of Pi Compute Modules. You can even hot plug modules, meaning you could add or remove Compute Modules while the cluster is running! So you could build your cluster today with three Compute Modules, and as your needs expand or you need a faster Pi or more eMMC on a Pi, you can add more Compute Modules or replace existing ones. Pretty neat!

IBM BladeCenter from Flickr - IMG_0909 by Robert

In fact, this is something that's commonly seen only in higher-end servers, and is known as 'blade' computing. But as with many things Raspberry Pi, the Turing Pi makes this cool blade technology easy for anyone to use.

So back to the topic of operating systems—you'll still need to flash the eMMC on a Compute Module so it can boot up properly. But how do you do that? You can't plug a Compute Module into your computer's USB port!

Flashing the Compute Module eMMC

Turing Pi USB Slave Port

The Turing Pi has a built-in 'USB slave' port, at the top of the board. This micro USB port can be plugged into another computer to allow the eMMC module on a Compute Module installed in slot 1 to be flashed, just like you would flash a microSD card in a card reader.

To put slot 1 into 'flash' mode, you need to move the top jumper into the position nearest the Compute Module. Note that the Turing Pi I'm using is a prototype, so things may have changed a little bit in the production version.

Then you plug the Turing Pi's USB slave port into your computer, and power up the Turing Pi board.

At this point, it's likely you won't actually see anything happen. That's because the Compute Module's eMMC needs to be set up in 'usbboot' mode so it can appear as a USB mass storage device, just like a microSD card. And lucky you, the Raspberry Pi Foundation has a utility to do just that!

I have a separate blog post (Flashing a Raspberry Pi Compute Module on macOS with usbboot) that details this process but here's what I did:

So first, I made sure the Turing Pi was in 'flash' mode, the Compute Module was firmly seated in slot 1, the micro USB 'slave' port was plugged into my computer, and the Turing Pi was powered on.

Second, I opened my Mac's 'System Report' and went to the USB section to make sure the "BCM2710 Boot" device appeared in the list. Note that different versions of the Compute Module appear as different device numbers, but all with the prefix 'BCM'.

Third, I downloaded usbboot from the Raspberry Pi Foundation's usbboot GitHub repository.

I opened up the directory it was in in Terminal, and ran make. For this to work, I had to also make sure the libusb library was installed on my Mac, which I installed with Homebrew, using brew install libusb.

Once the make process finished, there was a new rpiboot executable, which is what I needed to run to prep the Compute Module's eMMC for flashing.

So fourth, I ran sudo ./rpiboot, and entered my admin password.

This kicks off an automated script which searches for any attached Compute Modules, then writes a couple files to the eMMC, and then exits.

Once it exits, a Mac pops up an alert saying 'The disk you inserted was not readable by this computer'. You can click 'Ignore' for now. (This only happens the first time you set up a fresh Compute Module.)

Fifth, I followed the same basic directions as before, with microSD cards, to flash an OS to the Compute Module's eMMC storage. But for the Turing Pi, I decided to use Hypriot OS, which is a little easier to preconfigure for headless servers like the Compute Module:

I ran diskutil list to see what device the eMMC was mounted as.

I unmounted the disk with diskutil unmountDisk.

I wrote the image to the card, again using pv and dd so I could monitor the progress. The card write took a bit longer with the eMMC than with microSD cards.

Once it was done, I edited the /Volumes/HypriotOS/user-data module using nano. I set the hostname to something meaningful in my cluster, like master for the first card which would act as the Kubernetes master node, or worker-05 for the fifth worker card. You can use whatever naming convention you want, just be consistent.

I also pasted in my public SSH key in a new option for the default pirate user. I added the property ssh_authorized_keys, then added a list item with my personal SSH pubkey.

If you don't already have a public SSH key—you can check if there's already a file in the path ~/.ssh/id_rsa.pub, then you can create one by running the command ssh-keygen -t rsa -b 4096 -C "[email protected]", and press enter three times. Then an SSH key will be generated in the default location ~/.ssh/id_rsa.pub.

After I finished editing the user-data file, I saved the file by pressing Control-O, and closed it with Control-X.

Finally, I ran diskutil unmountDisk again, to 'eject' the Compute Module.

At this point, you can disconnect the USB cable from the Turing Pi, disconnect power (assuming you don't have any other Pis running at the time!), and remove the Compute Module from slot 1.

One Pi down, six to go!

Alternative flashing method

Raspberry Pi Compute Module PoE Board by Waveshare

Now, earlier I mentioned you could hot-swap Compute Modules on a running Turing Pi. But it wouldn't be convenient to have to flash Compute Modules in the Turing Pi all the time, especially if you have an important node running in slot 1, and don't want to shut it down to flash another Compute Module.

Luckily, there are 'Compute Module IO Boards' available that can be used to interact with a Compute Module and flash it's eMMC. If you do a lot of work with Compute Modules, it's a good idea to have one of these boards available; it gives you so many options, and makes it easy to boot Compute Modules on their own and test things on them, before installing them more permanently in the Turing Pi.

The IO board that I have is called the Waveshare Compute Module PoE Board, and I got mine from Amazon.

Adding a case

Now that you have all the Compute Module's ready to boot, wouldn't it be nice to have a sturdy case for the Turing Pi? You're in luck, because the Turing Pi uses an industry-standard mini ITX form factor, meaning it fits perfectly into any mini ITX computer case!

Turing Pi Mounted in a mini ITX case

I found this case on Amazon for about $30, but there are many options if you don't want to build your own case.

Booting it up for the first time

Now that you have all the Compute Modules flashed with an OS, and the Turing Pi is mounted and ready to go, you just need to plug in power and a network connection, and BOOM, you have an edge cluster that will change the world! [insert more dry sarcasm here]

On to Kubernetes!

So, we have a cluster of Raspberry Pis—or, actually, a bramble of Raspberry Pis. Now, what will we do with it? And how do you even connect to it?

To find out, subscribe to my YouTube channel—I'm going to show you how to connect to the individual nodes, introduce Kubernetes, and get Kubernetes installed on the cluster in the next video!

Before I wrap up, I wanted to call out two aspiring young engineers, Louis and Robert from Houston, Texas. In our current crazy world, their Dad reached out and asked some questions about Pi clusters, since they were building one and trying to figure out what to do with it. In discussing things further, I realized what an opportunity for growth and learning they have from a Dad who is willing to teach and experiment with them!

I also have a son and two daughters, and I find their imaginations go way beyond my own! I want to encourage them, and try to keep my own brain sharp. Projects like this help, even if they're not as practical and pragmatic as building a cluster with bigger, faster computers. So if nothing else, consider building a cluster to be a learning opportunity, and to help inspire a new generation of developers, makers, and hardware hackers!

If you liked this content and want to see more, consider supporting me on GitHub or Patreon.

And if there's anything I missed or questions you have about the Turing Pi and clustering, please feel free to ask in the comments below. Until next time, I'm Jeff Geerling!

Comments

Hi Jeff,
This is an awesome project and I have really enjoyed watching you put it together. I'm a computer engineering student and find the idea of a cheap, household cluster amazing! I was thinking if this kind of setup could be used to host something like a personal website. What are your thoughts?

I've been running www.pidramble.com for years now, in my basement. It works well enough, but it does take a little maintenance, you have to worry about power (using a UPS) and Internet connectivity (usually home internet is not as reliable and doesn't give you a static IP address, so you have to figure out ways to work around that).

Oh cool, thanks for getting back to me. www.pidramble.com is a great site and inspiring to think that it is all running in your basement! Also, how did you go about getting the Turing Pi before official release? They were scheduled for release June/July and have now been pushed back until August because of Covid-19 as far as I'm aware. I'm dying to get my hands on one haha

I was able to get my hands on a prototype board for testing—I think they're still hoping to get the board out in July. I know COVID has caused a lot of delays, especially since they had trouble finishing the inspection of the final process and sourcing some parts—at least judging by what I've heard from other people trying to build some electronics currently!

So what is a compatible 12V Power supply compatible with the Turing Pi. I have been searching and I am not seeing anything listed.

Thank you.

I just received my Turing Pi - the production board looks a little different than yours, Jeff.

I followed your video for flashing the Raspberry Pi in Slot 1 but I cannot get a mac or a PC to recognize the bcm USB chip on the Pi. I tried 2 boards and tried shifting both jumpers. No luck. Can you think of anything I may be doing wrong?

Great videos, BTW, I can't wait to get Kubernetes running

Yeah I noticed they made a couple small changes on the final production board, but I thought those jumpers would be in the same position. I don't have a production board so I can't verify, but you could try tweeting them at @turingpi.

I got past the flash issue - it was a "bad' USB cable. Worked on other devices, but not the Turing Pi. I am on to networking issues now :-)

Thanks for your Turing Pi posts. They've helped a lot in the setup of mine.

My current problem is the Ethernet connection. I've plugged my Turing Pi into my Ethernet switch and still have no network connection.

Anyone have any ideas?

Btw, will there be a turingpi subreddit or something similar for the community?

Thanks again!

Hmm... a few quick tries: have you tried a different network cable, and can you confirm when you power on the cluster, the Pis you have installed are powered up and booted?

On the prototype board I had, the lights on the actual network connector never lit up, but it was working fine and I could connect to the individual Pis without a problem.

Also, to your question about a subreddit, it looks like one was set up earlier today! https://www.reddit.com/r/turingpi/.

I am having the same problem - no ethernet connectivity. I verified the cable/switch via my macbook pro. I have a loopback address (inet 127.0.0.1) and a Docker IP address (inet 172.17.0.1), nut no IPv4 ethernet address (inet) on eth0. Digging through hypriot info right now for networking.

I have been continuing to troubleshoot unsuccessfully. I plugged my Turing Pi directly into my ethernet port on my Eero mesh network. The right ethernet light was lit solid, and the left one was blinking.

I still cannot get DHCP to work - I received no DHCPOFFERS. And on another CM3+, a static network configuration would not connect.

I feel like I am missing some configuration setting on the Turing Pi itself to forward traffic from the internal switch or something like that

The hot swap or hot plug is very fragile. I have tried this and many times it resets the board. The right way should be to power off the slot using i2c commands before plugging.

True; the best solution (which I didn't have time to implement sadly, but may be able to once I get my actual board) is to build a little interface that can let you press a button to toggle power via I2C and/or indicate which slots have power with LEDs for extra safety.

I didn't have an issue with the board itself resetting, but I suspect some power supplies may help cause that to happen.

Hi Jeff,
thanks for all the effort put in to this. You got me interested in Kubernetes on pi's.
Because the TuringPi (v1) isn't available anymore, I went looking for alternatives and found this: https://www.tindie.com/products/8086net/cluster-triple-for-raspberry-pi… a board for 3 Compute modules. Okay it isn't as fancy, but delivers power and networking (USB Gadget mode) to the cm's. Using two of the gives six nodes on one controller. Maybe worthwhile looking into as well ;)

Loving this series but I am struggling to get started. While editing the `user-data` file, I am unable to change the hostname. It's still returning `ubuntu`. Additionally, I'm having issues connecting via SSH, getting `Permission denied (publickey)`. Anyone have any pointers for me? I've been able to get Ubuntu Server 20.10 running fine with the default configuration.

I ended up ditching the cloud-init configuration, as much as I wanted it to work. Instead, I've gone ahead and just changed the hostname, copied the pub key manually for each device. Not too much of an inconvenience. Now onto part 3 :)

Jeff,
First thanks for all the great work you contribute to the community.

Second is HypriotOS being actively maintained? It seems that last release they put out was July 2020 that had Docker Engine version 19.03.12 -- it appears that Docker Engine is at 20.10, so... it kind of looks like things are getting stale?

Thanks,
-- Greg

Yeah, Hypriot is a little less maintained over the past year—I would recommend using Ubuntu Server for Pi, or Raspberry Pi OS 64bit, especially if doing anything with Docker or Kubernetes.

Thank you for this series. I recently received my Turing Pi and have had trouble finding compute modules (at least at a reasonable price) due to the current chip shortage. I did manage to find a few CM3's. I got a mix with different eMMC sizes (8/16/32 GB). Do you have any recommendation how I should use these in the Turing Pi? Is it better to use a 32 GB or 8 GB for the master node? Or should the larger modules be used as workers? Are 8GB modules suitable to for use?

Thanks,

Mark

It really depends on what you want to do with them. For some workloads, storage is not really important at all. Pi OS fits within 1-4 GB of space, and extra apps and container images may take up a few more GB.

If you have a workload that will need many gigabytes of storage, you might even want to set up NFS on one of the nodes and plug in a USB hard drive or something. If you have workloads that just need a few hundred MB or GB of storage for a database or something similar, you could run it on any of the nodes and maybe not worry about the space.

I typically put as many resources into master nodes as possible though, so I'd put the highest spec Pi into that purpose.

Thanks for the reply and advice.

I watched your 6 part series and will be using that to get me started.

Many thanks again,

Mark

The fate of Hypriot OS seems a bit unclear at the moment and as Raspberry Pi OS Bullseye is out now I was wondering what the proposed upgrade path for a Turing Pi cluster running right now on Hypriot is. For the worker nodes I assume one could take them one by one offline, upgrade them to RPiOS Bullseye (any additional packages needed?) and re-insert them. K3s should reintegrate them into the cluster itself, right? What about the master node? Any other path short of a complete fresh installation?
Thanks!

could i theoretically boot it on a pirated version of mac os, and use it for video editing/final cut pro, and how would it perform