ssh

For a recent project, I needed to install LibreELEC/Kodi on a Raspberry Pi Compute Module 4 with built-in eMMC storage.

Because it's inconvenient to be swapping the Pi around from the embedded display I was using it in to my preferred carrier board I use for flashing Pis and interacting with their filesystems, I wanted to manage my LibreELEC install over SSH.

It seems like whatever documentation the LibreELEC Wiki used to have for remote SSH access is missing, and all I could find were vague references to enabling SSH during a GUI setup wizard. I don't remember seeing that.

But after many, many searches, I surmised it was possible to enable SSH by adding ssh to the end of the line in the system's cmdline.txt file, and rebooting. So I pulled the Pi, used usbboot to mount the fat32 volume on my Mac, and opened cmdline.txt and added ssh. Then I popped the Pi back in the embedded display, and started it up.

Sure enough, I could now SSH in:

Since I've set this up a number of times, but I just realized I've never documented it on my blog, I thought I'd finally do that.

I have a set of servers that are running on a private network. That network is connected to the Internet through a single reverse proxy / 'bastion' host.

But I still want to be able to manage the servers on the private network behind the bastion from outside.

Method 1 - Inventory vars

The first way to do it with Ansible is to describe how to connect through the proxy server in Ansible's inventory. This is helpful for a project that might be run from various workstations or servers without the same SSH configuration (the configuration is stored alongside the playbook, in the inventory).

In my Ansible project, I had an inventory file like the following:

For a project I'm working on, I'll have a Raspberry Pi sitting behind a 4G LTE modem:

This modem is on AT&T's network, but regardless of the provider, unless you're willing to pay hundreds or thousands of dollars a month for a SIM with a public IP address, the Internet connection will be running behind CG-NAT.

What this means is there's no publicly routable address for the Pi—you can't access it from the public Internet, since it's only visible inside the cell network's private network.

There are a few different ways people have traditionally dealt with accessing devices running through CG-NAT connections:

1. Using a VPN
2. Using a one-off tool like ngrok
3. Using reverse tunnels, often via SSH

And after weighing the pros and cons, I decided to go with option 3, since—for my needs—I want to have two ports open back to the Raspberry Pi:

So... recently I acquired a Starlink 'Dishy', and I'm going to be installing it at a rural location near where I live, but since it's a bit of a drive to get to it, I wanted to set up a Raspberry Pi to monitor the Starlink connection quality over time.

I know the Starlink app has its own monitoring, but I like to have my own fancy independent monitoring in place too.

The wrinkle with a Starlink-based Internet connection, though, is that SpaceX is using Carrier-Grade NAT (CGNAT) on their network, so there won't be any kind of IPv4 address I could reach the Pi at, nor does SpaceX yet have IPv6 set up in their network.

So to make remote access possible, I would have to find a way to have the Pi reach out to one of my servers with a persistent connection, then I could 'tunnel' through that server from other locations to reach the Pi.

I recently got an offer from MacStadium to use one of their dedicated Mac minis to perform CI and testing tasks for my Mac-based open source projects (for example, my Mac Dev Ansible Playbook, which I use to configure my own Macs).

So I thought I'd document a little bit in this blog post about how I configured the Mac mini for more secure remote administration, since Macs tend to be a little more 'open' out of the box than comparable Linux machines that I'm used to working with.

Securing SSH

First of all, I used ssh-copy-id to add my SSH key to the default administrator account on the Mac mini that was created for me:

I noticed on one of the CI servers I'm running that the .ssh/known_hosts file had ballooned up to over 1,000,000 lines!

Looking into the root cause (I tailed the file until I could track down a few jobs that ran every minute), I found that there was the following line in a setup script:

"This can't be good!" I told myself, and I decided to add a condition to make it idempotent (that is, able to be run once or one million times but only affecting change the first time it's run—basically, a way to change something only if the change is required):

Now the host key for github.com is only scanned once the first time that script runs, and it is only stored in known_hosts one time for the host github.com... instead of millions of times!

I had a fun problem that made me spin my wheels an hour or so today. I was having no issue cloning a remote repository a number of times in the morning while debugging a Jenkins build job that runs a git clone + Docker image build and push operation.

Suddenly, when I was doing some final testing, I started to get the following:

I know that I had the repository's SSH key loaded (via eval "$(ssh-agent -s)" && ssh-add ~/.ssh/deploy-key), and if I unloaded the key, I would instead get:

One of Ansible's strengths is the fact that its 'agentless' architecture uses SSH for control of remote servers. And one classic problem in remote Git administration is authentication; if you're cloning a private Git repository that requires authentication, how can you do this while also protecting your own private SSH key (by not copying it to the remote server)?

As an example, here's a task that clones a private repository to a particular folder:

If you run this task, you'll probably end up with something like:

Recently, I found myself in a situation where I had to connect to a Raspberry Pi to set it up for a presentation, but I did not have:

• A keyboard and/or other input device to use to type anything into the Pi
• An HDMI cable to connect the Pi to a display so I could view anything on the Pi
• A microSD card reader so I could modify the contents of the Pi's microSD card

Because of this, none of the standard methods of setting a static IP address, reconfiguring the Pi's WiFi configuration, or logging in on the Pi itself to find it's IP address or set things up so I could connect over a local network would work.

I remembered that Mac OS X handily includes an 'Internet Sharing' feature, which sets up a bridged network interface so your Mac is effectively a router and DHCP server to any devices connected to the shared interface.

I build and rebuild servers quite often, and when I want to jump into the server to check a config setting (when I'm not using Ansible, that is...), I need to log in via SSH. It's best practice to let SSH verify the host key every time you connect to make sure you're not getting MITMed or anything else is going on.

However, any time you rebuild a server from a new image/OS install, the host key should be new, and this will result in the following message the next time you try to log in: