ssh

Using an Ansible playbook with an SSH bastion / jump host

Since I've set this up a number of times, but I just realized I've never documented it on my blog, I thought I'd finally do that.

I have a set of servers that are running on a private network. That network is connected to the Internet through a single reverse proxy / 'bastion' host.

But I still want to be able to manage the servers on the private network behind the bastion from outside.

Method 1 - Inventory vars

The first way to do it with Ansible is to describe how to connect through the proxy server in Ansible's inventory. This is helpful for a project that might be run from various workstations or servers without the same SSH configuration (the configuration is stored alongside the playbook, in the inventory).

In my Ansible project, I had an inventory file like the following:

SSH and HTTP to a Raspberry Pi behind CG-NAT

For a project I'm working on, I'll have a Raspberry Pi sitting behind a 4G LTE modem:

Raspberry Pi 4 with 4G LTE modem and antenna on desk

This modem is on AT&T's network, but regardless of the provider, unless you're willing to pay hundreds or thousands of dollars a month for a SIM with a public IP address, the Internet connection will be running behind CG-NAT.

What this means is there's no publicly routable address for the Pi—you can't access it from the public Internet, since it's only visible inside the cell network's private network.

There are a few different ways people have traditionally dealt with accessing devices running through CG-NAT connections:

  1. Using a VPN
  2. Using a one-off tool like ngrok
  3. Using reverse tunnels, often via SSH

And after weighing the pros and cons, I decided to go with option 3, since—for my needs—I want to have two ports open back to the Raspberry Pi:

Setting up a Pi for remote Internet connection monitoring

So... recently I acquired a Starlink 'Dishy', and I'm going to be installing it at a rural location near where I live, but since it's a bit of a drive to get to it, I wanted to set up a Raspberry Pi to monitor the Starlink connection quality over time.

Internet monitoring dashboard in Grafana

I know the Starlink app has its own monitoring, but I like to have my own fancy independent monitoring in place too.

The wrinkle with a Starlink-based Internet connection, though, is that SpaceX is using Carrier-Grade NAT (CGNAT) on their network, so there won't be any kind of IPv4 address I could reach the Pi at, nor does SpaceX yet have IPv6 set up in their network.

So to make remote access possible, I would have to find a way to have the Pi reach out to one of my servers with a persistent connection, then I could 'tunnel' through that server from other locations to reach the Pi.

Setting up a Mac mini from MacStadium for headless CI

I recently got an offer from MacStadium to use one of their dedicated Mac minis to perform CI and testing tasks for my Mac-based open source projects (for example, my Mac Dev Ansible Playbook, which I use to configure my own Macs).

Apple logo on glowy laptop background

So I thought I'd document a little bit in this blog post about how I configured the Mac mini for more secure remote administration, since Macs tend to be a little more 'open' out of the box than comparable Linux machines that I'm used to working with.

Securing SSH

First of all, I used ssh-copy-id to add my SSH key to the default administrator account on the Mac mini that was created for me:

Idempotently adding an SSH key for a host to known_hosts file with bash

I noticed on one of the CI servers I'm running that the .ssh/known_hosts file had ballooned up to over 1,000,000 lines!

Looking into the root cause (I tailed the file until I could track down a few jobs that ran every minute), I found that there was the following line in a setup script:

ssh-keyscan -t rsa github.com >> /var/lib/jenkins/.ssh/known_hosts

"This can't be good!" I told myself, and I decided to add a condition to make it idempotent (that is, able to be run once or one million times but only affecting change the first time it's run—basically, a way to change something only if the change is required):

if ! grep -q "^github.com" /var/lib/jenkins/.ssh/known_hosts; then
  ssh-keyscan -t rsa github.com >> /var/lib/jenkins/.ssh/known_hosts
fi

Now the host key for github.com is only scanned once the first time that script runs, and it is only stored in known_hosts one time for the host github.com... instead of millions of times!

Git gives 'ERROR: Repository not found.' when URL is correct and SSH key is used

I had a fun problem that made me spin my wheels an hour or so today. I was having no issue cloning a remote repository a number of times in the morning while debugging a Jenkins build job that runs a git clone + Docker image build and push operation.

Suddenly, when I was doing some final testing, I started to get the following:

git clone [email protected]:geerlingguy/my-project.git                            
Cloning into 'my-project'...
ERROR: Repository not found.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

I know that I had the repository's SSH key loaded (via eval "$(ssh-agent -s)" && ssh-add ~/.ssh/deploy-key), and if I unloaded the key, I would instead get:

Cloning private GitHub repositories with Ansible on a remote server through SSH

One of Ansible's strengths is the fact that its 'agentless' architecture uses SSH for control of remote servers. And one classic problem in remote Git administration is authentication; if you're cloning a private Git repository that requires authentication, how can you do this while also protecting your own private SSH key (by not copying it to the remote server)?

As an example, here's a task that clones a private repository to a particular folder:

- name: Clone a private repository into /opt.
  git:
    repo: [email protected]:geerlingguy/private-repo.git
    version: master
    dest: /opt/private-repo
    accept_hostkey: yes
  # ssh-agent doesn't allow key to pass through remote sudo commands.
  become: no

If you run this task, you'll probably end up with something like:

SSH into a Raspberry Pi with only a network cable using OS X's 'Internet Sharing'

Recently, I found myself in a situation where I had to connect to a Raspberry Pi to set it up for a presentation, but I did not have:

  • A keyboard and/or other input device to use to type anything into the Pi
  • An HDMI cable to connect the Pi to a display so I could view anything on the Pi
  • A microSD card reader so I could modify the contents of the Pi's microSD card

Because of this, none of the standard methods of setting a static IP address, reconfiguring the Pi's WiFi configuration, or logging in on the Pi itself to find it's IP address or set things up so I could connect over a local network would work.

I remembered that Mac OS X handily includes an 'Internet Sharing' feature, which sets up a bridged network interface so your Mac is effectively a router and DHCP server to any devices connected to the shared interface.

Fastest way to reset a host key when rebuilding servers on the same IP or hostname frequently

I build and rebuild servers quite often, and when I want to jump into the server to check a config setting (when I'm not using Ansible, that is...), I need to log in via SSH. It's best practice to let SSH verify the host key every time you connect to make sure you're not getting MITMed or anything else is going on.

However, any time you rebuild a server from a new image/OS install, the host key should be new, and this will result in the following message the next time you try to log in:

A brief history of SSH and remote access

This post is an excerpt from Chapter 11: Server Security and Ansible, in Ansible for DevOps.

In the beginning, computers were the size of large conference rooms. A punch card reader would merrily accept pieces of paper that instructed the computer to do something, and then a printer would etch the results into another piece of paper. Thousands of mechanical parts worked harmoniously (when they did work) to compute relatively simple commands.

As time progressed, computers became somewhat smaller, and interactive terminals became more user-friendly, but they were still wired directly into the computer being used. Mainframes came to the fore in the 1960s, originally used via typewriter and teletype interfaces, then via keyboards and small text displays. As networked computing became more mainstream in the 1970s and 1980s, remote terminal access was used to interact with the large central computers.