debian

Debugging networking issues with multi-node Kubernetes on VirtualBox

Since this is the third time I've burned more than a few hours on this particular problem, I thought I'd finally write up a blog post. Hopefully I find this post in the future, the fourth time I run into the problem.

What problem is that? Well, when I build a new Kubernetes cluster with multiple nodes in VirtualBox (usually orchestrated with Vagrant and Ansible, using my geerlingguy.kubernetes role), I get everything running. kubectl works fine, all pods (including CoreDNS, Flannel or Calico, kube-apiserver, the scheduler) report Running, and everything in the cluster seems right. But there are lots of strange networking issues.

Sometimes internal DNS queries work. Most of the time not. I can't ping other pods by their IP address. Some of the debugging I do includes:

Updating all your servers with Ansible

From time to time, there's a security patch or other update that's critical to apply ASAP to all your servers. If you use Ansible to automate infrastructure work, then updates are painless—even across dozens, hundreds, or thousands of instances! I've written about this a little bit in the past, in relation to protecting against the shellshock vulnerability, but that was specific to one package.

I have an inventory script that pulls together all the servers I manage for personal projects (including the server running this website), and organizes them by OS, so I can run commands like ansible [os] command. Then that enables me to run commands like:

Format the built-in eMMC storage on an Orange Pi Plus

To use the built-in 8GB of eMMC storage on the Orange Pi Plus as a writable volume in Linux, you need to delete the existing partitions (I think mine were formatted as FAT/WIN32), create a new partition, format the partition, then mount it:

  1. Delete the existing partitions:
    1. sudo fdisk /dev/mmcblk1
    2. p to list all partitions, then d and a number to delete all existing partitions, then w to write the changes.
  2. Create a new partition:
    1. sudo fdisk /dev/mmcblk1
    2. n to create a new partition, then use all the defaults, then w to write the changes.
  3. Format the partition: sudo mkfs.ext4 -L "emmc" /dev/mmcblk1p1
  4. Create a mount point: sudo mkdir /emmc
  5. Mount the disk: mount /dev/mmcblk1p1 /emmc