Debugging networking issues with multi-node Kubernetes on VirtualBox

Since this is the third time I've burned more than a few hours on this particular problem, I thought I'd finally write up a blog post. Hopefully I find this post in the future, the fourth time I run into the problem.

What problem is that? Well, when I build a new Kubernetes cluster with multiple nodes in VirtualBox (usually orchestrated with Vagrant and Ansible, using my geerlingguy.kubernetes role), I get everything running. kubectl works fine, all pods (including CoreDNS, Flannel or Calico, kube-apiserver, the scheduler) report Running, and everything in the cluster seems right. But there are lots of strange networking issues.

Sometimes internal DNS queries work. Most of the time not. I can't ping other pods by their IP address. Some of the debugging I do includes:

  • Run through everything in the Debug Services documentation. Determine services are configured fine, and see that my service endpoint are registering correctly.
  • Run through everything in the Debugging DNS Resolution documentation. Determine DNS is not working fine, but hit the end of the rope when I've tried every solution under the sun, and determine host DNS is fine, and CoreDNS is configured correctly... it's just not getting results.

One of the things that finally tipped me off to the problem was debugging DNS.

I started a debug pod running busybox:

$ kubectl run -i --tty --rm debug --image=busybox --restart=Never -- sh

Once it came up, I ran nslookup, but it said 'no servers could be reached':

/ # nslookup kubernetes.default
;; connection timed out; no servers could be reached

If I looked at the pod's /etc/resolv.conf, it seemed to be configured correctly:

/ # cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

But it couldn't reach CoreDNS running on 10.96.0.10. Hmm. I also tried ping [ip of another pod] and couldn't ping any other pods. This pointed to a general networking issue on the cluster, because you should be able to ping other pods by IP address in the same namespace.

I could also enable logging in CoreDNS (by editing the DNS ConfigMap) and then monitor DNS logs with kubectl logs -f -n kube-system -l k8s-app=kube-dns, and I would sometimes see many messages like:

2018/11/12 07:40:00 [ERROR] 2 AAAA: unreachable backend: read udp 192.168.18.1:36161->10.0.2.3:53: i/o timeout

Then I remembered a while back, while working on my local development environment for the Raspberry Pi Dramble cluster, I encountered similar networking issues. And the problem is VirtualBox creates a number of virtual network interfaces. Flannel and Calico, at least, both pick the first interface on a server, and use that to set up their own virtual networks. If they pick the default first VirtualBox network, they'll use a 10.0.2.x network that's used for the default NAT interface (enp0s3), and not the external bridge interface you configure (in my case, 192.168.7.x, or enp0s8 on Debian).

For Flannel, you need to edit the kube-flannel-ds-amd64 DaemonSet, adding the cli option - --iface=enp0s8 under the kube-flannel container spec.

For Calico, you need to edit the calico-node DaemonSet, adding the IP_AUTODETECTION_METHOD environment variable with the value interface=enp0s8.

After doing that, the networking should work correctly, and your Pods should be able to see each other. Your services (NodePorts, LoadBalancers, etc.) should also start magically working! Note that, even without this fix, iptables routes may be working correctly (they're configured via kube-proxy), so it may look like a NodePort is open and ready, but you won't get a response if the CNI (Flannel, Calico, Weave) is not configured correctly for your cluster.

A few of the issues / resources that were also helpful in debugging this problem:

Addendum

One other note: I'm using Debian 10 in this particular instance, and it uses nftables for the iptables backend... which is not currently compatible with Kubernetes' network proxying setup. So in addition to the Flannel/Calico CNI changes to use the correct interface, I had to set iptables to use iptables-legacy instead of nftables:

sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

If using Fedora, you just need the first of those two commands. With Ansible, I used the alternatives module to make the changes.

Comments

Hi, I'm trying to do something very similar right now. Any idea why things still aren't working after setting the `IP_AUTODETECTION_METHOD` variable?

Pods on my worker nodes (Calico) are not able to route to my API Server.

This really saved my trouble...have been working on this for 2 days....tried other solutions like changing vxlan to host_gw...not working

some add on to edit the flannel
kubectl -n kube-system edit daemonset kube-flannel-ds-amd64

THANK YOU FOR THIS!!!

I have been looking for why IPtables rules weren't working correctly. I tried Centos 8 and Debian 10. I understood that notables was part of the problem, but I didn't know you could do update-alternatives. I literally spent days tracing down each packets path to realize that the net filter rules weren't working.

As soon as I found this post, ran your Addendum commands and restarted my VMs, everything worked great.

Thanks again!

Joining my voice to the chorus of "THANK YOU"! I faffed around for almost a whole day before I found this post which got things working in minutes!

I just bought a copy of your k8s/ansible book and look forward to reading it.

Thank you for this solution. You have saved my life!!!

Many thanks for your excellent debugging note. Very useful indeed.
I experience exactly the same issue in VirtualBox. Unfortunately, my VM hosts are only configured with enp0s3 as the external bridge interface (192.168.1.0/24) and it is already used by Flannel to set up the overlay network.

> I0807 05:46:57.236662 1 main.go:531] Using interface with name enp0s3 and address 192.168.1.103

Do I need to add another NAT interface to the VirtualBox VMs? BTW, my VMs were all provisioned by hand without Vagrant?
Thanks a lot.

I had the same issue here, I wasnt able to resolve queries for pods and services on Ubuntu 16.04.
For some reason all my queries was using 10.0.2.3 instead of the IP use to start the cluser. What I did:

-> Add the both local IP address and the IP address where I start the cluster on /etc/network/interfaces:
dns-nameservers 192.168.0.1 192.168.33.113

# touch /etc/resolvconf/resolv.conf.d/head
# echo "nameserver 192.168.33.113" >> /etc/resolvconf/resolv.conf.d/head
# resolvconf -u
# reboot

Voilá!

Hope this helps someone!!!

Thanks very much, as with the other comments this has been troubling me for days.
Note, the nslookup may fail on recent versions of BusyBox with:

Server: 10.96.0.10
Address: 10.96.0.10:53

** server can't find kubernetes.default: NXDOMAIN

Link: https://github.com/kubernetes/kubernetes/issues/66924#issuecomment-4118…

Instead try:
kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml
kubectl exec -i -t dnsutils -- nslookup kubernetes.default

Is anyone aware of the reason why dns resolution fails on nat or host-only networking with VB?

Mr Geerling, you have nailed it!

I can report this solution has worked for kubernetes/flannel on Proxmox virtual machines
In this case, the external bridge interface (ens18) appears as the third, after cni-podman0 and docker0

Great debugging. I'm surprised that this solution hasn't been very well documented until your research

kind regards

Wow, it took me an entire week to find this post.
I have a very similar set up, using Proxmox VMs (FCOS) installing Kubernetes + Flannel and setting the --iface=ens18 flag magically made my DNS queries work!
I'm guessing, from the article, Flannel was using the loopback interface (lo), which isn't able to access anything else in the cluster. Kudos for this excellent solution and research!