Since this is the third time I've burned more than a few hours on this particular problem, I thought I'd finally write up a blog post. Hopefully I find this post in the future, the fourth time I run into the problem.
What problem is that? Well, when I build a new Kubernetes cluster with multiple nodes in VirtualBox (usually orchestrated with Vagrant and Ansible, using my geerlingguy.kubernetes role), I get everything running.
kubectl works fine, all pods (including CoreDNS, Flannel or Calico, kube-apiserver, the scheduler) report
Running, and everything in the cluster seems right. But there are lots of strange networking issues.
Sometimes internal DNS queries work. Most of the time not. I can't ping other pods by their IP address. Some of the debugging I do includes:
- Run through everything in the Debug Services documentation. Determine services are configured fine, and see that my service endpoint are registering correctly.
- Run through everything in the Debugging DNS Resolution documentation. Determine DNS is not working fine, but hit the end of the rope when I've tried every solution under the sun, and determine host DNS is fine, and CoreDNS is configured correctly... it's just not getting results.
One of the things that finally tipped me off to the problem was debugging DNS.
I started a debug pod running busybox:
$ kubectl run -i --tty --rm debug --image=busybox --restart=Never -- sh
Once it came up, I ran nslookup, but it said 'no servers could be reached':
/ # nslookup kubernetes.default ;; connection timed out; no servers could be reached
If I looked at the pod's
/etc/resolv.conf, it seemed to be configured correctly:
/ # cat /etc/resolv.conf nameserver 10.96.0.10 search default.svc.cluster.local svc.cluster.local cluster.local options ndots:5
But it couldn't reach CoreDNS running on
10.96.0.10. Hmm. I also tried
ping [ip of another pod] and couldn't ping any other pods. This pointed to a general networking issue on the cluster, because you should be able to ping other pods by IP address in the same namespace.
I could also enable logging in CoreDNS (by editing the DNS ConfigMap) and then monitor DNS logs with
kubectl logs -f -n kube-system -l k8s-app=kube-dns, and I would sometimes see many messages like:
2018/11/12 07:40:00 [ERROR] 2 AAAA: unreachable backend: read udp 192.168.18.1:36161->10.0.2.3:53: i/o timeout
Then I remembered a while back, while working on my local development environment for the Raspberry Pi Dramble cluster, I encountered similar networking issues. And the problem is VirtualBox creates a number of virtual network interfaces. Flannel and Calico, at least, both pick the first interface on a server, and use that to set up their own virtual networks. If they pick the default first VirtualBox network, they'll use a
10.0.2.x network that's used for the default NAT interface (
enp0s3), and not the external bridge interface you configure (in my case, 192.168.7.x, or
enp0s8 on Debian).
For Flannel, you need to edit the
kube-flannel-ds-amd64 DaemonSet, adding the cli option
- --iface=enp0s8 under the
kube-flannel container spec.
For Calico, you need to edit the
calico-node DaemonSet, adding the
IP_AUTODETECTION_METHOD environment variable with the value
After doing that, the networking should work correctly, and your Pods should be able to see each other. Your services (NodePorts, LoadBalancers, etc.) should also start magically working! Note that, even without this fix, iptables routes may be working correctly (they're configured via
kube-proxy), so it may look like a NodePort is open and ready, but you won't get a response if the CNI (Flannel, Calico, Weave) is not configured correctly for your cluster.
A few of the issues / resources that were also helpful in debugging this problem:
- Comment by @tmjd in "pod calico-node on worker nodes with 'CrashLoopBackOff'" thread
- Flannel Vagrant troubleshooting docs
- @lalo's answer on Stack Exchange for 'configuring flannel to use non-default interface'
One other note: I'm using Debian 10 in this particular instance, and it uses nftables for the iptables backend... which is not currently compatible with Kubernetes' network proxying setup. So in addition to the Flannel/Calico CNI changes to use the correct interface, I had to set iptables to use
iptables-legacy instead of nftables:
sudo update-alternatives --set iptables /usr/sbin/iptables-legacy sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
If using Fedora, you just need the first of those two commands. With Ansible, I used the
alternatives module to make the changes.