Recently, I noticed logs for one of my web services had strange entries that looked like a bot trying to perform scripted attacks on an application endpoint. I was surprised, because all the endpoints that were exposed over the public Internet were protected by some form of authentication, or were locked down to specific IP addresses—or so I thought.
I had re-architected the service using Docker in the past year, and in the process of doing so, I changed the way the application ran—instead of having one server per process, I ran a group of processes on one server, and routed traffic to them using DNS names (one per process) and Nginx to proxy the traffic.
In this new setup, I built a custom firewall using
iptables rules (since I had to control for a number of legacy services that I have yet to route through Docker—someday it will all be in Kubernetes), installed Docker, and set up a Docker Compose file (one per server) that ran all the processes in containers, using ports like
The Docker Compose port declaration for each service looked like this:
Nginx was then proxying the requests routing DNS names like
service-1234.example.com (on port 443) through to the process on port 1234,
service-1235.example.com to port 1235, and so on.
I thought the following port declaration would only expose the port on localhost:
127.0.0.1:1234:1234, according to the Published ports documentation. And in normal circumstances, this does seem to work correctly (see a test script proving this here).
But in my case, likely due to some of the other custom
iptables rules conflicting with Docker's rules, Docker's iptables modifications exposed the container ports (e.g.
1234) to the outside world, on the
I was scratching my head as to how external requests to
server-ip:1234 were working even though the port declaration was binding the port to the
localhost, and I found that I'm not the only person who was bumping into this issue. The fix, in my case, was to add a rule to the
iptables -I DOCKER-USER -i eth0 ! -s 127.0.0.1 -j DROP
This rule, which I found buried in some documentation about restricting connections to the Docker host, drops any traffic from a given interface that's not coming from
localhost. This works in my case, because I'm not trying to expose any Docker containers to the world—if you wanted to have a mix of some containers open to the world, and others proxied, this wouldn't work for you.
After applying the rule, verify it's sticking by restarting your server and checking the
$ sudo iptables -L
Chain DOCKER-USER (1 references)
target prot opt source destination
DROP all -- !localhost anywhere
RETURN all -- anywhere anywhere
To be doubly-sure your firewall is intact, you can verify which ports are open (
-p- says 'scan all ports, 1-65535) using
sudo nmap -p- [server-ip-address]
This behavior is confusing and not well-documented, even more so because a lot of these options behave subtly different depending on if you're using
docker stack. As another example of this maddening behavior, try figuring out how cpu and memory restrictions work with
docker run vs.
docker-compose v2, vs
docker-compose v3 vs 'Swarm mode'!
I think I might be going crazy with the realization that—at least in some cases—Kubernetes is simpler to use than Docker, owing to its more consistent networking model.
Thanks for this post, I have also realized this only very recently.
Thanks for this post!
I got caught by this bug too, after exposing a password-less Postgres instance to the internet and then finding someone had compromised it and was using it, presumably, to mine a cryptocurrency. Feel stupid for doing that and accept responsibility, but still can't help but feel Docker's default behaviour here ought to be far more verbose about the fact that it's exposing ports.
Thank you for the article!
I ran into an unfortunate side effect of the iptable rule "iptables -I DOCKER-USER -i eth0 ! -s 127.0.0.1 -j DROP". While it protected my containers from outside access as intended, it somehow blocked all DNS queries my containerized services tried to make.
I found that the suggested iptable rules by https://github.com/moby/moby/issues/32299#issuecomment-360421915 still protect my containers, but don't interfere with the DNS communication. The rules are
iptables -I DOCKER-USER -i eth0 -j DROP
iptables -I DOCKER-USER -m state --state RELATED,ESTABLISHED -j ACCEPT
Thanks, worked for me
It sounds like maybe the docker-compose "ports" directive is the same as the docker --publish directive? That is, docker-compose makes the ports public by default which is the opposite of docker. If you want internal ports only, you might have to use the "expose" directive in your docker-compose configuration.
From the docker-compose documentation:
Expose ports without publishing them to the host machine - they’ll only be accessible to linked services. Only the internal port can be specified.
In the news today: How a Docker footgun led to a vandal deleting NewsBlur's MongoDB database.
I'm so glad I found this - I have just created a basic web-server with a dockerised database and did some basic security checks with nmap to find that my iptables rule hadn't worked!
In more complex settings where there may be a range of external access required, blocking all but localhost is too agreesive. How to go about this in Docker and IMHO I think the Docker devs could do better.
I've found the most elegant approach is to focus on the prerouting and portmapping aspect of the docker NAT rules. This way we can intercept incoming traffic at the NAT layer before any other docker filter rules are applied. To do this we need to look inside the NAT translation port to reverse engineer and unmangle the original destination port.
Put simply, you can block a single incoming port 8080 for all containers like this:
iptables -I DOCKER-USER -i $DEFAULT_ROUTE_IF -p tcp -m conntrack --ctorigdstport 8080 -j DROP
Or to block a port for all but a specific IP range, and so on:
iptables -I DOCKER-USER -i $DEFAULT_ROUTE_IF -s 192.168.10.0/24 -p tcp -m conntrack --ctorigdstport 8080 -j ACCEPT
Put all this in a script and start it as a systemd service to keep the rules persistent accros reboots (the iptables-persistent package breaks docker!)
If deploting more widely, to acertain the correct interface where exteranally derviced connections through NAT will be processed, add this to your script:
DEFAULT_ROUTE_IF=$(ip route show to default | grep -Eo "dev\s*[[:alnum:]]+" | sed 's/dev\s//g')
Adding to my own comments. I forgot to add one extra workaround that is also needed...
You need to start the new firewall script AFTER docker (the DOCKER-USER filter needs to exist first!) You can either run this at boot with a cron job ie @reboot && sleep 30 yourscript.sh (a litte imprecise and therfore not as secure) , or preferably as a service that is aware of docker (or any docker restarts) as follows
Description=must load this rule after docker starts