DockerLab

Chapter 16 of 16 · Visual guide

Networking: isolation, security and troubleshooting

When a connection fails, test the container and the path separately. Follow a ladder instead of guessing.

key idea / Docker itselfcontainer / good pathhost / watch outproblem / blockedneutral

The ladder

The debugging ladderCheck in order: container running, port published, process answers inside, same network, name resolves, packets reach the bridge.1 · Running?docker ps2 · Published?inspect .Ports3 · Answers inside?exec wget 127.0.0.14 · Same network?network inspect5 · Name resolves?exec nslookup api6 · Packets seen?tcpdump -i docker0then continue on the next rowFix the first failing rung before moving down.
Each rung rules out one layer. Stop at the first rung that fails.

The most common trap

Works inside, fails outside: the listen addressA process that listens on 127.0.0.1 inside the container is unreachable through a published port. It must listen on 0.0.0.0.listens on 127.0.0.1app on 127.0.0.1:3000loopback onlypublished port forwards to eth0: refusedlistens on 0.0.0.0app on 0.0.0.0:3000all interfacespublished port reaches it
If wget works inside the container but curl on the host fails, check the listen address first.

Where does it fail?

Test from three placesA request that works from inside the container and from a sibling container, but not from the host, points at the publish rule or the listen address.inside the containerexec wget 127.0.0.1fails: the process is downa sibling containerwget http://api:3000fails: network or DNSthe hostcurl localhost:8080fails: -p or listen addressapi containerport 3000, published 8080

Isolation and security

Isolation as security--internal blocks traffic out of the network. Containers on it can talk to each other but not to the internet.backend (internal)api10.5.0.2db10.5.0.3Internetno route
docker network create --internal backend: the database tier cannot call out, even if it is compromised.

Symptoms and commands

SymptomLikely causeCheck
host curl: connection refusedno -p, or app on 127.0.0.1docker inspect Ports, listen address
bad address 'api'default bridge, or different networkdocker network inspect
works by IP, fails by nameno DNS on the default bridgeuse a user-defined network
published port exposed on the LANbound to 0.0.0.0bind to 127.0.0.1
cross-host overlay silentUDP 4789 blockedfirewall between swarm nodes
CommandTells you
docker inspect api --format '{{json .NetworkSettings.Ports}}'what is published
docker exec api ip addr / ip routethe stack the process really sees
docker network inspect appevery container on that network
tcpdump -i docker0 (on the host)whether packets reach the bridge

Try it yourself

Start from a container that is running and still refuses a connection. Each step answers one question.

  1. Step 1 · The host curl fails 
    $ docker run -d --name api --rm nginx:1.27
$ curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8080
000
  2. Step 2 · Is the container running? 
    $ docker ps --filter name=api
CONTAINER ID   IMAGE        STATUS         NAMES
aa11bb22cc33   nginx:1.27   Up 4 seconds   api
  3. Step 3 · Is a port published?

    An empty mapping means the process can be healthy and the host still has no route to it.

    $ docker inspect -f '{{json .NetworkSettings.Ports}}' api
{"80/tcp":null}
  4. Step 4 · Does the process answer inside? 
    $ docker exec api wget -q -S -O /dev/null http://127.0.0.1:80
  HTTP/1.1 200 OK
  5. Step 5 · Publish the port and curl again 
    $ docker stop api
$ docker run -d --name api --rm -p 8080:80 nginx:1.27
$ curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8080
200
$ docker stop api

Recap

Quick check

wget http://127.0.0.1:3000 works inside the container but the host curl is refused, and -p 8080:3000 is set. What is wrong?
Show answer

The app almost certainly listens on 127.0.0.1 inside the container. Published traffic arrives on eth0, so make the app listen on 0.0.0.0.