DockerLab

Chapter 15 of 16 · Visual guide

Networking: overlay networks and VXLAN

An overlay network makes containers on different hosts look like they share one switch, by tunneling their traffic.

key idea / Docker itselfcontainer / good pathhost / watch outproblem / blockedneutral

One network, many hosts

One logical network across two hostsContainer A on host 1 and container B on host 2 are on the same overlay subnet. Their packets travel inside VXLAN tunnels between the hosts.Overlay network "app" 10.0.1.0/24 (logical)Host 1Container A10.0.1.2eth0 192.168.1.11VXLAN endpoint (VTEP)Host 2Container B10.0.1.3eth0 192.168.1.12VXLAN endpoint (VTEP)UDP 4789Physical network between the hosts
The overlay is logical. Physically the packets ride a UDP tunnel between the hosts.

VXLAN: a packet inside a packet

A packet inside a packetThe original Ethernet frame between containers is wrapped in a VXLAN header, then in UDP and IP addressed to the other host.Outer: IP + UDP 4789 src 192.168.1.11 → dst 192.168.1.12VXLAN header network ID (VNI) identifies the overlayInner: the original frame between the containerssrc 10.0.1.2 → dst 10.0.1.3payload: your HTTP request
VXLAN = Ethernet frames carried inside UDP datagrams. The inner packet is untouched.
A request from A to BA10.0.1.2 → .3Host 1 wrapsVXLAN + UDPPhysical netroutes UDPHost 2 unwrapsremoves outerBreceives frame
Host 1 encapsulates, the physical network routes an ordinary UDP packet, host 2 decapsulates.

Swarm and the ports it needs

The built-in overlay driver expects Swarm to share network state.

Swarm: managers and workersManagers hold cluster state. Hosts exchange control traffic on TCP 2377 and 7946, and overlay data on UDP 4789.Managerswarm initWorker 1swarm joinWorker 2swarm joinTCP 2377TCP 2377UDP 4789 overlay dataTCP+UDP 7946 carries node discovery between all three
PortProtocolCarries
2377TCPswarm management (managers)
7946TCP and UDPnode-to-node discovery and gossip
4789UDPVXLAN overlay traffic

Try it yourself

One machine is enough to see the objects. Packets only cross hosts after a second node joins.

  1. Step 1 · Start Swarm 
    $ docker swarm init
Swarm initialized: current node (abc123) is now a manager.
docker swarm join --token SWMTKN-1-… 192.168.1.20:2377
  2. Step 2 · Create the overlay

    --attachable lets a normal docker run join it. Without that flag, only Swarm services can attach.

    $ docker network create -d overlay --attachable app
$ docker network inspect app --format '{{.Driver}} {{.Scope}}'
overlay swarm
  3. Step 3 · Run a service on it 
    $ docker service create --name api --network app alpine:3.20 sleep 300
$ docker service ls
ID        NAME   MODE        REPLICAS   IMAGE
xy91aa    api    replicated  1/1        alpine:3.20
$ docker service ps api
ID        NAME    NODE        DESIRED STATE   CURRENT STATE
cd33ef    api.1   this-node   Running         Running
  4. Step 4 · The ports between nodes

    Open these between managers and workers before you expect cross-host traffic. UDP 4789 carries the VXLAN packets.

    TCP 2377      swarm management
TCP/UDP 7946  node discovery
UDP 4789      overlay data (VXLAN)
$ docker service rm api
$ docker network rm app
$ docker swarm leave --force

Recap

Quick check

Two Swarm hosts cannot ping each other's containers across the overlay. Which port do you check first?
Show answer

UDP 4789, the VXLAN data port. A firewall that blocks it breaks overlay traffic while node discovery on 7946 may still work.