Docker Overlay Networks for Multi-Host Communication

Docker overlay networks enable communication between containers running on different Docker hosts. Built on VXLAN encapsulation, overlay networks create a virtual Layer 2 network that spans multiple physical hosts. This is the foundation for multi-host container deployments in Docker Swarm. This guide covers overlay network creation, configuration, and troubleshooting.

Prerequisites

• Docker Swarm initialized (docker swarm init)
• Ports open between hosts: TCP/UDP 7946 (cluster management), UDP 4789 (VXLAN)
• Port 2377 TCP for Swarm management

Creating Overlay Networks

# Create an overlay network
docker network create --driver overlay --attachable myoverlay

# With specific subnet
docker network create --driver overlay \
    --subnet 10.10.0.0/16 \
    --gateway 10.10.0.1 \
    --attachable \
    production-net

# Encrypted overlay (encrypts VXLAN traffic between hosts)
docker network create --driver overlay --opt encrypted --attachable secure-net

Using Overlay Networks with Services

# Create services on the overlay network
docker service create --name web --network myoverlay --replicas 3 -p 80:80 nginx
docker service create --name api --network myoverlay --replicas 2 myapp/api
docker service create --name db --network myoverlay --replicas 1 postgres

# Containers can reach each other by service name across hosts
# web → api (resolves via Docker DNS)
# api → db (resolves via Docker DNS)

Stack Deployment with Overlay

# stack.yml
version: "3.8"
services:
  web:
    image: nginx
    deploy:
      replicas: 3
    networks:
      - frontend
    ports:
      - "80:80"

  api:
    image: myapp/api
    deploy:
      replicas: 2
    networks:
      - frontend
      - backend

  db:
    image: postgres:16
    deploy:
      replicas: 1
      placement:
        constraints: [node.labels.storage == ssd]
    networks:
      - backend
    volumes:
      - db_data:/var/lib/postgresql/data

networks:
  frontend:
    driver: overlay
  backend:
    driver: overlay
    internal: true    # No external access

volumes:
  db_data:

Attachable Overlay Networks

# --attachable allows standalone containers (not Swarm services) to connect
docker network create --driver overlay --attachable debug-net

# Run a standalone debug container on the overlay
docker run -it --network debug-net --name debugger nicolaka/netshoot
# Now you can ping Swarm services from this container

Overlay Network Internals

# Overlay uses VXLAN (Virtual Extensible LAN)
# Each overlay network gets a VXLAN Network Identifier (VNI)

# Inspect overlay network
docker network inspect myoverlay

# Shows:
# - Subnet and gateway
# - Connected containers on each host
# - VNI and peer information

# Debug with netshoot
docker run -it --net host nicolaka/netshoot
bridge fdb show | grep vxlan
ip -d link show type vxlan

Load Balancing

# Swarm provides built-in load balancing for services on overlay networks

# Routing mesh: any node can accept traffic for any service
# Publish port on all nodes
docker service create --name web --publish 80:80 --replicas 3 nginx
# Port 80 is accessible on ANY swarm node, traffic is routed to healthy replicas

# VIP (Virtual IP) mode (default)
# Each service gets a VIP — Docker load balances across tasks
docker service inspect --format '{{json .Endpoint.VirtualIPs}}' web

# DNSRR mode (DNS round-robin)
docker service create --name api --endpoint-mode dnsrr --replicas 3 myapp/api
# Returns multiple A records — client does load balancing

Troubleshooting

# Check overlay network connectivity
docker run -it --network myoverlay nicolaka/netshoot
nslookup web     # Should resolve to VIP
ping web         # Should work across hosts

# Check Swarm network ports
# Ensure these ports are open between all nodes:
# TCP/UDP 7946 — node discovery and gossip
# UDP 4789 — VXLAN data traffic
# TCP 2377 — Swarm management

nc -zv other-node 7946
nc -zuv other-node 4789

# Check overlay driver status
docker network inspect myoverlay | jq '.[0].Peers'

Best Practices

• Use --opt encrypted for overlay networks carrying sensitive traffic between hosts
• Use internal: true for backend networks that should not have external connectivity
• Keep overlay networks scoped — do not put all services on a single overlay
• Use the routing mesh for user-facing services and DNSRR for internal service-to-service
• Monitor overlay network latency — VXLAN adds approximately 50-100 microseconds of overhead
• Ensure firewall rules allow UDP 4789 and TCP/UDP 7946 between all Swarm nodes