How to Configure VXLAN Overlay Networks

VXLAN (Virtual Extensible LAN) creates Layer 2 overlay networks over Layer 3 infrastructure, allowing you to build private networks between servers in different data centers or even different providers. This is the foundation of container networking in Kubernetes and Docker Swarm.

What Is VXLAN?

• Encapsulates Layer 2 Ethernet frames in UDP packets
• Uses VNI (VXLAN Network Identifier) to create separate virtual networks
• Supports up to 16 million virtual networks (vs 4096 for VLANs)
• Works across Layer 3 boundaries (different subnets, data centers)

Setting Up a Point-to-Point VXLAN

# On Server A (public IP: 198.51.100.1)
sudo ip link add vxlan100 type vxlan id 100 \
  remote 203.0.113.1 dstport 4789 dev eth0
sudo ip addr add 10.200.0.1/24 dev vxlan100
sudo ip link set vxlan100 up

# On Server B (public IP: 203.0.113.1)
sudo ip link add vxlan100 type vxlan id 100 \
  remote 198.51.100.1 dstport 4789 dev eth0
sudo ip addr add 10.200.0.2/24 dev vxlan100
sudo ip link set vxlan100 up

# Test connectivity
ping 10.200.0.2  # From Server A
ping 10.200.0.1  # From Server B

VXLAN with Multicast

# For multiple servers, use multicast for automatic peer discovery
sudo ip link add vxlan100 type vxlan id 100 \
  group 239.1.1.1 dstport 4789 dev eth0
sudo ip addr add 10.200.0.1/24 dev vxlan100
sudo ip link set vxlan100 up

Firewall Rules

# Allow VXLAN traffic (UDP port 4789)
sudo ufw allow 4789/udp

# Allow traffic on the overlay network
sudo ufw allow from 10.200.0.0/24

Making VXLAN Persistent

# Netplan (Ubuntu)
network:
  version: 2
  tunnels:
    vxlan100:
      mode: vxlan
      id: 100
      link: eth0
      remote: 203.0.113.1
      port: 4789
      addresses:
        - 10.200.0.1/24

Use Cases

• Multi-datacenter private networking
• Container overlay networks (Flannel, Calico, Weave)
• Connecting VMs across different hypervisors
• Lab environments spanning multiple physical locations

Performance Considerations

# VXLAN adds ~50 bytes of overhead per packet
# MTU should be reduced to avoid fragmentation:
sudo ip link set vxlan100 mtu 1450
# Or increase the underlying network MTU to 1550+ (jumbo frames)