Diagnose Network Packet Loss on Your Server

Packet loss causes slow applications, broken connections, and degraded user experience. Unlike total outages, packet loss is insidious — everything works but poorly. This guide covers systematically diagnosing packet loss from your VPS to clients and between servers.

Measure Packet Loss

# Basic ping test (look for packet loss percentage)
ping -c 100 target-ip
# At the end: "X packets transmitted, Y received, Z% packet loss"

# MTR — the gold standard for path analysis
mtr --report --report-cycles 100 target-ip
# Shows per-hop loss and latency

# Continuous monitoring
mtr -n --report-wide target-ip

# Test from both directions
# From your server TO a target
# From external TO your server (use a monitoring service)

Diagnose Where Loss Occurs

# MTR output interpretation:
# Host           Loss%  Snt  Last  Avg  Best  Wrst  StDev
# 1. 10.0.0.1     0%    100  0.5   0.6  0.4   1.2   0.1    ← your gateway
# 2. 203.0.113.1  0%    100  1.2   1.3  1.0   2.1   0.2    ← ISP
# 3. 198.51.100.1 5%    100  15.3  14.8 13.2  45.1  5.3    ← LOSS STARTS
# 4. 10.20.30.40  5%    100  16.1  15.5 14.0  42.3  4.8    ← carrier network
# 5. target-ip    5%    100  17.2  16.4 15.1  43.5  4.9    ← destination

# Loss at hop 3 carries through to all subsequent hops
# This means the problem is AT hop 3, not at later hops

# IMPORTANT: Some routers deprioritize ICMP — showing "loss" at a hop
# but if subsequent hops show 0% loss, the intermediate "loss" is ICMP throttling

# Check NIC-level statistics
ip -s link show eth0
# Look for: RX dropped, RX errors, TX dropped, TX errors
# These indicate the server itself is dropping packets

# Check for network buffer overflows
netstat -s | grep -i "segments retransmitted\|packets dropped"
cat /proc/net/snmp | grep -A1 "Tcp:"

Server-Side Causes

# NIC ring buffer too small
ethtool -g eth0
# Increase ring buffers
sudo ethtool -G eth0 rx 4096 tx 4096

# CPU too busy to process packets (softirq backlog)
cat /proc/net/softnet_stat
# Third column (time_squeeze) > 0 means CPU couldn't process all packets

# Fix: Increase backlog
echo "net.core.netdev_max_backlog = 10000" | sudo tee -a /etc/sysctl.d/99-network.conf

# Conntrack table full (drops new connections)
dmesg | grep "nf_conntrack: table full"
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max

# TCP retransmissions (symptom of packet loss)
ss -ti | grep retrans

Network-Side Fixes

# If loss is in the network path:
# 1. Contact your VPS provider with MTR reports
# 2. Try different routes (if multiple uplinks available)
# 3. Use a CDN to reduce the network path

# If loss is at your server:
# 1. Increase NIC ring buffers
# 2. Tune kernel network parameters
# 3. Enable RPS/RFS for multi-CPU packet processing

echo "net.core.rmem_max = 16777216" | sudo tee -a /etc/sysctl.d/99-network.conf
echo "net.core.wmem_max = 16777216" | sudo tee -a /etc/sysctl.d/99-network.conf
echo "net.ipv4.tcp_rmem = 4096 87380 16777216" | sudo tee -a /etc/sysctl.d/99-network.conf
echo "net.ipv4.tcp_wmem = 4096 65536 16777216" | sudo tee -a /etc/sysctl.d/99-network.conf
sudo sysctl -p /etc/sysctl.d/99-network.conf

Best Practices

• Use MTR, not just ping — MTR shows per-hop loss to identify where the problem is
• Test bidirectionally: Loss can be asymmetric — different in each direction
• Check NIC counters: ip -s link show reveals server-side packet drops
• Increase ring buffers on high-traffic servers
• Provide MTR reports when contacting your provider about network issues
• Monitor continuously: Intermittent loss is best caught with long-running MTR