How to Diagnose Network Latency Issues

How to Diagnose Network Latency Issues

Network latency can degrade application performance and user experience on your Breeze instance. Systematic diagnosis helps pinpoint whether the issue lies in your server, the network path, DNS resolution, or the remote endpoint. This guide covers the essential tools and techniques for tracking down latency problems.

Measuring Baseline Latency with ping

Start with a simple ICMP ping to establish baseline round-trip times:

ping -c 20 target-host.example.com

Look at the min/avg/max/mdev values. High standard deviation (mdev) indicates jitter, which can be more problematic than consistently high latency. Compare results against known-good baselines for your network.

Tracing the Network Path with mtr

mtr combines ping and traceroute into a single real-time diagnostic tool:

sudo apt install -y mtr-tiny
mtr -rwzbc 100 target-host.example.com

The -r flag produces a report after -c 100 cycles. Look for hops with significantly higher latency than the previous hop — that indicates where the delay is introduced. Note that packet loss at intermediate hops is often due to ICMP rate limiting and may not indicate a real problem if the final destination shows no loss.

Analyzing TCP Connection Latency

ICMP latency does not always reflect TCP performance. Use hping3 to measure TCP SYN latency:

sudo apt install -y hping3
sudo hping3 -S -p 443 -c 10 target-host.example.com

This sends TCP SYN packets to port 443 and measures the response time, giving you a more accurate picture of application-layer connectivity.

DNS Resolution Latency

Slow DNS resolution can add hundreds of milliseconds to every connection. Measure it with dig:

dig target-host.example.com | grep "Query time"
dig @8.8.8.8 target-host.example.com | grep "Query time"
dig @1.1.1.1 target-host.example.com | grep "Query time"

If your configured resolver is slow, consider switching to a faster one or running a local caching resolver like unbound.

Application-Layer Latency with curl

Use curl with timing variables to break down HTTP request latency:

curl -o /dev/null -s -w "\
  DNS:        %{time_namelookup}s\n\
  Connect:    %{time_connect}s\n\
  TLS:        %{time_appconnect}s\n\
  TTFB:       %{time_starttransfer}s\n\
  Total:      %{time_total}s\n" \
  https://target-host.example.com

This shows you exactly where time is spent: DNS lookup, TCP handshake, TLS negotiation, server processing (TTFB), and total transfer time.

Checking for Local Resource Issues

Sometimes latency originates from the Breeze itself. Check for network buffer issues and socket queue overflows:

ss -s
netstat -s | grep -i retransmit
dmesg | grep -i "nf_conntrack: table full"

High retransmission rates or a full conntrack table can cause significant latency spikes. Increase the conntrack limit if needed:

sudo sysctl -w net.netfilter.nf_conntrack_max=262144

Continuous Monitoring

For intermittent latency issues, set up continuous monitoring with smokeping or simply log periodic measurements to a file for later analysis. Correlating latency spikes with system load, traffic patterns, or external events often reveals the root cause.