Skip to main content

Network Debugging: tcpdump, wireshark, netstat, ss

What This Concept Is​

When a network problem shows up, guessing is expensive. Four tools let you replace guesses with evidence:

  • tcpdump -- CLI packet capture. Shows the actual bytes going in and out of an interface, filtered by BPF expressions. Ubiquitous on Unix-like systems. Saves captures as .pcap files.
  • Wireshark -- GUI packet analyzer that reads tcpdump's .pcap files (or captures its own). Extensive protocol dissectors: it decodes TCP, TLS, HTTP/1, HTTP/2, DNS, and hundreds more, one layer at a time.
  • netstat -- shows sockets, their states, and per-protocol statistics. Older and slower; still useful cross-platform (including Windows).
  • ss (Linux) -- modern replacement for netstat. Much faster, richer filters, and can show TCP internal state like cwnd, retransmits, RTT, and pacing.

The workflow is usually: start with ss to see what sockets exist and what state they are in; use tcpdump to capture traffic; open the capture in Wireshark for protocol-level interpretation.

Why It Matters Here​

Almost every non-trivial production networking bug -- stuck connections, sudden latency spikes, mysterious resets, TLS handshake failures -- is solved by picking the right layer and watching real bytes or real socket states. These tools are how senior engineers confirm or refute a hypothesis in minutes instead of hours.

Concrete Example​

Capture the TCP handshake of a local connection:

sudo tcpdump -n -i lo -S "port 9000"

Typical output:

10:00:00.000001 IP 127.0.0.1.54321 > 127.0.0.1.9000: Flags [S], seq 1000, win 65535, length 0
10:00:00.000010 IP 127.0.0.1.9000 > 127.0.0.1.54321: Flags [S.], seq 4000, ack 1001, win 65535, length 0
10:00:00.000020 IP 127.0.0.1.54321 > 127.0.0.1.9000: Flags [.], ack 4001, win 65535, length 0

Three packets: SYN, SYN-ACK, ACK. Exactly the state machine from Concept 9.

List sockets and their states:

ss -tan state established '( sport = :9000 or dport = :9000 )'

See per-connection TCP internals:

ss -tin '( sport = :443 or dport = :443 )'

This shows cwnd, rtt, retrans, and more.

Common Confusion / Misconception​

"tcpdump shows me everything." It shows what reaches the kernel on that interface with the BPF filter you gave. Hardware offload (TSO, GRO) can make packets look merged; VPN/encapsulation can hide inner protocols; and TLS obviously hides payload. Use -s 0 for full packets and be aware of layer-of-capture effects.

"Wireshark is just a prettier tcpdump." Wireshark adds dissectors: it reconstructs TCP streams, decodes HTTP across segments, decrypts TLS (with session keys), and knows the quirks of hundreds of protocols. It is qualitatively different.

How To Use It​

The debugging loop:

  1. Form a hypothesis at a specific layer (e.g., "the server is sending RST after the first data segment").
  2. Use ss to confirm the socket state you expect.
  3. Use tcpdump with a narrow filter (port + host) to capture the relevant window.
  4. Open the .pcap in Wireshark and follow the TCP stream.
  5. Confirm or reject the hypothesis; repeat one layer higher or lower.

Check Yourself​

  1. Why is ss -tin more useful than netstat -ant for performance debugging?
  2. What BPF filter captures just traffic to or from 10.0.24.53 on port 443?
  3. When would Wireshark's "Follow TCP Stream" feature mislead you?
  4. Why might tcpdump on the same host that runs the server miss packets that a mid-path capture would see?
  5. How do you tell, from ss -ti, whether a connection is limited by the receiver's window or by congestion control?

Useful Filter Recipes​

  • Only traffic for one host: host 10.0.24.53.
  • Only one direction: src host 10.0.24.53 and port 443.
  • Only SYN packets: tcp[tcpflags] & tcp-syn != 0.
  • Only RST packets: tcp[tcpflags] & tcp-rst != 0.
  • Full-packet capture to disk for later analysis: sudo tcpdump -s 0 -w out.pcap "port 443".

Keep a short personal cheatsheet; good filters are the difference between a five-second confirmation and a half-hour scroll.

Mini Drill or Application​

  1. Start the echo server from Concept 13 on port 9000.
  2. Capture one connection's full lifecycle to a file: sudo tcpdump -i lo -w echo.pcap "port 9000".
  3. From nc localhost 9000, send two lines, Ctrl-D, stop the capture.
  4. Open echo.pcap in Wireshark. Identify: 3 handshake packets, data segments, 4 close packets. Note any ACKs the human eye would miss.
  5. Run ss -tin '( sport = :9000 )' while the server is idle and while it has a connection; compare.

Read This Only If Stuck​