Network Troubleshooting
The best troubleshooting habit I know is boring: start with what you can prove. The goal isn't to run every command you know or come up with a clever theory. It's to shrink the problem down to the one thing that's failing, and to collect evidence before you start changing stuff. The packet is usually less imaginative than we are.
This is roughly the process I follow.
1. Figure out what's actually broken
"The network is down" isn't something you can troubleshoot. So I ask questions first:
- Who or what is affected? Which users, devices, sites, or apps?
- Is it completely dead, slow, dropping packets, or coming and going?
- When did it start?
- What changed right before that? (Something almost always did.)
- IPv4, IPv6, or both?
- Can I make it happen again?
2. Figure out how big it is
Find something that works and compare it to something that doesn't. Test from the same VLAN, a different VLAN, the gateway, and somewhere remote. That tells you whether to look at one endpoint, an access switch, the routing path, a service, or something upstream that isn't yours.
3. Walk the path
| Layer | What I'm checking |
|---|---|
| Physical | Is the link up? Any errors? How are the light levels on the optics? Is it plugged in? (Ask. Really.) |
| Data link | Right VLAN? Is the MAC being learned? Is spanning tree blocking the port? |
| Addressing | Does the client have the right address, prefix, gateway, and DNS? |
| Neighbors | Can ARP or IPv6 ND resolve the next hop? |
| Routing | Is there a route both ways? Is traffic taking the path I expect? |
| Policy | Are ACLs, firewalls, NAT, or security groups letting it through? |
| Transport | Is anything listening on that port? Does the TCP handshake finish? |
| Application | Is DNS right? Is the certificate valid? Is the service itself healthy? |
4. Test outward, one step at a time
ping 127.0.0.1 ping <local-interface-address> ping <default-gateway> ping <remote-address> traceroute <remote-address> dig <hostname> curl -v https://example.com/
Each test should answer exactly one question. If pinging the IP works but the hostname doesn't, stop looking at routing. That's DNS or the application.
5. Don't forget the way back
I'll keep saying this: a huge number of problems are a missing return route, asymmetric routing through a stateful firewall, or NAT doing something unexpected. The outbound path can look perfect while the replies go somewhere else or get dropped.
6. Look at counters and logs
- Interface input and output errors
- Discards and queue drops
- CRC errors, and speed/duplex mismatches
- Routing neighbors going up and down
- Firewall deny counters
- DHCP pool usage
- CPU, memory, and control-plane load
A counter by itself doesn't tell you much. It needs to be watched over time. Write down the current number (or clear it, if that's safe), reproduce the problem, and see if it goes up.
7. Capture packets in the right spot
When I'm stuck, I take a packet capture. It shows what actually crossed the interface, and it settles a lot of arguments. Capture as close to the suspected problem as you can, and filter tightly so you're not drowning in noise.
tcpdump -ni eth0 host 192.0.2.25 tcpdump -ni eth0 port 53 tcpdump -ni eth0 'tcp port 443 and host 203.0.113.10' tcpdump -ni eth0 -s 0 -w incident.pcap
Wireshark filters I use a lot
ip.addr == 192.0.2.25 tcp.port == 443 dns icmp || icmpv6 tcp.analysis.retransmission tcp.flags.syn == 1 && tcp.flags.ack == 0
That last one shows SYNs with no reply yet. It's great for spotting connections that never get answered.
8. Change one thing at a time
This is the hardest rule to follow when people are waiting on you. Change five things at once and even if it starts working, you don't know why, and you've wiped out your evidence. Write down how things were, make one change, test, and put it back if it didn't help.
Patterns worth recognizing
- Works by IP, not by name: DNS, a search domain, or a stale cache.
- Local subnet works, remote doesn't: gateway, routing, ACL, or the return path.
- Small stuff works, big transfers hang: MTU. Tunnel overhead, broken PMTUD, or MSS. It's always MTU more often than you'd think.
- Intermittent loss: interface errors, congestion, a flapping route, wireless interference, or a full state table.
- Just one app is broken: port rules, TLS, a proxy, a load balancer, or the app itself.
- Only IPv6 is broken: router advertisements, prefix delegation, AAAA records, firewall rules, or a missing return route.
Keep notes while you work
I try to jot these down during an incident, not after. It makes the write-up easy and stops me from re-testing things I already ruled out.
Start time: when someone first noticed.
Impact: which sites, users, services, and protocols.
Known good: what I've tested that works.
Known bad: what fails, and how I know.
Recent changes: deployments, maintenance, provider work, config changes.
What I did: commands, captures, changes, and what happened.
Resolution: the actual cause, the fix, and how to keep it from happening again.
Last Modified: October 12, 2026, 11:46 am