Network Troubleshooting
Good network troubleshooting is a repeatable process. The goal is not to run every command you know. The goal is to reduce the problem to the smallest failing component and gather evidence before making changes.
1. Define the Symptom Precisely
“The network is down” is not a useful problem statement. Ask:
- Which users, devices, sites, or applications are affected?
- Is the problem complete failure, slowness, packet loss, or intermittent behavior?
- Did it begin at a specific time?
- What changed shortly before the problem began?
- Does the failure affect IPv4, IPv6, or both?
- Can the problem be reproduced?
2. Establish the Scope
Compare a failing client with a working client. Test from the same VLAN, another VLAN, the default gateway, and a remote location. Scope tells you whether to focus on an endpoint, access switch, routing path, service, or broader provider issue.
3. Work Through the Path
| Layer or function | Questions to answer |
|---|---|
| Physical | Is the interface up? Are optics, cabling, light levels, errors, or power abnormal? |
| Data link | Is the port in the correct VLAN? Is the MAC learned? Is spanning tree blocking? |
| Addressing | Does the client have the correct address, prefix, gateway, and DNS servers? |
| Neighbor discovery | Can ARP or IPv6 ND resolve the next hop? |
| Routing | Is there a route in both directions? Is the selected path expected? |
| Policy | Do ACLs, firewalls, NAT, or security groups permit the traffic? |
| Transport | Is the target port listening? Does the TCP handshake complete? |
| Application | Is DNS correct? Is TLS valid? Is the service healthy? |
4. Test Incrementally
ping 127.0.0.1 ping <local-interface-address> ping <default-gateway> ping <remote-address> traceroute <remote-address> dig <hostname> curl -v https://example.com/
Each test should answer one question. A successful ping to an IP address but failed access by hostname suggests DNS or application behavior, not a general routing failure.
5. Check Both Directions
Many incidents are caused by missing return routes, asymmetric firewall state, or NAT behavior. A forward path may look correct while response traffic takes another route or is dropped.
6. Use Counters and Logs
- Interface input and output errors
- Discards and queue drops
- CRC errors and speed or duplex mismatch
- Routing adjacency changes
- Firewall deny counters
- DHCP pool utilization
- CPU, memory, and control-plane load
A counter is most useful when observed over time. Clear a noncritical counter or record its current value, reproduce the problem, and check whether it increases.
7. Capture Packets at the Right Point
Packet capture reveals what actually crossed an interface. Capture as close as possible to the suspected failure and use narrow filters to reduce noise.
tcpdump -ni eth0 host 192.0.2.25 tcpdump -ni eth0 port 53 tcpdump -ni eth0 'tcp port 443 and host 203.0.113.10' tcpdump -ni eth0 -s 0 -w incident.pcap
Useful Wireshark display filters
ip.addr == 192.0.2.25 tcp.port == 443 dns icmp || icmpv6 tcp.analysis.retransmission tcp.flags.syn == 1 && tcp.flags.ack == 0
8. Change One Thing at a Time
Multiple simultaneous changes destroy evidence and make rollback difficult. Record the original state, make one controlled change, retest, and revert it if it does not improve the symptom.
Common Patterns
- Works by IP but not name: DNS resolution, search suffix, or stale cache.
- Local subnet works, remote does not: Gateway, routing, ACL, or return path.
- Small packets work, large transfers fail: MTU, PMTUD, tunnel overhead, or MSS.
- Intermittent loss: Interface errors, congestion, flapping routes, wireless interference, or overloaded state tables.
- Only one application fails: Port policy, TLS, proxy, load balancer, or application health.
- Only IPv6 fails: Router advertisements, prefix delegation, DNS AAAA records, firewall policy, or missing return route.
Incident Notes Template
Start time: When the problem was first observed.
Impact: Affected sites, users, services, and protocols.
Known good: Tests and components that work.
Known bad: Reproducible failures and evidence.
Recent changes: Deployments, maintenance, provider work, or configuration changes.
Actions: Commands, captures, changes, and results.
Resolution: Root cause, corrective action, and prevention.