Network Monitoring and Observability
The whole point of monitoring is knowing what "normal" looked like before someone reports that it changed. Monitoring tells you when something you already know about crosses a line. Observability is having enough data to figure out the weird failures you didn't see coming. You want both, and they come from the same handful of sources: availability checks, metrics, logs, flows, config history, and tests that act like a real user.
What I watch
- Is the device up? Is the interface up?
- Interface utilization, errors, discards, and queue drops
- Optic light levels (they often fade before they fail)
- CPU, memory, temperature, fans, and power supplies
- Routing neighbors and route counts
- VPN tunnels
- Firewall sessions, denies, and state table usage
- DHCP pools, DNS response times, and IP usage
- Latency, loss, and whether the app is actually reachable
- Config changes and logins
SNMP
SNMP is old and kind of clunky, and it's still everywhere because it works. Use SNMPv3 when you can, since it actually has authentication and encryption. Pick polling intervals that fit the metric. A five-minute average can completely hide a 30-second burst that dropped traffic.
Streaming telemetry
Newer gear can push structured data out much more often than you'd want to poll it. It's great for high-resolution interface stats, routing state, and big networks. The trade-off is way more data, so plan how long you keep it and be careful how you alert on it.
Syslog
Central logs give you the "what happened" around a graph that suddenly changed:
- Interfaces going up and down
- Routing neighbors resetting
- Failed logins
- Config commits
- Spanning-tree topology changes
- Power, temperature, and hardware alarms
Make sure everything is using NTP. Lining up events from five devices is miserable when their clocks disagree.
Flow data
NetFlow, IPFIX, and sFlow tell you who talked to whom, over what protocol, and how much. It's what you want when a link suddenly fills up and you need to know why, when you're looking into a DDoS, or when you're planning capacity.
Test like a user
A router answering SNMP doesn't prove anyone can reach the website. I like checks that test the real path and service:
- Ping reachability and loss
- Can I open a TCP connection to the port?
- Does DNS resolve?
- HTTP status code and response time
- Is the TLS certificate about to expire?
- Did the traceroute path change?
Baselines first, thresholds second
A flat "alert at 80%" is noisy on a link that bursts a lot and too late on a small circuit. Learn what normal looks like by time of day and day of week first. Then alert on things that matter: utilization that stays high, big changes from normal, errors, loss, or running out of headroom.
Alerts that are worth getting
Every alert should be something you can act on. It should say what's affected, how bad it is, the actual value and the threshold, when it started, and ideally what to check first. And when a core switch dies, you want one alert, not 48 separate ones for every interface behind it.
| Severity | Example | What I do |
|---|---|---|
| Info | Config change, minor topology change | Log it, look later |
| Warning | Errors climbing, lost redundancy, sustained high usage | Look into it before customers notice |
| Critical | Service down, routing neighbor lost, power loss | Drop everything |
Back up and diff configs
Automatic config backups aren't just for disasters. Diffing today's config against yesterday's is one of the fastest ways to answer "what changed?" before an outage. Keep them somewhere secure, though. Configs are full of addresses, password hashes, and community strings.
A stack that works for a small shop
You don't need anything fancy. Something like this covers a lot:
- An SNMP-based monitoring platform (I've written about LibreNMS)
- Central syslog
- Flow collection on the edge and core
- Config backups with diffs
- External checks for public services, run from outside your network
- A status page and a plan for telling people what's going on
Ways monitoring fails
- The monitoring server depends on the same network it's watching, so it goes blind exactly when you need it.
- Alerts are so noisy that everyone ignores them.
- You only watch device uptime, not whether things actually work for users.
- Graphs keep averages that hide short spikes.
- Nobody owns the alert, or nobody knows what to do when it fires.
- IPv6, backup links, or the second power feed aren't monitored at all.
Last Modified: October 12, 2026, 11:41 am