Monitoring Infrastructure with LibreNMS

Graphs are nice. Useful signals at 2:00 a.m. are nicer. LibreNMS is one of my favorite monitoring tools because it gets you a lot of the first part almost for free. But monitoring isn't about collecting graphs. It's about turning what your gear is doing into alerts that reach the right person, with enough context that they know what to do next.

Where it fits

LibreNMS is at its best as one part of how you run things, not an island of pretty graphs nobody looks at.

Devices and servers
      |
      +---- SNMP polling
      +---- ICMP reachability
      +---- Syslog / traps / service checks
      +---- Flow or traffic data
      |
      v
LibreNMS discovery and polling
      |
      +---- Dashboards and historical graphs
      +---- Alert rules and operations
      +---- Notification transports
      +---- API and integrations
      +---- Device configuration backup integration
      |
      v
Ticket / chat / email / on-call response
    

And monitor the monitoring. Use something outside LibreNMS to check the web UI, whether polling is up to date, the dispatcher, the database, disk space, and whether notifications actually get delivered. A monitoring server that quietly died is worse than none, because you think you're covered.

Adding devices

Answering SNMP doesn't mean a device is monitored. When I add something, I want to know what it is, who owns it, what it does, where it is, how to get into it, and how it should alert.

My checklist

1. Name it properly. A stable hostname, location, and description that match the inventory.
2. Lock down access. SNMPv3 where you can, only from the monitoring server's address, with credentials stored safely.
3. Add it. Check that discovery found the interfaces, sensors, routing protocols, storage, and hardware you expected.
4. Organize it. Location, device group, poller group, customer, or role.
5. Mute noise on purpose. Turn off alerts for unused ports instead of living with permanent red.
6. Test an alert. Make sure a real alert actually reaches the right place.
7. Link the runbook. Management URLs, console path, circuit ID, warranty, and what to do when it breaks.

SNMP is a trust boundary

• Only allow polling from known monitoring addresses.
• Don't use the same community string everywhere. And please not "public."
• Never expose SNMP to the Internet.
• Write down engine IDs and the auth/privacy methods for SNMPv3.
• Keep an eye on the monitoring server's source address so a routing change doesn't silently stop polling.

Alerts worth getting

Every alert should answer three things:

1. What happened?
2. Why does it matter?
3. What should I check first?

State, duration, and severity

A port that blips for a second and a ten-minute outage shouldn't wake you up the same way. Use duration, repeat timing, dependencies, and severity to cut the noise without hiding real problems.

Operations and transports

LibreNMS keeps the "what happened" separate from "who to tell and how." Set up reusable patterns: page right away, open a ticket during business hours, escalate after a delay, or keep nagging until it clears.

What a hosting provider should watch

Network

• Core and edge reachability from more than one place
• BGP session state and changes in routes received and advertised
• Interface errors, discards, optic levels, and usage
• Spanning-tree changes, LACP members dropping, and weird VLAN behavior
• Transit and peering capacity, including 95th percentile if that's how you're billed

Servers and virtualization

• Power supplies, fans, temperature, memory, RAID, and drive health
• Hypervisor CPU, memory pressure, storage latency, and thin pool usage
• How old the last backup is, and when it was last verified
• Whether the out-of-band controllers are reachable
• Cluster quorum and storage paths, through service checks or integrations

Facilities

• UPS load, runtime, battery, input voltage, and bypass state
• Room temperature and humidity
• PDU current per circuit
• Generator, transfer switch, leak, or door sensors if you have them

Services

SNMP can say a server is perfectly healthy while the website on it is broken. Add service checks or outside monitoring for:

• HTTP status, page content, redirects, and TLS validity
• Authoritative DNS answers
• SMTP banner, STARTTLS, and queue health
• Database connectivity and replication
• Backup freshness
• Control panel, API, and login workflows

Config backups

LibreNMS can hook into Oxidized or RANCID for network config backups. Then each device page shows the current config and how it's changed over time, which is fantastic.

Why config history is great

• Compare what's running now with the last known-good config.
• Spot unplanned changes right around when an incident started.
• Get replacement hardware up and running faster.
• See drift between switches or routers that are supposed to be identical.
• Have a record to look back at when reviewing changes.

Scaling it

Start with one solid server. Tune polling and the database before you add complexity. Distributed polling spreads the work across multiple nodes, but they still share the database, Redis, and RRD storage, depending on how you set it up.

Grow it gradually

1. Measure first. Is the bottleneck latency, device timeouts, disk, the database, or worker count? Don't guess.
2. Use the dispatcher service and RRDCached, like the current LibreNMS docs recommend.
3. Split heavy parts out when you need to. Database, web, RRD, and polling can each live on their own box.
4. Add Redis and more pollers. Keep the app keys the same everywhere and node IDs unique.
5. Use poller groups on purpose. Group by workload or administrative boundary. They're not a substitute for being able to route to a remote network.

HA is per component

A redundant web UI doesn't mean redundant polling or RRD storage. Think through the database, Redis, RRD, pollers, web, DNS, and notifications separately.

When an alert fires

1. Acknowledge it. So everyone knows someone's on it and two people don't chase the same thing.
2. Figure out the impact. Customers? Just redundancy? Or only the monitoring path itself?
3. Look at related signals. Power, interfaces, routes, sensors, logs, recent changes.
4. Stabilize. Contain it or fail over, per the runbook.
5. Communicate. Update the incident channel and customer status page if needed.
6. Fix it and verify. Check the service from the outside. The alert clearing isn't proof.
7. Improve. Adjust thresholds, dependencies, docs, or the design based on what you learned.

What a good alert message looks like

Severity: Critical
Device: edge01.ric1.example.net
Location: RIC1 / Rack A12
Condition: Transit interface xe-0/0/0 down for 120 seconds
Customer impact: Potential loss of primary transit; backup path expected active
Next checks:
  1. Confirm BGP state on backup transit
  2. Check optical receive power and interface errors
  3. Review facility cross-connect status
Runbook: NET-EDGE-TRANSIT-01
    

Half-asleep you will appreciate getting "what to check next" right in the alert.

Official docs

• LibreNMS Alerting
• LibreNMS Distributed Polling
• LibreNMS High Availability
• LibreNMS and Oxidized

Related: using monitoring data for capacity planning and the alert-response checklist.