Monitoring Infrastructure with LibreNMS
Monitoring is not the act of collecting graphs. It is the process of turning service behavior into useful signals, routing those signals to the right people, and attaching enough context that an operator can decide what to do next.
LibreNMS in the Operations Stack
LibreNMS is strongest when it is part of a larger operations workflow rather than an isolated graphing system.
Devices and servers
|
+---- SNMP polling
+---- ICMP reachability
+---- Syslog / traps / service checks
+---- Flow or traffic data
|
v
LibreNMS discovery and polling
|
+---- Dashboards and historical graphs
+---- Alert rules and operations
+---- Notification transports
+---- API and integrations
+---- Device configuration backup integration
|
v
Ticket / chat / email / on-call response
The monitoring server itself needs monitoring. Use an external check for the LibreNMS web interface, polling freshness, dispatcher state, database health, storage use, and notification delivery.
Onboard Devices Consistently
A device is not fully monitored merely because it answers SNMP. Onboarding should establish identity, ownership, purpose, location, access method, and alert behavior.
Device onboarding checklist
- Set identity. Use a stable hostname, location, and description that match inventory.
- Configure secure management access. Prefer SNMPv3 where practical, restrict source addresses, and protect credentials.
- Add the device. Confirm discovery finds expected interfaces, sensors, routing protocols, storage, and hardware.
- Classify it. Assign location, device group, poller group, customer, or service role.
- Suppress noise deliberately. Disable alerts for known unused interfaces rather than accepting permanent alarms.
- Test notifications. Confirm the device can produce a real alert that reaches the correct destination.
- Link the runbook. Record management URLs, console path, circuit ID, warranty, and response instructions.
SNMP is a trust boundary
- Restrict polling to known monitoring addresses.
- Avoid reusing one community string or credential across every environment.
- Do not expose SNMP management interfaces to the public Internet.
- Document engine IDs, authentication methods, and privacy methods for SNMPv3.
- Monitor the monitoring source address so a routing change does not silently stop polling.
Design Alerts Around Action
An alert should answer three questions:
- What condition occurred?
- Why does it matter?
- What should the operator check next?
State, duration, and severity
A transient interface flap and a ten-minute outage should not have identical behavior. Use duration, repeat timing, dependencies, and severity to reduce noise without hiding real problems.
| Signal | Weak alert | Better alert |
|---|---|---|
| CPU | CPU over 80 percent once | CPU over 90 percent for 15 minutes on a latency-sensitive device |
| Disk | Disk over 80 percent | Less than 15 percent free or forecasted to fill within seven days |
| Interface | Any port down | Operationally important port down while administratively enabled |
| Bandwidth | Traffic over arbitrary Mbps value | Sustained utilization above a circuit-specific threshold with errors or drops |
| Temperature | Any sensor warning | Vendor threshold crossed, with rack/location and recent trend |
| Backup | No direct device alert | Service check reports stale or failed backup state |
Alert operations and transports
LibreNMS separates the condition from notification behavior. Reuse alert operations for patterns such as immediate paging, business-hours ticketing, escalation after a delay, or repeated notification until clear.
Immediate page
Core router unavailable, facility power alarm, storage controller failure, or customer-impacting node outage.
Ticket or daytime review
Capacity trend, redundant power supply failure, noncritical sensor warning, or certificate expiration.
Record only
Known maintenance state, low-value access-port event, or information retained for correlation.
Signals That Matter to a Hosting Provider
Network
- Core and edge reachability from more than one viewpoint
- BGP session state and received/advertised route changes
- Interface errors, discards, optical levels, and utilization
- Spanning-tree changes, LACP member loss, and unexpected VLAN behavior
- Transit and peering capacity, including 95th-percentile billing views where applicable
Servers and virtualization
- Power supply, fan, temperature, memory, RAID, and drive health
- Hypervisor CPU, memory pressure, storage latency, and thin-pool use
- Backup age and last successful verification
- Out-of-band controller availability
- Cluster quorum and storage-path state through service checks or integrations
Facilities
- UPS load, runtime, battery status, input voltage, and bypass state
- Environmental temperature and humidity
- Power distribution unit current per circuit
- Generator, transfer-switch, leak, or door sensors where available
Services
SNMP may report a server as healthy while the customer-facing service is broken. Add service checks or external monitoring for:
- HTTP status, content, redirects, and TLS validity
- Authoritative DNS answers
- SMTP banner, STARTTLS, and queue health
- Database connectivity and replication state
- Backup freshness
- Control-panel, API, and authentication workflows
Network Configuration Backups
LibreNMS can integrate with configuration-backup systems such as Oxidized or RANCID. This turns a device page into a place to review current configuration and changes over time.
Why configuration history matters
- Compare the current state with the last known-good configuration.
- Identify unplanned changes near the start of an incident.
- Rebuild replacement hardware faster.
- Review drift between supposedly identical switches or routers.
- Provide evidence for change reviews.
Scaling LibreNMS
Start with a stable single-server installation. Optimize polling and database performance before adding complexity. Distributed polling can spread discovery and polling tasks across multiple nodes, but the nodes still depend on shared database, Redis, and RRD infrastructure according to the design.
A gradual scale path
- Measure poller performance. Confirm whether latency, device timeouts, disk, database, or worker count is the actual limit.
- Use the dispatcher service and RRDCached. Follow current LibreNMS guidance.
- Separate heavy components when needed. Database, web, RRD, and polling workloads can be placed on different systems.
- Add Redis and additional pollers. Keep application keys consistent and node IDs unique.
- Use poller groups deliberately. Group by workload or administrative boundary, not as a substitute for routing to remote networks.
High availability is component-specific
A redundant web interface does not guarantee redundant polling or RRD storage. Map the database, Redis, RRD, poller, web, DNS, and notification dependencies separately.
Responding to an Alert
- Acknowledge. Show that someone owns the alert and prevent duplicated effort.
- Confirm impact. Determine whether customers, redundancy, or only a monitoring path is affected.
- Check related signals. Correlate power, interfaces, routes, sensors, logs, and recent changes.
- Stabilize. Contain the issue or fail over according to the runbook.
- Communicate. Open or update the incident channel and customer status when required.
- Resolve and validate. Confirm the service from an external viewpoint, not only that the alert cleared.
- Improve. Adjust thresholds, dependencies, documentation, or architecture based on what was learned.
Example alert template content
Severity: Critical
Device: edge01.ric1.example.net
Location: RIC1 / Rack A12
Condition: Transit interface xe-0/0/0 down for 120 seconds
Customer impact: Potential loss of primary transit; backup path expected active
Next checks:
1. Confirm BGP state on backup transit
2. Check optical receive power and interface errors
3. Review facility cross-connect status
Runbook: NET-EDGE-TRANSIT-01