Lessons from Operating Infrastructure Since the Mid-2000s
These are things I believe a lot more strongly now than when I started. The technology keeps changing (new control panels, new hypervisors, new storage, new automation tools), but somehow the lessons keep repeating. Things still fail in the same familiar ways: nobody clearly owns it, the backups weren't there, a dependency was hiding, nobody told the customers, or someone made a change with no way back.
1. Simple and understood beats advanced and mysterious
A complicated cluster is not automatically more reliable than a pair of well-understood systems. Complexity must earn its place by solving a specific failure or scaling problem.
2. A control panel is not the operating system
Panels automate common tasks, but logs, filesystems, processes, sockets, DNS, and resource limits still determine what the server is actually doing.
3. Backups are judged by restores
A green checkmark is reassuring. A timed restore onto clean hardware is proof.
4. Monitoring must lead to action
A graph nobody looks at and an alert nobody trusts are both just expensive decorations.
5. Capacity disappears during failure
Normal free capacity is not the same as the capacity available when a node, circuit, storage path, or operator is unavailable.
6. DNS is part of nearly every migration
Even when files and databases move perfectly, DNS caching, delegation, mail routing, PTR records, and external allowlists can keep the old system alive.
7. Email is a reputation system
SMTP can be technically functional while delivery fails because identity, content, volume, compromise, or history causes remote systems to distrust the sender.
8. Out-of-band access should be tested before it is needed
A management controller with an expired license, unknown password, broken console, or unreachable VLAN is not a recovery path.
9. The quiet warning matters
A failed redundant power supply, growing disk, stale backup, or intermittent optic may not create immediate impact. Ignoring it spends the redundancy that was supposed to protect the next failure.
10. Customers remember communication
Honest, regular updates build more trust than confident guesses. It's fine not to know the cause yet. It's not fine to go silent.
11. Every exception becomes a future incident question
One-off firewall rules, custom package repositories, special mail routing, and undocumented storage paths all have to be rediscovered under pressure.
12. “Managed” must be a list of tasks
If management scope is not written down, the customer and provider will invent different definitions.
13. Security containment should be narrow but decisive
Disable the compromised capability when possible, but do not preserve availability at the cost of allowing active harm to continue.
14. Logs need time, identity, and retention
A log entry without synchronized time, source ownership, or enough retention may be useless when an abuse report arrives weeks later.
15. The old server should not be destroyed immediately
After a migration, preserve the source in a controlled state until data, mail, DNS, monitoring, and backup behavior have been proven.
16. Automation should make behavior more visible
Good automation records what it changed, handles partial failure, and can be safely rerun. Automation that hides its decisions just gets you to the mystery faster.
17. Small providers should lean into continuity
Knowing the customer, remembering the environment, and providing consistent technical ownership are real advantages over a giant anonymous platform.
18. Do not sell every resource shown as free
Maintenance, migrations, restores, backups, and failures all need headroom that does not appear on an invoice.
19. Documentation is an availability feature
A diagram, a cable map, a recovery key, or one tested command can shorten an outage more than another redundant box nobody knows how to run.
20. A system is not finished until it can be handed off
The design should be explainable to another competent operator, including normal work, failure behavior, recovery, and the reasons behind unusual choices.
21. The first answer is often only the symptom
High CPU may be a stuck backup, a database lock, malicious traffic, or a monitoring process. Keep asking what changed and what customers actually experience.
What I'd teach a new hosting operator first
- Learn the request path. Follow a user from DNS to network, TLS, web server, application, database, storage, and response.
- Learn to read logs. Know which service owns each log and how timestamps relate across systems.
- Learn safe change habits. Capture state, validate access, define rollback, and change one variable at a time.
- Learn backup recovery. Restore a file, account, VM, database, and whole platform before being responsible for them.
- Learn the network. Understand VLANs, routing, DNS, NAT, MTU, and how management access survives a failure.
- Learn customer impact. Translate a technical fault into who is affected and what they can or cannot do.
- Learn when to escalate. Asking for a second set of eyes early beats protecting your pride while an incident keeps growing.
Questions I ask before adding a new platform
- What problem does this solve that the existing platform cannot?
- Who will own it after installation?
- How is it backed up and restored?
- How is it monitored?
- What happens when its database, controller, or management interface fails?
- How are updates tested and rolled back?
- Can it be operated without the original builder?
- What is the exit or migration path if the product changes?
- Does it reduce total complexity, or only move complexity somewhere less visible?
The goal: boring
When I say I want infrastructure to be boring, I don't mean old or unambitious. I mean it behaves predictably, tells you what's going on, fails in ways you understand, and can be recovered with a process you've actually practiced.
The best maintenance window ends with no drama. The best alert tells you exactly where to start. The best restore feels routine. And the best migration leaves the customer wondering if anything happened at all.
The long-term view
Infrastructure isn't something you build once. It's a promise to keep understanding it, maintaining it, and recovering it for as long as people depend on it.