Backups and Disaster Recovery

A backup is only comforting after you've proved you can restore it. That's the main thing I believe about backups. A backup is just a copy. Disaster recovery is everything it takes to actually get a service back after something bad happens: the copies, plus documentation, credentials, people, somewhere to restore to, and keeping everyone informed.

Start with RPO and RTO

Set these per service, not once for the whole company. A billing database, a brochure website, a huge archive, and a customer VPS all need different things.

Layers of backups

I like to keep quick everyday restores separate from real disaster recovery:

Production workload
      |
      +---- Local snapshot or short-retention copy
      |          Quick fixes for "oops, I deleted that"
      |
      +---- Primary backup repository
      |          History and routine restores
      |
      +---- Secondary or off-site copy
                 Losing the building, ransomware, or the repo dying
    

3-2-1-1-0

• 3 copies: production plus at least two more.
• 2 different systems or media: so one technology or one admin mistake can't take out every copy.
• 1 off-site: for when the whole building has a bad day.
• 1 offline, immutable, or well isolated: something production credentials can't delete.
• 0 errors you haven't checked: monitor the jobs and test restores.

The numbers matter less than the independence. A production server and a backup VM on the same storage array are two copies, but one failure.

Back up more than the data

• Customer files, databases, mailboxes, and VM disks
• DNS zones and IP assignments
• Hypervisor, switch, firewall, and load balancer configs
• App secrets and encryption keys (stored securely)
• License info and vendor logins
• Monitoring rules, alert routing, and dashboards
• Runbooks, diagrams, and inventory
• Billing, support, and customer contact info

It's easy to have perfect backups of every VM and no copy of the firewall config that makes any of them reachable.

Retention is a recovery feature

Keep backups long enough to match how long it takes to notice problems. Seven days of retention won't help with quiet corruption found after two weeks, a deleted mailbox someone notices a month later, or financial data needed at the end of the year.

An example schedule

More retention isn't automatically better, though. It costs storage, it's more data to protect for privacy, and you end up keeping old compromised or regulated data around. Decide when things expire on purpose.

Pruning vs. actually getting space back

Deduplicating systems often split "forget this backup" from "free the space." Watch both. The prune job can succeed while the disk stays full until garbage collection runs.

Keep backups safe from production

Ransomware and admin mistakes usually use legitimate credentials. If your backup server trusts the same directory, root passwords, and management network as production, it can get wiped in the same incident.

Encryption changes the plan

Encryption protects what's in your backups, but lose the key and a perfectly healthy backup is just random noise. Keep keys, recovery codes, passwords, and docs in at least two secure places, and test a recovery from a clean machine using only those stored materials.

Restore testing

A green checkmark on a backup job only proves something got written somewhere. A restore test proves you can actually use it.

Test at different levels

1. A file. Restore one customer file somewhere else and compare it.
2. An application. Restore the files and database together, then test logins, scheduled jobs, and integrations.
3. A whole VM. Boot it on an isolated network, check the services, and time it.
4. A whole host. Install a replacement and restore enough config to run workloads again.
5. A whole site. Pretend the main facility and your usual admin tools are gone.

Write down what happened

That "problems" row is the whole point. Every restore test I'd count as a success finds at least one thing like that.

Write the runbook before the disaster

A recovery runbook should work for someone who didn't build the system. And it should cover the decisions, not just the commands.

What goes in it

• When it counts as a disaster, and who gets to decide
• Which services come back first
• Credentials, keys, and vendor contacts you'll need
• Where the replacement compute, storage, network, and IPs come from
• Exactly where to restore from, and how far back you can go
• DNS, BGP, NAT, firewall, or proxy changes
• How to check each app actually works
• Message templates for customers and staff
• How to go back to the primary once it's fixed
• Getting backups and monitoring running again afterward

Restore in dependency order

1. Admin access and a way to communicate
2. Core network, DNS, identity, and monitoring
3. Backup catalog, keys, and repository access
4. Shared databases and storage
5. Customer-facing compute and apps
6. Internal nice-to-haves
    

Restoring an app before its DNS, database, secrets, or storage is just wasted time. Write the order down.

Scenarios to think through

A quarterly gut check

• Which backups failed, and did anyone do anything about it?
• Which systems have never had a test restore?
• Are backup credentials actually separate from production?
• Can the biggest workload meet its RTO at the restore speed you've actually measured?
• Can you get to the encryption keys and recovery docs if the primary site is down?
• Did any new systems get added without a backup policy?
• Is retention longer than you legally or practically need?
• Does the off-site copy secretly share a provider, account, or network with production?

Related: Proxmox Backup Server, cPanel backups, and the restore-test checklist.