Designing Backup and Disaster-Recovery Systems

A backup is a copy. Disaster recovery is the complete process of using copies, documentation, credentials, people, replacement capacity, and communication to restore a service after a serious failure.

Start with RPO and RTO

These objectives should be set per service, not per company. A billing database, a public website, a large archive, and a customer VPS may all need different recovery designs.

Build Multiple Layers

A useful backup architecture separates fast operational recovery from disaster recovery.

Production workload
      |
      +---- Local snapshot or short-retention copy
      |          Fast restore from common mistakes
      |
      +---- Primary backup repository
      |          Historical backups and routine restores
      |
      +---- Secondary or off-site copy
                 Facility loss, ransomware, or repository failure
    

The 3-2-1-1-0 idea

• 3 copies: Production plus at least two additional copies.
• 2 media or systems: Avoid one technology or administrative failure taking every copy.
• 1 off-site: Protect against facility-scale events.
• 1 offline, immutable, or strongly isolated: Limit destructive access from production credentials.
• 0 unverified errors: Monitor backup jobs and test restores.

The labels are less important than the independence. A production server and a backup VM on the same storage array are two copies but one failure domain.

Back up more than data

• Customer files, databases, mailboxes, and VM disks
• DNS zones and IP assignments
• Hypervisor, switch, firewall, and load-balancer configurations
• Application secrets and encryption keys, stored through an appropriate secure process
• License information and vendor access
• Monitoring rules, alert routes, and dashboards
• Runbooks, diagrams, and inventory
• Billing, support, and customer-contact data

Retention Is a Recovery Feature

Retention should match the ways problems are discovered. A seven-day retention cannot help with quiet corruption found after two weeks, an unnoticed deleted mailbox found after a month, or financial data needed at year end.

Grandfather-father-son style example

More retention is not always safer. It increases storage, privacy exposure, and the amount of old compromised or regulated data retained. Define expiration deliberately.

Pruning and garbage collection

Deduplicated systems may separate the removal of backup references from reclamation of underlying storage. Monitor both operations. A prune job can succeed while disk use remains high until garbage collection runs.

Protect the Backup System from Production

Ransomware and administrator mistakes often use legitimate credentials. A backup repository that trusts the same directory, root passwords, and management network as production may be deleted during the same incident.

Independence controls

Encryption changes the recovery plan

Encryption protects backup content, but losing the key converts a healthy backup into unusable data. Keep keys, recovery codes, passwords, and documentation in at least two secure locations. Test recovery using a clean system and the stored materials.

Restore Testing

A successful backup job proves only that a process wrote something to a destination. Restore testing proves that the data is understandable and that the organization can use it.

Test at multiple levels

1. File-level restore. Recover a specific customer file to an alternate location and compare it with the expected content.
2. Application-level restore. Restore files and database together, then test login, background jobs, and integrations.
3. Full VM restore. Boot the VM on an isolated network, validate services, and measure time.
4. Bare-metal or platform rebuild. Install a replacement host and restore enough configuration to accept workloads.
5. Site-recovery exercise. Assume the primary facility and normal administrative systems are unavailable.

Record evidence

Write the Recovery Runbook Before the Disaster

A recovery runbook should be usable by someone who did not design the system. It should identify decisions, not only commands.

Runbook structure

• Incident criteria and who may declare disaster recovery
• Priority order for services
• Required credentials, keys, and vendor contacts
• Replacement compute, storage, network, and IP capacity
• Exact restore sources and retention expectations
• DNS, BGP, NAT, firewall, or proxy changes
• Application validation steps
• Customer and internal communication templates
• Rollback or return-to-primary process
• Post-recovery backup and monitoring steps

Prioritize dependencies

1. Administrative access and communications
2. Core network, DNS, identity, and monitoring
3. Backup catalog / keys / repository access
4. Shared databases and storage services
5. Customer-facing compute and applications
6. Noncritical internal tools
    

Restoring an application before its DNS, database, secrets, or storage dependencies wastes time. Dependency order should be explicit.

Real-World Recovery Scenarios

Backup Program Review

At least quarterly, answer these questions:

• Which backups failed, and were failures acted upon?
• Which systems have never been restored in a test?
• Are backup credentials independent from production?
• Can the largest workload meet its RTO at measured restore speed?
• Are encryption keys and recovery documents accessible during a primary-site outage?
• Have new systems been added without backup policy?
• Has retention grown beyond legal or business need?
• Does the off-site copy share a hidden provider, account, or network dependency?

Related: Proxmox Backup Server operations, cPanel backup design, and the restore-test checklist.