Proxmox VE and Proxmox Backup Server

Installing Proxmox VE takes about fifteen minutes. Building a virtualization platform you'd trust with other people's VMs takes a lot more thought. Cluster boundaries, quorum, storage, network paths, independent backups, a way in for maintenance, and what happens when things fail all need to be decided before the first customer VM exists. These are my notes on those decisions.

Think in failure domains

The "Datacenter" in the Proxmox interface isn't a disaster recovery plan. For every node, I list what it physically depends on:

• Facility, room, rack, power circuit, and UPS
• Top-of-rack switch and the path upstream
• Management, cluster, storage, and guest networks
• Local disks, shared storage, or distributed storage
• The backup server and its datastore
• Out-of-band controller and console access

                    Public / customer networks
                              |
                      Redundant switching
                         /           \
                    PVE-01           PVE-02
                       \               /
                        \  Cluster    /
                         \ network   /
                          \         /
                        Shared or replicated
                              storage
                                 |
                         Proxmox Backup Server
                                 |
                       Off-site sync / second copy
    

That diagram is only resilient if the lines are actually independent. Two switches with one upstream, two power supplies on one circuit, or backups sitting in the same rack might be convenient, but they won't save you from the failure you're actually worried about.

Clusters, quorum, and reality

A cluster gives you shared config, migrations, and HA. It also gives you quorum and a dependency on the cluster network. Don't build a cluster just to see all your nodes in one screen.

Good reasons to cluster

• VMs need to move between nodes.
• Shared config and coordinated storage actually make your life easier.
• You understand quorum, fencing, and how to recover when a node or the network disappears.
• The cluster network is stable, low-latency, and not fighting other traffic.

Reasons to keep nodes separate

• They're in different sites with slow or flaky links between them.
• Each runs its own workloads that never need to move.
• Joining them would turn several small failure domains into one big one.
• You really just want one dashboard, not cluster behavior.

Why three nodes

An odd number of votes is much easier to reason about. With two equal nodes and a network split, neither side can know it's the one that should keep running. In small setups, a QDevice can be the tiebreaker, but treat it like a real dependency, because it is one.

Storage

There's no one right answer between local ZFS, NFS, iSCSI, Ceph, and the rest. It depends on your workloads, how fast you need to recover, your budget, and honestly, what you know how to fix at 2 a.m.

Answer these before going to production

• What happens when storage hits 100%?
• How will you find out about a failing disk or controller?
• Can you restore a VM onto a different type of storage?
• How long does a full restore take at real-world speeds?
• Does migration traffic compete with customer or backup traffic?
• Is any storage shared between clusters that can't coordinate locking? (Don't do that.)
• How do old snapshots get cleaned up?

Thin provisioning needs a hard ceiling

Thin provisioning lets you hand out more space than you physically have. That's great until a bunch of VMs grow at the same time. Alert on free space and how fast it's shrinking, and keep enough in reserve to do an emergency migration, snapshot, or restore.

Separate traffic by purpose

A small cluster might not have many physical NICs, but you should still plan the traffic types separately:

An example bridge setup

auto lo
iface lo inet loopback

iface eno1 inet manual
iface eno2 inet manual

auto bond0
iface bond0 inet manual
    bond-slaves eno1 eno2
    bond-miimon 100
    bond-mode 802.3ad
    bond-xmit-hash-policy layer3+4

auto vmbr0
iface vmbr0 inet static
    address 10.20.0.11/24
    gateway 10.20.0.1
    bridge-ports bond0.20
    bridge-stp off
    bridge-fd 0

# VLAN-aware trunk for guests
auto vmbr1
iface vmbr1 inet manual
    bridge-ports bond0
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 100-399
    

Two NICs in an LACP bond, management on VLAN 20, and guest VLANs 100–399 on a VLAN-aware bridge. Yours will be different. The bond mode, switch config, and VLANs all have to match your actual setup. And never change networking on a remote node without working out-of-band access. It's very easy to lock yourself out.

Proxmox Backup Server

PBS is a lot more than somewhere to dump backup files. You get datastores, deduplication, verification, pruning, garbage collection, optional client-side encryption, and syncing between backup servers. I think it's one of the best things about the Proxmox ecosystem.

A setup that makes sense

PVE cluster A ----\
                   >---- PBS-A datastore ---- sync ---- PBS-B at another site
PVE cluster B ----/              |
                                 +---- verification jobs
                                 +---- prune policy
                                 +---- garbage collection
                                 +---- restore tests
    

The datastore jobs

• Prune: decides which backups to drop based on your retention rules.
• Garbage collection: actually frees up the space once pruning is done. Pruning alone doesn't give space back.
• Verify: reads the backups back and checks them, so you find corruption before you need to restore.
• Sync: copies backups to another PBS.

Be careful with encryption keys

Client-side encryption keeps backup contents private even from whoever runs the backup server. But it changes how recovery works: lose the key, lose the backups. Store keys and recovery info somewhere outside the systems you're backing up. Then actually test a restore from a clean machine using only what you wrote down.

Day-to-day operations

Before maintenance

• Cluster and storage are healthy.
• There are recent backups, and important restores have been tested.
• Out-of-band access works.
• The other nodes have room to take the VMs.
• You know which customer workloads are sensitive and any restrictions.
• You've written down the rollback plan and the repo state.

Commands I check with

# Cluster status
pvecm status

# Everything running on the cluster
pvesh get /cluster/resources --type vm

# Storage status
pvesm status

# ZFS pool health
zpool status
zpool list

# Anything systemd says failed
systemctl --failed

# The task log in the web UI is worth checking too.
    

Don't update everything at once

Roll through one node at a time: check health, move or stop VMs according to plan, update one node, reboot if needed, make sure storage and networking are good, move VMs back, and only then do the next one. Keep at least one known-good node around until you're sure the update is fine.

Failures worth practicing

Official docs

• Proxmox VE Administration Guide
• Proxmox Backup Server Documentation
• Proxmox VE Storage
• Proxmox Backup Server Maintenance Tasks

Related: the bigger backup and DR picture and out-of-band access.