Capacity Planning for Small Hosting Providers

"We still have some free space" is not a capacity plan. When you run a small provider, you can't add a rack, a circuit, a storage shelf, or another experienced person overnight. Capacity planning is buying or reserving those things before customers notice you're short. These are my notes on how I think about it.

Installed capacity is not sellable capacity

A cluster can show plenty of free CPU, RAM, and disk and still not have enough room to survive a node failure, finish a migration, or get through the backup window. The way I think about it:

Installed capacity
  - platform overhead
  - reserve to survive a failure
  - room for maintenance and migrations
  - performance safety margin
  - growth you'll have before new gear shows up
  = what you can actually sell
    

N+1, for real

If a three-node cluster has to keep running after losing a node, the two survivors need enough CPU, memory, storage, and network for everything that has to restart on them. The capacity on the dead node doesn't count. That sounds obvious written down, and it's still easy to forget when the dashboard says you're at 60%.

Compute

CPU you've allocated and CPU that's actually being used are two different numbers. Overcommit works because guests don't all peak at once, but pick your ratio from what you've actually measured on your workloads, not from what sounds good on a pricing page.

The CPU things I watch: sustained host utilization and how long the peaks last, load average (with an eye on blocked processes, not just the number), CPU ready or steal time inside guests, NUMA placement for big VMs, and what performance does during backups, migrations, and malware scans.

For memory: real available memory on the host, guest working sets, ballooning and swapping, how much cache the databases and filesystems need, and the overhead nobody budgets for (kernel, ZFS ARC, the hypervisor itself). And the big one: how much memory you need free to restart guests after a node fails.

A quick example

Your numbers will be different. The point is that 1.5 TB installed does not mean 1.5 TB for sale.

Storage

I think of storage as four separate questions: how much raw space there is, how much is usable after RAID/replication/snapshots, how fast it is (latency, IOPS, throughput), and whether there's room left for rebuilds, restores, and temporary copies during migrations. You can run out of any one of those while the others look fine.

Percent free is a bad alarm by itself

20% free on a 1 TB volume is 200 GB. 20% free on a 100 TB array is 20 TB. Whether either one is a problem depends on how fast it's growing and how big the biggest thing you need to fit is. So I care much more about runway:

Days to full = free capacity / average daily growth

  Free:    12 TB
  Growth:  180 GB/day
  Runway:  about 68 days
    

Now subtract how long it takes to order, ship, install, migrate, and test. If expansion normally takes 60 days, that 68-day runway is already a problem.

Things worth alerting on besides percent free: absolute free space, forecast days to full, thin-pool metadata usage (this one bites people), snapshot growth, RAID rebuild state, latency drifting from its baseline, and whether backup-repository pruning and garbage collection are actually finishing.

Network

A 10 Gbps port does not mean you have 10 Gbps of service to sell. Commits, burst rules, oversubscription, packet rate, and what happens during a failure all matter. Besides plain bandwidth (average, peak, 95th percentile), I keep an eye on packets per second, interface errors and drops, optic light levels, latency and loss, BGP route count, and state tables on firewalls and load balancers.

Plan for the bad day

Two 10 Gbps transit links don't give you 20 Gbps of resilient capacity. If you need to survive losing one, the other has to carry the load alone. I like to walk through a few scenarios:

Power, rack space, and ports

Physical capacity grows in chunks, not smoothly. One more server can mean a new circuit, a new PDU, a new switch, a cross-connect, or a whole new cabinet. Keep track of the stuff that sneaks up on you:

• Free rack units (at a depth your servers actually fit)
• Power draw against breaker limits on both A and B feeds
• PDU outlets of the right connector type
• Free switch ports and optics
• Patch panel and cross-connect room
• IP addresses and VLANs
• Spare rails, cables, drives, PSUs, and optics on the shelf
• How long the facility takes to deliver and install things

IPv4 deserves special mention. It can run out before CPU or disk does, and then it's limiting what you can sell. Track what's assigned, reserved, sitting in quarantine, used for infrastructure, and actually free. And treat IPv6 as a normal thing you offer, not a someday project.

People are capacity too

You can have empty racks and still be out of capacity if every new customer adds manual work nobody has time for. At a small shop, that "nobody" is often you. Signs you're running out:

• Tickets per customer creeping up
• More after-hours alerts, especially false ones
• Lots of time spent provisioning by hand
• A growing pile of one-off configurations
• Patching, lifecycle, and documentation work that keeps getting pushed
• Things only one person knows how to fix
• No realistic way to get another person up to speed

Licenses and vendors are capacity too. Control panel licenses, virtualization subscriptions, backup licenses, transit commits, support contracts, warranties. Sometimes the hardware is ready and the expansion is stuck waiting on a quote or a license tier.

The monthly review

Once a month I like to put the important resources in one table. It doesn't need to be fancy:

Don't plan around a single forecast. I try to think through a few: things keep going the way they are, a big customer shows up all at once, a node or circuit dies, I need double the capacity for a while during a migration, or a supplier's lead time doubles. That last one happens more often than anyone would like.

Then I ask a few questions:

• Which resource hits its "act now" point first?
• What has the longest lead time?
• What capacity disappears in the most likely failure?
• Is there anything I can safely reclaim before buying more?
• Is any monitoring data missing or lying to me?
• Does this expansion create a new single point of failure?
• Can the people I have actually run the bigger setup?

Related: Collect the data needed for forecasting, plan physical capacity, and use the monthly capacity checklist.