Data-Center Networking and Remote Management

The physical layer gets very real when the server isn't near you. A server in a data center is only useful if you can power it, reach it, figure out what's wrong, and get it fixed when the normal network path is gone. Good remote operation starts with how you rack things, what you write down, and out-of-band access. It doesn't start with a panicked remote-hands ticket in the middle of an outage.

Racks and power

How you rack things affects cooling, how easy cables are to get to, power redundancy, and how long it takes to swap hardware. A tidy rack isn't about looking nice. It's about not pulling the wrong cable during an emergency.

Before mounting anything

• Write down which U each thing is in, and how deep it is.
• Make sure the rails fit and there's clearance front and back.
• Keep heavy stuff low enough to service safely.
• Keep network and power cables on separate paths where you can.
• Label both ends of every cable with something that'll last.
• Leave room for replacement hardware and moving cables.
• Note which way air flows, and use blanking panels where needed.

Power is a topology

Utility / generator
       |
       v
Facility UPS and distribution
       |
       +---- PDU A ---- Server PSU A
       |
       +---- PDU B ---- Server PSU B
    

Two power supplies are only redundant if they go to independent power paths. Plug both into the same PDU and you're protected against a PSU dying, but not against that PDU, its breaker, or its feed going down.

Measure power, don't guess

• Track circuit amps against the breaker limits.
• Know the facility's continuous-load rules (often 80% of the breaker).
• Watch PDU load and phase balance.
• Record actual draw at normal and peak load.
• Include startup and failover load. When one feed drops, the other one picks up everything.
• Know whether you're billed per circuit, committed capacity, or metered use.

Spell out the network handoff

Every circuit and switch port should have a clear description of what the handoff actually is. "Internet port" doesn't cut it.

Don't assume diversity

Two carriers can share the same building entrance, conduit, fiber route across town, or even the same upstream. Two cross-connects can land on the same line card. Ask about the failure that actually matters to you, and write down what's been confirmed, not what someone assumed.

How I lay out management

Administrative VPN
       |
       v
Management firewall
       |
       +---- Switch management
       +---- Router management
       +---- Hypervisor management
       +---- iDRAC / iLO / IPMI / Redfish
       +---- Console server
       +---- PDU management
    

Management interfaces should never be sitting on the public Internet. Put them on their own VLAN or network, behind a strict firewall, with strong authentication and a separate way in for recovery.

Out-of-band management

Out-of-band means a way in that still works when the OS or the normal network config is broken. It's the thing that turns a 3 a.m. drive to the data center into a 10-minute fix from your couch.

Out-of-band checklist

• Controller firmware is supported and kept up to date.
• Default passwords are gone.
• You need the VPN (or another restricted path) to reach it.
• MFA where it's available.
• Certificates and DNS names are good enough to know which box you're on.
• Power control, remote console, and virtual media have actually been tested.
• Any license needed for remote console is active. (Some vendors still lock this behind a license.)
• The management network is monitored from somewhere independent.

Redfish

Redfish is a vendor-neutral REST API for out-of-band management. It's great for scripting inventory, power, health, firmware, and events across different vendors, as long as their implementation supports what you need.

# Query the Redfish service root
curl --user 'operator:password' \
    --cacert management-ca.pem \
    https://server-mgmt.example.net/redfish/v1/
    

Don't leave credentials in shell history or scripts. Use a proper secrets mechanism, and verify the controller's certificate.

Working with remote hands

Remote hands technicians do their best work when they don't have to guess. A good request names the exact rack, U position, device label, component, cable, what the lights should look like, and when to stop.

A request that'll go badly

Please check the server and reboot it.
    

A request that'll go well

Location: RIC1, cabinet A12, U18-U19
Device: pve03.ric1.example.net, asset NS-00482
Task:
  1. Confirm the front LCD shows service tag ABC1234.
  2. Photograph the front-panel status and any amber indicators.
  3. Do not power cycle yet.
  4. Verify both power supplies show green LEDs.
  5. Verify network cables labeled PVE03-NIC1 and PVE03-NIC2 are seated.
Stop and call the escalation contact if the asset label or service tag does not match.
    

The "stop if it doesn't match" line is the important one. It's how you avoid someone rebooting the server one rack over.

What to include

• Facility, room, cage or cabinet, rack, and U position
• Hostname, asset ID, serial number, and front/rear photos if you have them
• Exactly what to do, and what they're allowed to do
• Cable labels, ports, LED states, and part numbers to expect
• Any ESD, shutdown, or safety requirements
• When to stop, and who to call
• What proof you want when they're done (photos, readings)

Spares and hardware lifecycle

Know which failures you can fix from parts on site and which mean waiting for a shipment or migrating to different hardware.

Track more than warranty dates

• Firmware and OS compatibility
• Whether you can still get drives and optics for it
• Vendor support status
• Power efficiency and how much you can fit in a rack
• Whether the management controller still works with modern browsers and TLS (old iDRACs, I'm looking at you)
• How hard it'll be to migrate off, and whether there's somewhere to migrate to

What to write down for each rack

• Front and rear rack elevations
• Asset list with serial numbers and service tags
• Which PDU outlet each power supply is plugged into
• Network ports, VLANs, IPs, and circuits
• Out-of-band addresses and how to get to them
• Cable labels and cross-connect IDs
• Where the spare parts are
• Facility access and remote hands contacts
• Photos, updated after anything significant changes

And make sure you can get to all of that when your main network and password manager are down. Keep a secure copy somewhere outside the environment.

Failure scenarios

Official reference

• DMTF Redfish specifications and schemas

Related: planning rack, power, and network capacity and the data center deployment checklist.