Data Center Networking
Data center networking is where the neat diagram meets the very physical reality of racks, optics, power feeds, cross-connects, and cable trays. The protocols aren't all that different from anywhere else. What's different is the scale, how much traffic goes server-to-server instead of out to the Internet, and the fact that the gear is usually somewhere you can't just walk over to.
What a DC network has to be good at
- Staying up: redundant paths and devices, so one failure isn't an outage.
- Growing: adding racks without redesigning everything.
- Low, predictable latency: especially between servers.
- Separation: firewalls and segmentation so tenants and workloads can't wander into each other.
Spine-leaf
Most newer data center networks use a spine-leaf design instead of the old three-tier model. The reason is east-west traffic. These days a lot more traffic goes between servers (databases, storage, VM migrations, backups) than in and out of the building, and spine-leaf handles that much better.
- Leaf switches sit at the top of each rack (or row) and connect the servers and storage.
- Spine switches connect to every leaf, and nothing else.
- Any server is the same number of hops from any other server: leaf, spine, leaf.
- Need more bandwidth between racks? Add a spine. Need more racks? Add leaves.
+---------+ +---------+
| Spine 1 | | Spine 2 |
+---------+ +---------+
| \ / |
| \ / |
| \ / |
| \ / |
+--------+ \ / +--------+
| Leaf 1 |----+ +----| Leaf 2 |
+--------+ +--------+
| | | |
Server Server Server Server
Every leaf connects to every spine.
These days the links between leaves and spines are usually routed (often BGP), and ECMP spreads traffic across all the spines. Keeping layer 2 inside the rack and routing everything above it is a big part of what makes spine-leaf stable.
The old three-tier model
You'll still run into this, especially in older or smaller setups:
- Core: the backbone and routing.
- Distribution: aggregates the access switches and applies policy.
- Access: where servers plug in.
It works fine at small scale. Its weak spots are big layer 2 domains held together by spanning tree, and server-to-server traffic that has to go up and back down through the hierarchy.
Other topologies you'll hear about
- Full mesh: everything connects to everything. Very redundant, expensive, and it gets messy fast.
- Partial mesh: only the important stuff is meshed.
- Ring: redundant, but doesn't scale well. More common in metro and campus networks.
- Hybrid: real networks are usually a mix of whatever made sense at the time.
The gear
Switches
- Top-of-rack (ToR): one or two in each rack. Short cables, easy to reason about.
- End-of-row (EoR): bigger switches at the end of a row, cabled to every rack. Fewer boxes, a lot more cable.
- Core / spine: the fast stuff in the middle.
Routers, load balancers, and firewalls
Routers handle the edge: transit, peering, and getting traffic to and from other networks. Load balancers spread requests across servers and route around dead ones. Firewalls enforce policy at the edge and, more and more, between internal zones too.
Protocols and tech
VLANs and VXLAN
VLANs split up layer 2. VXLAN wraps layer 2 frames in UDP so you can carry them over a routed network, with millions of segment IDs instead of VLAN's 4,094. That's what makes it popular for multi-tenant environments. Two things to plan for: MTU (VXLAN adds about 50 bytes of headers) and a control plane, usually EVPN, so you're not flooding to figure out where every MAC address lives.
A plain old access port for reference:
Switch(config)# vlan 100 Switch(config-vlan)# name Servers Switch(config)# interface GigabitEthernet0/1 Switch(config-if)# switchport mode access Switch(config-if)# switchport access vlan 100
BGP
BGP shows up twice in the data center: at the edge for transit and peering, and more and more inside the fabric itself as the routing protocol between leaves and spines. A minimal edge example:
router bgp 65001 neighbor 192.0.2.1 remote-as 65002 network 203.0.113.0 mask 255.255.255.0
There's a lot more in my BGP notes.
QoS
QoS lets important traffic jump the line when links are congested. Something like:
Switch(config)# class-map match-any HighPriority Switch(config-cmap)# match ip dscp 46 Switch(config)# policy-map QoSPolicy Switch(config-pmap)# class HighPriority Switch(config-pmap-c)# bandwidth percent 50
Honestly, in a lot of data centers the better answer is just having enough bandwidth. QoS helps most where congestion can't be avoided, like WAN links or storage traffic sharing a network.
SDN and network virtualization
Software-defined networking moves policy and config into a central controller instead of switch-by-switch. VMware NSX and Cisco ACI are the big names. They're powerful, but they bring their own complexity and licensing. For a smaller shop, well-templated configs and some automation often get you most of the benefit.
Security
- Micro-segmentation: controlling traffic between workloads even when they're on the same network.
- Firewalls: at the edge, and between zones inside.
- DDoS protection: remember it has to happen upstream of whatever link would get filled up.
Monitoring
- A monitoring system (LibreNMS, Zabbix, Nagios, whatever you like) for availability and counters.
- tcpdump and Wireshark for when you need to see actual packets.
- NetFlow or sFlow to see where traffic is going and who's using it.
- Optic light levels and interface errors. In a data center, a lot of "network problems" turn out to be dirty fiber or a dying transceiver.
What I'd keep in mind
- Check that redundancy is real. Two paths that share a power feed, a line card, or a conduit aren't really two paths.
- Label and document everything. Cabling, ports, cross-connect IDs. Remote hands can only help if they can find the right cable.
- Automate the repetitive stuff. Ansible or similar for consistent configs.
- Audit now and then. Configs drift. Make sure they still match what you think is running.
- Have out-of-band access. When the network is what broke, you still need a way in.
Last Modified: October 12, 2026, 11:41 am