NightFox

ISP and Hosting Network Operations

Running a provider network is a different animal from running an office network. You're swapping routes with other networks, handing out public address space, keeping customers away from each other, protecting infrastructure everybody shares, and dealing with the fact that one failure can hit a lot of people at once. These are the things that matter most.

Who you're talking BGP with

An autonomous system is a network under one routing policy, identified by its ASN. As a provider, you'll usually have BGP sessions with:

• Transit providers: you pay them to reach the rest of the Internet.
• Peers: networks you swap traffic with directly, usually without paying each other.
• Internet exchanges: a shared switch where lots of networks can peer at once.
• Customers: networks that get transit or routed connectivity from you.

Transit and peering

Having more than one transit provider gives you resilience and some ways to steer traffic. Peering can cut latency and transit bills, but every session is more policy, monitoring, and upkeep. Before adding a provider or an exchange, I think about:

• Is the physical path actually diverse from what I already have?
• Commit and burst pricing
• How good their routes are, and where they reach
• What they offer for DDoS mitigation
• How fast their support answers when something's broken
• Remote peering vs. actually being in the building

Filter everything

Never trust a BGP neighbor to send you only good routes. Every session gets explicit policy, both directions:

• Only accept the prefixes a customer is actually authorized to announce.
• Reject default routes unless you specifically want one.
• Reject private, reserved, and bogus address space.
• Set max-prefix limits.
• Never export your internal infrastructure routes.
• Build filters from IRR route objects, RPKI, and what the customer told you in writing.

RPKI

With RPKI, whoever holds an address block publishes a ROA (Route Origin Authorization) saying which ASNs are allowed to announce it and how specific the announcement can be. Route-origin validation then marks announcements as valid, invalid, or not found.

It doesn't replace your other filters, but dropping RPKI-invalid routes cuts down on a lot of accidental leaks and some hijacks. If you hold address space, create ROAs for it. It takes a few minutes in your RIR's portal.

IRR records

Internet Routing Registry objects describe routing policy and which prefixes belong to whom. Providers use them to build customer prefix filters automatically. The catch is that IRR data is often stale, so use it together with RPKI and direct confirmation from the customer instead of trusting it blindly.

Keeping track of public addresses

IPv4 is scarce and expensive now, so track it carefully: assignments, which customer has what, what's being announced, reverse DNS, abuse contacts, and how full each block is. And for IPv6, give customers generous, clean allocations. Don't carry IPv4 scarcity habits into a protocol that doesn't have the problem.

Customer handoffs

Customers might connect with static routes, BGP, a VLAN handoff, a routed port, a tunnel, or their own VRF. Whatever it is, I want these spelled out:

• Where the demarcation point is
• Who owns which addresses
• What prefixes are accepted and announced
• The max route count
• MTU
• What happens during a failover
• Who's monitoring what

Most arguments during an outage come from one of those being fuzzy.

Keeping customers apart

In a hosting network, one customer must never be able to reach or mess with another. The usual tools: VLANs, VRFs, routed access, private VLANs, hypervisor switch policy, firewalls, and anti-spoofing filters.

Anti-spoofing

Customer-facing ports should only allow source addresses that actually belong to that customer. It stops spoofed traffic from leaving your network, and it keeps you from being part of someone else's reflection attack. It's one of the easiest ways to be a good neighbor on the Internet.

DDoS

Have more than one way to deal with an attack, and set them up before you need them:

• Your upstreams' blackhole communities
• Remotely Triggered Black Hole (RTBH) routing
• FlowSpec, if you have it, and if you're careful with it
• Filtering on your own gear for attacks smaller than your circuits
• Cloud or carrier scrubbing for the big ones
• Flow data and automatic detection

Day-to-day operations

• Watch BGP sessions, route counts, prefix changes, and path shifts.
• Graph interface usage often enough to actually see bursts.
• Collect flows at the Internet and customer edges.
• Back up configs and keep the change history.
• Actually test failover between routers, providers, and power.
• Keep out-of-band console access working.
• Keep escalation contacts for carriers, exchanges, and facilities where you can find them.

My maintenance checklist

1. Record a baseline: routes, traffic, errors, and neighbor state.
2. Make sure console or out-of-band access works before starting.
3. Have the rollback config ready.
4. Drain or move traffic away if you can.
5. Make the smallest change that does the job.
6. Check IPv4 and IPv6 routing separately.
7. Confirm customers can reach things, flows look normal, and monitoring is happy.
8. Write down the final state and anything that didn't go to plan.

Last Modified: October 12, 2026, 11:46 am