The idea: one address, many servers
Normally an IP address belongs to one machine in one place. With anycast, the same address block is
announced from many places at once. Here AS64500 announces 192.0.2.0/24 from Virginia (IAD), Frankfurt (FRA) and
Singapore (SIN), and each site runs a DNS server on 192.0.2.53. Nothing in a packet says which site it wants:
every router on the way just follows its best route for the prefix, so a user in Hanoi ends up in Singapore and a user in
Berlin in Frankfurt. The routes come from BGP, the protocol networks use to tell each other which addresses
they can reach.
On the canvas the users' networks are at the top, four transit networks in the middle (each with its BGP table), and the
three sites plus a rogue network at the bottom. The coloured row marked * is a network's best route; its colour
is the site the route leads to. BGP messages fly along the links, one round at a time.
BGP in a few sentences
The internet is made of about 75,000 autonomous systems (ASes): networks with their own routing policy and a number, such as AS64500. Neighbouring ASes keep a BGP session open over TCP port 179. To offer a route, an AS sends an UPDATE with the prefix and the AS_PATH, the list of ASes the route has passed through; each AS puts its own number in front before passing it on. An AS that finds its own number in a path drops the route, which keeps routes from going in circles. When a route is no longer available, a WITHDRAW takes it back (Demo 1: 3 rounds, 16 UPDATEs until every network knows a path).
Choosing the best path
A network usually hears several routes for the same prefix and uses exactly one. Real routers go through a long list; the first steps that matter most are:
| Step | Prefer | Who decides |
|---|---|---|
| 1. LOCAL_PREF | the higher value (default 100) | the network itself: business policy, cheaper links, customers before peers |
| 2. AS_PATH length | fewer ASes | the shape of the internet |
| 3. origin, MED | lower values | hints from the neighbour |
| 4. eBGP over iBGP, IGP cost | the nearer exit | inside the network |
| 5. tie-break | oldest route, lowest router ID | chance |
The page uses steps 1, 2 and a tie-break on the lower neighbour AS number. "Nearest" therefore means the fewest AS hops, which is often, but not always, the closest place. In Demo 4 the LatAm network gives its Europe link LOCAL_PREF 200, and São Paulo's queries go to Frankfurt instead of Virginia: policy comes before path length.
Withdrawals and failover
When a site fails its health check, it stops announcing the prefix and its router sends a WITHDRAW. Its transit network picks the next best route and tells its neighbours; only the users of that site move (Demo 2: Berlin goes to Virginia, 3 AS hops instead of 2). When the site comes back, the shorter path wins again (Demo 3). This takes seconds, much less than waiting for a DNS TTL. If a whole router dies without saying goodbye, its neighbours notice only when the BGP hold timer runs out (90 s by default) unless they run BFD, which detects it in under a second.
Anycast suits short, stateless exchanges such as DNS over UDP: the next query simply goes to the new site. A TCP connection that moves to another site mid-way breaks, because the new server knows nothing about it. CDNs that use anycast for HTTP keep routes stable and accept that rare failovers reset some connections.
Hijacks and leaks
BGP believes what it is told. If another network announces your prefix, some networks will prefer its route. With the
same prefix (Demo 5), only networks that are nearer to the hijacker follow it: here São Paulo, one
hop from the rogue AS. With two more specific halves, 192.0.2.0/25 and 192.0.2.128/25
(Demo 6), every network follows, because forwarding always uses the longest matching prefix, whatever
the AS_PATH. The 2008 YouTube outage (Pakistan Telecom announcing a /24 inside YouTube's /22) and the 2018 Amazon Route 53
hijack (used to steal cryptocurrency) both worked this way. A route leak is the accidental version: a network
passes on routes it should have kept to itself.
RPKI: checking who may announce a prefix
With RPKI, the owner of an address block publishes a signed ROA (route origin authorization): "192.0.2.0/24 may be announced by AS64500, no longer than /24". Networks that do route origin validation mark each route Valid, Invalid or NotFound and drop the Invalid ones. Both hijacks on this page are Invalid: the origin is wrong, and the /25s are longer than the ROA allows (Demo 7). RPKI checks only the last AS in the path; a forged path that ends in AS64500 would still pass, which BGPsec and ASPA aim to fix.
Where you meet anycast
- The 13 root server addresses are served by more than 1,800 machines worldwide, all anycast.
- Public resolvers such as
1.1.1.1and8.8.8.8. - Route 53's name servers and many CDN edges (see How a CDN and Route 53 Work). CloudFront itself mostly steers users with DNS answers; AWS Global Accelerator gives a service two anycast addresses.
- DDoS protection: an attack aimed at one address is spread over every site that announces it.
See also How a Router Forwards a Packet for longest prefix match inside one router, and How a DNS Name Is Resolved for the queries the sites answer.
What the page leaves out
- iBGP inside a network, MED, communities, AS_PATH prepending, and the export rules networks really follow (customer routes go everywhere, peer and provider routes only to customers).
- Timers: real BGP batches updates (MRAI), so convergence takes seconds, and unstable routes are damped.
- ECMP (several equal paths used at once) and how anycast sites spread load inside a city.
- BGPsec, ASPA and the RPKI repositories and caches themselves.
- Real networks check for loops on receipt; here the sender skips neighbours already in the path, which gives the same routes with fewer messages.