dn42

dn42 is an imitation of the Internet, for the purpose of learning about the internet. Whereas the real Internet consists of high-speed routers connected by fiber-optic cables, DN42 consists of low-cost virtual servers connected by agreed virtual tunnels based on the real Internet, so it's very cheap to join and experiment with.

I was inspired by burble (link requires dn42 connection) to publish how my network, AS4242421855, is designed.

This page assumes you don't know much about DN42, so a lot of the information is redundant.

"Hardware"

Like most DN42 participants with non-trivial networks, I have many virtual servers spread across several continents, running bird2 and interconnected with Wireguard virtual links. I don't connect every node to every other node because that makes for a boring network. Instead I use a network shape that makes some rough geographical sense and also respects the traffic limits of the various servers (even though dn42 traffic is unlikely to hit any limit).

All of my nodes run Linux, and dn42 occupies a separate network namespace from the default. This means that dn42 traffic is strictly segregated from all other networking on the device. Some people choose to run dn42 together with the internet, but I don't.

Each node and tunnel has to be individually configured, although I have some scripts to automate it.

One of my hosting providers implements a private network interface between virtual servers owned by the same customer at the same location. I have two of these virtual networks, and the whole virtual network is treated as part of DN42, without additional Wireguard encapsulation.

Addressing and routing

The network is primarily IPv6. Out of my overall /48 block, each node gets a /56. A few IPv4 addresses are allocated ad-hoc. IPv4 traffic can be carried between IPv4 peers, or between an IPv4 peer and one of my IPv4 addresses, even through IPv6-only nodes. I use OSPFv3 as an IGP, even for IPv4. OSPFv3 is only for IPv6, so each IPv4 address is mapped to an IPv6 address in a reserved /96 address block. OSPFv3 calculates routes to these mapped addresses, and they are reverse-mapped using ospf6to4. IPv4 traceroutes passing through IPv6-only routers see the dummy address 192.0.0.8.

Peering with other networks

OSPF handles routing within my own network (autonomous system) so it appears as a coherent whole from the outside. Routes received from other autonomous systems are handled by BGP (as usual).

BGP is fundamentally a point-to-point protocol that runs separately on each peer connection, not a single distributed system like OSPF. This is why it's used on connections between networks - it allows networks to set rules at their border. This isn't as much of a thing in DN42 as it is on the real Internet. For example, on the real Internet, a connection between an ISP and a large customer would typically be made to only allow routes to that customer's address ranges and ASN. If the customer advertised a route to another ISP, it should be blocked.

The only inviolable rule is that you mustn't cause a loop. This is enforced using the path attribute of each route. It contains the ASNs which the traffic will go through. Networks must reject routes that already contain their own ASNs, except in very unusual situations, and they must add their own ASN.

The same BGP protocol is used inside the network to make each router aware of BGP routes received by other routers from their peers. This is known as IBGP (internal BGP), not to be confused with IGP. Since internal connections do not add to the path attribute, a different method is used to avoid routing information loops. A basic configuration for IBGP is for every router to connect to every other router via IBGP. To avoid routing information loops, when a route is received on a router through IBGP, it does not redistribute it back through IBGP. A slightly more advanced configuration is for every router to connect to a route reflector - a designated router which does redistribute IBGP routes. Loops are avoided because non-reflector nodes only connect to the reflector. It is possible to use multiple route reflectors, for redundancy. In my network I am using two.

All routers (including those without external connections) receive the same external route information from the route reflector. It is important they come to the same conclusions about the best external routes, or there can be a routing loop. This is assured by default as they all run the same protocol, but I don't like the implied brittleness, so I would like to find an alternative in the future. It's possible to avoid any possibility of routing loops if each router tags each externally-destined packet with the specific external route it thinks that packet should take, and then other routers obey that tag. This can be done with MPLS, but running MPLS over Wireguard introduces a lot of overhead. It could also be done with some completely custom protocol.

All of this is a very standard arrangement and many real networks work the same way.

I have future plans to build a virtual internet exchange at Hetzner since traffic inside Hetzner's network is completely free and unlimited.

Geographical network map

Logical network map (easier to read)