"The only real problem is scaling."

"The only real problem is scaling. All others inherent are from that one. If you can scale, everything else must be working." - Mike O'Dell, Chief Scientist, UUNet, MPLS Conference, Nov 1998.

The above was quoted in the book "MPLS - Technology and Applications", by Bruce Davie and Yakov Rekhter, in chapter 8, "Virtual Private Networks", under the topic of "Scaling".

I was reading this book in around 2003. This quote really struck me, as I'd encountered enough instances in past 13 years of working where I'd either encountered scaling limits, or where I'd seen scaling limits overcome.

The other example of a scaling issue described in the book was the scaling problem of router control plane routing protocol neighbor adjacencies across a large ATM network.

In this case, the large ATM network was built to perform traffic engineering at layer 2, while many routers on the edge of the ATM network then formed layer 3 routing protocol neighbor adjacencies across the layer 2 network.

The trouble with this model is that attached to a large ATM network there can be a large number of routers, and there are limits as to how many routing protocol adjacencies a router can maintain. For example, in a normal network a router would typically have no more than perhaps a maximum of 3 to 5 routing protocol neighbor adjacencies, where as in this ATM model, the routing protocol neighbor adjacencies may number in the 100s, because there are 100s of routers attached to the ATM network.

These two things from the book were also significant to me because I'd fairly recently finished working at UUNet (August 2000 - July 2002) and knew the ATM example had come from UUNet.

More significantly, while at UUNet, I worked on their UUsecure product, which was an IPsec based VPN product, which operated over the top of UUNet's Internet service backbone. We had encountered the same sort of router control plane scaling problem.

One of the VPNs being built using the product had 10 000 sites (yes, 10K - I think an unimaginable number for most people who have or are building IPsec based VPNs). For redundancy, each site had two point-to-point IPsec tunnels, individually going to each of two central hubs, meaning 20 000 tunnels and 20 000 routes (1 route for each site, times 2 because they were announced twice).

With IPsec tunnels, the only way to detect if they're working or not is to send traffic over them. IPsec keep-alives weren't available at the time, and anyway, we needed to use dynamic routing to support fail over between the central hubs and any multihomed sites. BGP was the choice, because OSPF wouldn't scale to 10 000 sites and 20 000 routes (so yes, we were using BGP as an IGP, and also had 20 000 BGP sessions, because BGP sessions are also point-to-point).

The trouble was that although the routers in question (Cisco 7206VXRs with NPE-400s if I recall), with the addition of a hardware crypto accelerator, could handle up to 600 IPsec tunnels, once we added the overhead of running BGP and in particular BGP keep-alives over the tunnels, we could only support 200 IPsec tunnels per 7206VXR. If you do the maths, that means a total of 100 7206VXRs to build this VPN were required at the hub sites. I've seen pictures of them .... racks and racks of 5 RU 7206VXRs. (Very costly, but still cheaper and more secure than alternative VPN technologies available at the time.)

If we could have some how avoided all the BGP sessions and corresponding keep-alives, we would have only needed 32 instead of 100 7206VXRs to support 20 000 IPsec tunnels, meaning spending much less money on 7206VXRs, much less power and much less rack space.

In "MPLS - Technology and Applicatons", the authors say that the solution to this router neighbor adjacency scaling problem in the ATM network scenario would be to have the layer 3 routers become adjacent directly with the ATM switches, and have the routers and ATM switches communicate with each other about topology and path setup, including for traffic engineering purposes. This would reduce the number of router routing protocol neighbor adjacencies back down to the normal numbers of no more than 3 to 5.

In other words, the ATM switches effectively become members of the layer 3 routing domain, and the routers effectively become members of the ATM forwarding domain. The layer 2 and layer 3 networks have been flattened into a single network, rather than one network overlaid on top of another.

This is one of the problems that MPLS solves.

During development of MPLS, it was realised that it wouldn't be all that hard to build MPLS based VPNs using a label stack. This probably put an end to large scale IPsec VPNs, despite the loss of Confidentiality, Integrity and Authenticity that IPsec provides and MPLS VPNs don't. I'm sure certain government agencies welcomed the success of unencrypted and unauthenticated MPLS VPNs.

Over the years I've occasionally looked to see if there is work on methods of encrypting LSP traffic, and occasionally there has been. Looking just now, there is a current Internet Draft to do so - Opportunistic Security in MPLS Networks. Still not quite as secure as IPsec, as this would be PE-to-PE encryption, and it is opportunistic, rather than mandatory CE-to-CE/site-to-site encryption (unless the CEs were involved in setting up encrypted LSPs). Also note the draft is Experimental rather than Standards Track.

Hmm, well that has ended up a lot longer than I expected. I was going to use that story as an intro to some observations and thoughts I've had before and since on scaling since reading Mike O'Dell's quote. I'll keep them for another blog post.

To finish on at least some sort of moral, lesson or observation, "flatten to reduce neighbors"! :-)

Actually, I'll finish on something else. I've come to consider the second most important property of a network is the ability to scale it. I'll leave the reader to have a think about what might be the most important property (or at least, what you might think my opinion of what the most important property is).

Lorem Ipsum

"Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum."