“The internet is broken”. Uh? This is a common complaint a) from my wife who isn’t particularly technology savvy and b) from people whose broadband connection has gone down for whatever reason.
Yesterday this complaint, slightly tongue in cheek, came from our development engineering team. US network operator Level3 had a problem with a bug in it’s Juniper routers. This affected access to quite a few websites worldwide and is certainly likely to have hit more than a few ISPs. Word is that this was BGP related and leads to the need to reboot the Juniper kit. In fact a number of ISPs issued emergency maintenance window alerts last night so that they could upgrade to newer versions of their Junos operating system.
Most of the time you can live with a bug – design around it perhaps.I’m sure the problem will get sorted. My observations here though relate to the
security of the internet itself. The network was designed to be resilient but is dependent on the equipment of a handful of large manufacturers to operate.
If you drill into any of these large scale routers there are many bugs (features) identified with each one at any given point in time. These are complicated bits of kit running many interdependent software protocols. It isn’t possible to design one that is bug free especially as technology is advancing at such a rapid pace and networks grow ever more complex. Sometimes you only find out about a bug under certain conditions – eg when traffic through a router reaches certain levels. Most of the time you can live with the bug – design around it perhaps. At other times a bug develops into a critical problem and this usually results in an emergency patch by the vendor of the particular router. To avoid this dependency some networks are designed as two totally different networks based on different vendor kit – LINX (London Internet Exchange) for example which carries so much traffic that a total outage would have a huge knock on effect. This is an expensive option.
Where am I getting to here? The Level3 outages today show just how vulnerable the world could be if an unfriendly entity wanted to cripple the global economy. It’s a racing certainty that many engineers out there know how to go about it, just by exploiting known weaknesses in routers.
We are dependent on these engineers remaining good guys – the ISP industry is by and large staffed by folk who pride themselves on their professionalism and integrity.