ECIX started as a classical Internet Exchange in Berlin, Germany. We tried to sell peering ports to ISPs, carriers and hosters on our platform, hoping to give them some advantage over buying ip transit. Mostly smaller companies made use of it. Shortly after Berlin, we opened an IX in Düsseldorf, which is the biggest internet hotspot on the biggest urban agglomeration in Germany.

Obviously having local peers from carriers with end customers and datacenter operators with content is the best mix. There were people offering content and people eager to consume it.

How does peering work? You run a service called Border Gateway Protocol (BGP) on your edge routers installed at the demarcation points of your AS. BGP manages the connections between the AS edge routers and is responsible to build a table of IP prefixes. This is the Routing Information Base (RIB), summing up as much as possible IP addresses under one entry by aggregation. BGP also is responsible to make a selection of the best path between two AS networks based on rules und communities. After the best bath has been chosen, you will forward this information to the forwarding daatabase used by the router to move IP packets.

An ISP is normally connected to more than one AS. In the beginning you will buy one link from a bigger AS and connect the second port to an IX. You are doing so, because you will not have all worldwide available prefixes on an IX. Therefore you will need a default path for all the AS you can not yet connect to via the IX. The IX traffic should normally be much cheaper than the IP traffic you are buying from a bigger AS. This is because you are paying only for the physical link to the IX port and the management fee at the IX.

One day one of our bigger partners offered us connectivity between Berlin and Düsseldorf. This was the starting point of our remote peering product. Starting small with just a Gigabit-Ethernet port and a few hundred Mbps of traffic, customers started to use that service. Especially smaller ISP were happy to take part and by that way they extended their service reach to Berlin or Düsseldorf respectively.
Over the last four years we expanded this service to new locations like Hamburg and Frankfurt. We upgraded our transport backbone from pure Ethernet-transport between the cities carrying only VLAN to MPLS/VPLS transport over rented wavelength.

To be clear about it, the ECIX POPs are working standalone with their own peering mesh and peering IP address space. The remote peering product connects the customer using a virtual ethernet link to that local peering mesh.

With enough experience and knowledge in remote-peering, we searched for new opportunities in Europe and found LU-CIX to be interested in connecting Luxemburg to Düsseldorf and the rest of the ECIX POPs. Later that year we joined AMSI-X remote peering, with competitive pricing and a large number of customers using this service right from the start.

What we did wrong: definitely using a layer 2 Ethernet Backbone link to connect the peering lans. This was a hard time, filtering broadcast traffic, counting local traffic, etc.
What we did right: switching to MPLS/VPLS and using virtual point-to-point Ethernet links between the remote customer and the local peering LAN.

Any issues? Yes there are some unsolved things to be fixed. First of all, the unbalanced selling of ports. Normally the customer of a smaller IX buys remote peering, not the other way around. Second the BGP routing must be very consequent on the customer edge routers. Not sending traffic over long distance links, if the peer is also present on a IX nearby. We try to help with our route-servers and sophisticated BGP communities.

Would we do it again? Yes! Remote Peering is one of our main features and we are continuing to find new interesting remote IX to the benefit of our customers.

Other peering week posts you might like to read include:

UK internet history – The Early Days of LONAP by Raza Rizvi
INEX’s IXP Manager – Tools to help manage an Internet Exchange by Barry O’Donovan
Regional Peering in the UK by James Blessing
Co-operation makes internet exchanges future proof by Pauline Hartsuiker
Experience of launching an IXP in North America by Ben Hedges
The evolution of an IXP network engineer by Rob Lister
Why does Scotland need an Internet Exchange? by Charlie Boisseau