Categories
broadband Business H/W internet Net servers

FTTC Broadband — Upgrade Your Router

FTTC installed…and then the problems started.

Once again, Trefor.net welcomes contributor Tim Bray, Technical Director for ProVu Communications. “FTTC — Upgrade Your Router” is Tim’s second “Broadband Week” post.

At ProVu we, don’t often do onsite installations, preferring instead to leave them to our resellers. Sometimes, though, a problem comes along that requires that we get involved in helping to figure out what is going on.

One of our customer’s sites was activated for FTTC broadband. This customer ran an office with a small call centre and about 10 office PCs, and they thought the higher bandwidth would be useful. Zen (the ISP, in this case) had a special offer on ADSL to FTTC upgrades, so the time seemed right for upgrade. Our customer swapped their onsite router out for a model that could do both ADSL and FTTC, and all appeared ready for an easy change over once the Openreach engineer arrived.

ProVu logo

On the scheduled day the Openreach man showed up, and our customer had just 10 minutes downtime while he performed the jumpering in the cabinet. Up came the new 40 Mbps download line (which also had, more importantly, a massive upload speed). Magic. Everything worked, and the internet seemed to be lightning fast. And then the problems started. “The internet is slow!” “We’ve got bad call quality!” And so, a site working properly and perfectly had stopped doing so because of a service upgrade.

We added lots of monitoring. Smokeping and Nagios. Sure enough, we learned of intermittent bad packet loss on the line that came and went, usually at such quiet times as evenings and weekends. We could tell that something was on the network opening a large number of sessions through the NAT in the router, and we knew that the problems started as we got towards 600 TCP sessions. We wondered whether with FTTC when you open a browser window with all your saved tabs the computer would hit those tabbed sites — Facebook, Twitter, Gmail, BBC News and all their associated ad networks and image CDNs — all at the same time, perhaps causing these events to happen too quickly and to throw too many ports open at the same time.

Running just a small consumer type router, we couldn’t diagnose the issue to the point where we could determine what was causing it. As such, as we needed better instrumentation to investigate further, we decided to install a proper linux server as a router in lieu of the dedicated hardware. BT Openreach provides PPPoE termination, so it is easy to deploy standard PC hardware with 2 ethernet cards to act as a router. We used Munin to add every kind of monitoring. We had graphs of UDP sessions, TCP sessions, and traffic graphs for voice traffic against other traffic…you name it, we graphed it.

Everything we could think of that might help us to figure out what was causing the issues being experienced was in place. And it was that moment that the problems went away. Again, magic. Once the new router was installed, everything worked. We saw large throughput and sessions through the router, but no corresponding packet loss. And no user complaints.

Very puzzling.

Then one Saturday I noticed the traffic graph on the router rise up to 30 Mbps download speed and stay there. Not the first time this had happened, of course, but it was the first time I was there to watch. My suspicions were raised, so I phoned the call centre. “No, all our calls are fine.”  The new router was coping with this traffic fine. So I ran Wireshark and discovered that the call centre staff were watching telly using Sky Player on a sneaked-in laptop. And from watching the trace, I could see that Sky Player was streaming the video by opening a new TCP session every few seconds, which coupled with the large number of phone calls must have been what was overwhelming the old router.

I phoned the call centre manager with my findings, and she sussed that they were watching the footie. And regarding a remedy, lets just say some HR Department action occurred!

At this point, let me sum up the learning points:

  1. A bigger router might be needed for FTTC, as the router could be the slowest bit and not the ISP.
  2. The router might have a limit for packets per second.
  3. Even a small office can open a lot of ports through a NAT, something for which small routers cannot cope.
  4. With a good enough router, it is possible to run a small call centre and stream TV at the same time.

As an aside, I think this is a great point where IPv6 would help. IPv4 and NAT is stateful on the router. The router has to record each session and rewrite the packets. IPv6, though, would be stateless, so the router would have only need to pass on the packets rather than having to track sessions and rewrite port numbers. Also, there is the old adage: Use a separate connector for voice to your data. I suspect that some of the poor voice quality that encourages this is actually the voice and data services acting in conjunction to overwhelm the router, rather than there simply not being enough bandwidth. Bufferbloat may be part of the problem as well. But I suspect a router with more grunt may make it so the second line isn’t required.

I’ve done various consultancy jobs to investigate ‘SIP phones dropped off network’, and by scripting to monitor the NAT state table have found the router/firewall just dropping the session from the NAT table, which is obviously either a bug or just not enough capacity in the device.

Editorial note – check out our new site – BroadbandRating.

Categories
broadband End User

The (Hidden) FTTC Wall

Local exchange FTTC-enabled, cabinet within easy view, power and fibre laid down…

Trefor.net welcomes Broadband Week contributor Tim Bray, Technical Director for ProVu Communications.

Here is a small tale about my own company’s experience with FTTC.

ProVu logo

The ProVu Communications offices are in Milnsbridge, which is just outside Huddersfield. As we are heavy broadband users, we were really happy to discover that our local exchange was being enabled for FTTC. A new cabinet appeared directly across the road from our front door. Some BT men came, dug up the road right outside our front door, and laid power and fibre to the new box.

Here is the view from our front door, with the FTTC box across the road. Note the fresh line of the roadworks coming across the road to us.

We eagerly checked on the DSL checker, but our phone numbers never activated. Then we started to dig around, checking the phone numbers of our neighbours.  It seems our phone lines are exchange-only lines, and thus there is no cabinet…and no FTTC. Although just across the road, all the houses and commercial properties have FTTC.

Here is a picture that offers…well, the whole picture. The houses beyond the road junction and past the No Entry signs can all get FTTC. Our office? The red front door on the right.

The problem is that this information isn’t public, and there are no public maps of which lines connect to which cabinets.   If we were a business moving premises, for instance, there is no way we could be sure about getting FTTC in our new location without first ordering a phone line and checking the number.

FTTC can make a massive difference to a business, with availability potentially meaning the difference between using a hosted email provider or installing a server onsite. Or between deploying a hosted phone system versus having to buy an onsite PBX. Or between having workers who can work at home via VPN or requiring that all work be performed from the office at all times.

The moral of the story? Don’t get excited just because your exchange is FTTC-enabled and there is a cabinet nearby. Wait first to see if the BT checker displays “Available”.

Categories
voip

Secured SIP Provisioning

Trefor.net welcomes VoIP Week contributor Tim Bray, Technical Director for ProVu Communications

Most SIP providers in the UK use auto provisioning to look after their SIP phones, with the phones calling home to a central server via HTTP to download configuration files.

Auto provisioning is an essential part of the hosted SIP and SIP PBX market in the UK, which would be unviable without it. The advantages offered are a consistent phone setup and an ongoing ability for the support team to manage and support the device.

Parts of the system

  • Provisioning server
  • Provisioning client on phone
  • Redirection server at manufacturer
  • Multicast detection (for a PBX to detect phones on local networks)
  • File format – usually key/value based or XML

Recent security disclosures (Cal Leeming, et al) have given the impression that all auto provisioning is insecure, the basic argument being that phone MAC addresses are predictable and thus a provisioning server can be easily scanned. I am not sure these disclosures have really brought out anything that was not already understood by the competent players in the market, but they did bring to light the fact that some people are acting in an insecure manner and probably need to tidy up their systems a bit.

SIP usernames and passwords have a value in the underworld of VoIP fraud.

I know from personal experience that security holes in phones cause more damage than exploited provisioning servers, and having the ability to rapidly upgrade thousands of vulnerable phones by way of a provisioning server is invaluable.

At Provu we run a provisioning system for many thousands of phones, and we act as a provisioning service provider for ITSPs who need it. We have always had a policy to only provision SIP passwords one time and then to immediately delete those passwords, and phones that never call home get their passwords deleted as well, all of which provides some level of protection.
ProVu logo

Authentication

It is my view that the provisioning session between the phone and the server should be authenticated. A very good way to do this is to use HTTPS with client certificates (the certificates are for client authentication, with the https encryption almost secondary) that are installed in the phones at the factory. A provisioning server can then use the public part of the Certificate Authority (CA) to authenticate the phone. Each phone has a unique certificate and the MAC address of the phone is embedded as a field within the certificate, and thus a provisioning server can know for certain which phone it is talking to simply by checking the certificate.

The main advantage of the certificate authentication method is that no setup is required on the phone.  The certificates are inserted at the factory and can be validated by anybody with the CA file. Some phone vendors already support this, too, it being an idea that was first put to use by Sipura sometime around 2005.  For years, I have been asking the phone vendors I deal with to add certificates as part of their manufacturing process, and I would very much like to see a world where client certificates are standard on all SIP phones. The certificates can also be used for SIP as well, serving to immediately block an avenue for fraud.

Wider Security

There are many phone configuration best practices that can be enforced by a provisioning server, including:

  • Enforcement of strong passwords on web interface
  • Disablement of dialing from web interface
  • Updating firmware with all the latest security fixes
  • Configuration of SIP on a random port number
  • Disablement of backdoor entry points for click-to-dial software
  • Disablement `hidden` web access usernames and passwords
  • Enforcement of long SIP passwords (much easier to provision a 20 character random password than have the end user type it in)

Provisioning Server Security

  • Use authentication — Must be not replayable
  • Rate Limits — Basic sysadmin firewall type tasks
  • Patched up-to-date with security fixes
  • No directory indexes
  • Use script that deletes passwords once provisioned

VoIP Week Posts: