End User fun stuff servers

BT speaking clock is 80 years old on Sunday

At the third stroke lets all sing happy birthday

I usually ignore the zillions of press releases I get in my inbox. I made the mistake of once agreeing to go on some PR database and I get lots of crap from people I’ve never heard of.

On this occasion however I am going to republish verbatim the whole press release because I find it of interest. I’ll just add that it would make sense to me to provide an octogenarian voice to the clock for the day. Something along the lines of “hello dearie, at the third stroke it will be time for my weak tea and a biscuit”. All spoken in a shaky voice.

No offence intended to the many fit and healthy octogenarians still in possession of all their teeth and faculties.

Whilst feeling nostalgic and warm towards the speaking clock I must say it is probably thirty years since I rang them. Who needs it with the time on your phone and pc being right on the beep.

Anyway here’s the press release – happy birthday to the speaking clock.


Speaking Clock celebrates its 80th birthday on July 24, 2016

Audio and images can be found here

Britain’s famous Speaking Clock celebrates its 80th birthday on July 24, 2016. Now a national institution and part of Britain’s heritage, the Speaking Clock was the first of the pre-recorded information services in the UK, provided through telephones.

Created for people who wanted to know the time and did not have a watch or clock to hand, the clock was initially only available in the London directory area, with the first British Speaking Clock introduced on July 24, 1936.

The Speaking Clock was designed and constructed at the Post Office Engineering Research Station at Dollis Hill in North London. The time announcements were automatically co-ordinated on the hour with Greenwich meantime signals.

In order to access the service, subscribers would dial the first three letters of the word ‘time’ as dials at the time included letters as well as numbers to aid automatic calls. Dialling T. I. M. led to its common name ‘TIM’. The service went national six years later.

David Hay, head of BT Heritage, said: “The BT Speaking Clock is a national treasure. Even though we live in the digital age, more than 12 million calls are made each year to the BT Speaking Clock to get an accurate time check.

“Eighty years ago BT’s technology created the Speaking Clock which remains a much loved part of British life today. The Speaking Clock has reached octogenarian status and celebrating its birthday demonstrates BT’s determination to preserve the heritage of the world’s oldest communications company.”

Jane Cain was the first voice, winner of a Post Office ‘Golden Voice’ competition, and used from 1936 until 1963. Pat Simmons, a London telephone exchange supervisor, became the second voice from 1963 until 1985. The third voice belonged to Brian Cobby who became the first male voice at 11am on April 2, 1985. An actor by profession before he joined BT as an assistant supervisor at a Brighton exchange, Brian was selected from 12 finalists in BT’s competition on December 5, 1984. Users who were around in the 1960s who listen hard enough might detect a familiarity – Brian was also the voice of “5-4-3-2-1 Thunderbirds are go!” in the famous Gerry Anderson TV series.

The fourth and current voice is Sara Mendes da Costa from Brighton & Hove. She became Speaking Clock voice at 8am on April 2, 2007. Sara won a BT competition during 2006 to find a new voice from the public, which had almost 18,500 entrants, simultaneously raising more than £200,000 for BBC Children in Need.

Sara Mendes da Costa, said: “I am very proud to be the fourth permanent voice for the Speaking Clock and have been since April 2, 2007, nearly ten years ago.”

Originally the accuracy of the BT Speaking Clock was one-tenth of a second, but it is now accurate to within 30 microseconds.


Permanent voices


First voice             Jane Cain                                1936 – 1963

Second voice        Pat Simmons                          1963 – 1985

Third voice                        Brian Cobby                            1985 – 2007

Fourth voice          Sara Mendes da Costa           2007 – to present


Quick facts


  • The BT Speaking Clock has been ticking 24-hours a day, seven days a week since 24 July 1936 – which is 80 years, more than 29,000 days, more than 700,000 hours or more than 42 million minutes, more than 2.5 billion seconds
  • Big Ben checks its time with the Speaking Clock
  • The Speaking Clock is accurate to within 30 microseconds
  • In its first year the service registered nearly 13 million calls
  • Initially only available in the London area and went nationwide in 1942
  • The Speaking Clock is also known as TIM and Timeline


Temporary voices


There have been a number of temporary Speaking Clock voices, recorded for charity:

Lenny Henry: March 10 to March 23, 2003 (Sport Relief)

Alicia Roland (12-year-old schoolgirl): October 13 to October 23, 2003 (Childline)

Mae Whitman: October 26, 2008 until February 9, 2009 (to promote Disney’s Tinker Bell)

Kimberley Walsh, Cheryl Fernandez-Versini, Gary Barlow, Chris Moyles and Fearne Cotton: February 3 to March 23, 2009 (Sport Relief)

David Walliams, Gary Barlow, Chris Moyles, Kimberley Walsh and Fearne Cotton: March 7 to April 9, 2012 (Sport Relief)

Clare Balding: February 12 to March 15, 2013 (Comic Relief)

Davina McCall: January 27 to March 23 2014 (Sport Relief)

Sir Ian McKellen: February 24 to March 13, 2015 (Comic Relief)

Jo Brand: January 20 to March 20, 2016 (Sport Relief)

broadband Business H/W internet Net servers

FTTC Broadband — Upgrade Your Router

FTTC installed…and then the problems started.

Once again, welcomes contributor Tim Bray, Technical Director for ProVu Communications. “FTTC — Upgrade Your Router” is Tim’s second “Broadband Week” post.

At ProVu we, don’t often do onsite installations, preferring instead to leave them to our resellers. Sometimes, though, a problem comes along that requires that we get involved in helping to figure out what is going on.

One of our customer’s sites was activated for FTTC broadband. This customer ran an office with a small call centre and about 10 office PCs, and they thought the higher bandwidth would be useful. Zen (the ISP, in this case) had a special offer on ADSL to FTTC upgrades, so the time seemed right for upgrade. Our customer swapped their onsite router out for a model that could do both ADSL and FTTC, and all appeared ready for an easy change over once the Openreach engineer arrived.

ProVu logo

On the scheduled day the Openreach man showed up, and our customer had just 10 minutes downtime while he performed the jumpering in the cabinet. Up came the new 40 Mbps download line (which also had, more importantly, a massive upload speed). Magic. Everything worked, and the internet seemed to be lightning fast. And then the problems started. “The internet is slow!” “We’ve got bad call quality!” And so, a site working properly and perfectly had stopped doing so because of a service upgrade.

We added lots of monitoring. Smokeping and Nagios. Sure enough, we learned of intermittent bad packet loss on the line that came and went, usually at such quiet times as evenings and weekends. We could tell that something was on the network opening a large number of sessions through the NAT in the router, and we knew that the problems started as we got towards 600 TCP sessions. We wondered whether with FTTC when you open a browser window with all your saved tabs the computer would hit those tabbed sites — Facebook, Twitter, Gmail, BBC News and all their associated ad networks and image CDNs — all at the same time, perhaps causing these events to happen too quickly and to throw too many ports open at the same time.

Running just a small consumer type router, we couldn’t diagnose the issue to the point where we could determine what was causing it. As such, as we needed better instrumentation to investigate further, we decided to install a proper linux server as a router in lieu of the dedicated hardware. BT Openreach provides PPPoE termination, so it is easy to deploy standard PC hardware with 2 ethernet cards to act as a router. We used Munin to add every kind of monitoring. We had graphs of UDP sessions, TCP sessions, and traffic graphs for voice traffic against other traffic…you name it, we graphed it.

Everything we could think of that might help us to figure out what was causing the issues being experienced was in place. And it was that moment that the problems went away. Again, magic. Once the new router was installed, everything worked. We saw large throughput and sessions through the router, but no corresponding packet loss. And no user complaints.

Very puzzling.

Then one Saturday I noticed the traffic graph on the router rise up to 30 Mbps download speed and stay there. Not the first time this had happened, of course, but it was the first time I was there to watch. My suspicions were raised, so I phoned the call centre. “No, all our calls are fine.”  The new router was coping with this traffic fine. So I ran Wireshark and discovered that the call centre staff were watching telly using Sky Player on a sneaked-in laptop. And from watching the trace, I could see that Sky Player was streaming the video by opening a new TCP session every few seconds, which coupled with the large number of phone calls must have been what was overwhelming the old router.

I phoned the call centre manager with my findings, and she sussed that they were watching the footie. And regarding a remedy, lets just say some HR Department action occurred!

At this point, let me sum up the learning points:

  1. A bigger router might be needed for FTTC, as the router could be the slowest bit and not the ISP.
  2. The router might have a limit for packets per second.
  3. Even a small office can open a lot of ports through a NAT, something for which small routers cannot cope.
  4. With a good enough router, it is possible to run a small call centre and stream TV at the same time.

As an aside, I think this is a great point where IPv6 would help. IPv4 and NAT is stateful on the router. The router has to record each session and rewrite the packets. IPv6, though, would be stateless, so the router would have only need to pass on the packets rather than having to track sessions and rewrite port numbers. Also, there is the old adage: Use a separate connector for voice to your data. I suspect that some of the poor voice quality that encourages this is actually the voice and data services acting in conjunction to overwhelm the router, rather than there simply not being enough bandwidth. Bufferbloat may be part of the problem as well. But I suspect a router with more grunt may make it so the second line isn’t required.

I’ve done various consultancy jobs to investigate ‘SIP phones dropped off network’, and by scripting to monitor the NAT state table have found the router/firewall just dropping the session from the NAT table, which is obviously either a bug or just not enough capacity in the device.

Editorial note – check out our new site – BroadbandRating.

Engineer engineering internet peering servers

The 3rd LINX router modelled by @neilmcrae and Keith Mitchell #LINX85

So you think you know your routers? This SPARCstation,,  was LINX’s 3rd router installed at Demon in Finchley (AS2529) in 1994. Before most of today’s ISPs were a twinkling of a Microsoft egg timer.

The SPARCstation is modelled by Keith Mitchel and Neil McRae who I realise don’t look old enough to remember those days but you would be wrong:)

The lads might be able to enlighten us re the throughput capability and route capacity of this box. It would be a far cry from the 100Mbps 100Gbps toys that Neil plays with nowadays at BT. It probably didn’t need to support more than 20 routes!

Neil is holding the router, Keith has the cup of tea. Note that the box is being held higher up than the cup of tea. That’s in case Keith drops the cup – safety in mind.:)

One also wonders at which point racks were standardised at the U dimensions they have today. Many an ISP had rows of tower PCs stood on metal shelves. Of course U’s these days are often Virtual.

linx85 keith neil

SPARCstation IPX

Business servers

My name is Andy and I work for Tesco

Walking to the station this morning en route to an ITSPA meeting in Town I noticed a young suit staring into his phone.

As I got closer I saw that his name was Andy and that he worked for Tesco. His name was displayed on a badge on his lapel. I’m not sure what font size Tesco use but it did the job. I could see it clearly without having to stop and peer.

Don’t ask me why this stuck in my mind, other than the fact that I made a note of his name using my voice recorder. I do that sort of thing.

Continuing with the thread, I used to attend dinners thrown by Dell. A guest speaker entertained and we would have an after dinner debate on the theme of the evening. They were good dinners fair play to Dell.

My only gripe was the size of the font on the name badges. It was far too small to be able to easily read the name, especially considering these dinners were held in private dining rooms dimly lit for atmospheric effect. Clearly labelled badges are important if you are in a room full of strangers with lots of wine flowing. How do they expect me to remember names after all that wine. After one of the dinners I completed the assessment form and said all was good except for the badges.

At the next Dell dinner the badges were the same. No change in font size. Far too small to read. I’m sure it was the usual excellent evening but at the end of it I refused point blank to provide feedback. What was the point? They obviously didn’t read the feedback. Either that or they didn’t consider my feedback worth responding to.

Taking feedback to the extreme one of the readers of this blog was at a trade show in London looking to buy a server. He hung around the HP stand waiting to be sold a box. No sales pitch came forth and in due course, after having his badge scanned and informing HP of his enquiry, left serverless and bought one off IBM just down the aisle.

A few weeks later he received a phone call from a HP sales person following up on the exhibition lead. He related his story, told them they were too late and considered the matter closed.

Wind the clock forward another few weeks and HP were back in touch again. The PA of the VP running the HP server division wanted to know if he would have lunch with the VP to provide feedback of his experience at their trade show booth. Sure said my friend. Anytime.

Only problem is the lunch was to be at HP’s Corporate HQ in California!!! They flew him out a couple of days early and he had lunch with said VP in their company canteen. The whole thing lasted 90 minutes and then they flew him home. I’m sure he had a good time.

One wonders what effect his feedback may have had. I’m also sorry the VP must remain nameless. That’s because I can’t remember his name – nothing to do with the  font size of his badge.

Other server posts:

2 out of 7 Lloyds Bank servers down
Cisco UCS with 96GB of RAM
Telegraph and UPS DNS servers hacked

Engineer fun stuff servers

Stop Looking At My Finger, Silly Dog

My friend manages to piss me off and it’s not yet 8:00 in the morning. A thought flashes across my brain, “Maybe he’s not the root cause.”

This morning, the first text I see: Priority 1 emergency ticket.

Damn. I hope the server isn’t down.

As usual, my alarm went off this morning at 6:00. I look at my phone and see a text message that was sent late last night, an automated message from our Linux server passing along a support ticket. A surge of anxiety rushes through me. “Oh no, I forgot to re-enable the submit button in our application!” A suspicion, though I don’t really know yet for certain. I jump out of bed.

I check the emergency ticket’s timestamp. About 10 hours ago. “Oh please,” I pray-but-do-not-say as I head for my computer, “Please let it be that someone jumped in and handled this last night.”

Last night we performed a software upgrade to the server. The entire team on the phone, dialing in from multiple geographies, talking through headsets and typing commands on the server, a virtual everyone-from-home Mission Control. All talking on the same call, as we’ve done dozens of times before. The launch pad ready, countdown begun, I was walking us through the checklist and then gave the go ahead to deploy. But wait! There’s an error, a build error that is picked up by the programming group doing the deployment. We investigate and discover that the server is offline and end users are waiting. Finally, we call it. Abort.

Engineer media servers

BBC website down – Error 500 – Internal Error

bbc error 500BBC website down

It ain’t often you can’t reach the BBC. They have 700Gbps of connectivity1 to their servers. As one of the world’s foremost media organisations their website will rank as the most robust going.

It comes as a mild surprise therefore to see the error message on the screen in front of me -“Error 500 – Internal Error”. Is the BBC website down? The little clown icon is quite cool and totally in keeping with the creative nature of the BBC. You can almost hear the clown laugh as if this is some macabre late night feature. The audience is spellbound, silently gripping their seats.

The hiccup was over in a moment. It’s one of those glimpses of an event in life that you think back on and wonder if it really happened except that here I have the screenshot.

It isn’t as big deal but my curiosity is aroused and I think to myself it would be quite nice nice to understand what actually happened. What architecture does the BBC use for its website and what caused the error message? It’s one of those things that in real life is not worth wasting time drilling down to a root cause. Still if anyone can chip in it would be nice:)

Note added 20th July. I’m guessing this happened again yesterday as there was a huge spike of visitors to this site with people reading this post around 11am.

Other great Beeb reads:

Tablets shifting our viewing habits
A visit to Broadcasting house
BBC promo int

1 last time I asked.

datacentre dns Engineer internet servers

Diagnosing very slow website loading problem

downtime_graph_smallBeen having intermittent problems with since moving the site to a new virtual platform at Christmas. It’s all sorted now. Thanks to the lads at the Timico Datacentre.

I asked Ian Christian to describe the issue and how it was resolved:

Well… explaining it is a little hard…. The key to figuring it out was this:

At the bottom of every page it shows when the page was generated, and how long it took. I suspect in wordpress somewhere it might have told you this too – but I’m not sure.

What we were seeing was

Engineer servers

Is this a symptom of the recession or has the Moore’s Effect plateaued?

Moore’s Law predicts that advances in device fabrication double the number of transistors possible on any given piece of silicon approximately every two years.

This has lead to a constant performance increase in computing over the last 40 years. That is, a near-linear improvement in CPU speed and storage capacity.

It has also lead to another notable effect – a rapid depreciation in the cost of new hardware over the first couple of years after a device first goes on sale.

For example I built a server in 2005. Within 2 years the price I paid for the motherboard, CPU and RAM had collapsed to less than half what I paid initially. In fact it allowed me to upgrade the CPU, RAM and hard disks to give a vast performance increase for less than I paid for the original components.

Fast forward 5 more years to early summer 2012 when I built my last server. 18 months later the CPU and motherboard are still on sale. But the i7 3820 CPU is today exactly the same price I paid for it and the X79 chipset motherboard has actually increased in value, from just under £200 to over £260.

Even the Ripjaws 16GB quad-channel memory kit I bought has gone up 20 quid, something quite unusual after years of tumbling memory prices.

In fact the only notable depreciation is in the cost of enterprise-class hard disk storage, due to a couple of high performance 4TB disks entering the market in the intervening period.  So I might just be able to afford to upgrade from 2TB to 4TB RAID-1.

Of course this could just be a temporary blip due to a combination of the severe economic down turn we’re [hopefully] heading out of and the rapid decline of the desktop computer reducing the demand for traditional motherboards and leading to a temporary glut in the market in 2012.

However it could be an economic indicator that the Moore’s Effect is plateauing, meaning computer hardware will hold its value for longer and I’m denied the cheap performance boost I’d grown used to 2 years after I build a server.

Engineer security servers

Lloyds Bank – 2 out of 7 servers “down”

Problems with Lloyds Bank & TSB cashpounts attributed to failiure of 2 out of 7 servers by BBC.

Interesting article on BBC Radio 4 Today Prog this morning. Apparently last night some Lloyds & TSB customers were unable to use their debit cards for a couple of hours or so. Not me. I was at home.

The point is that apparently two servers were down. It’s a bit of an eyebrow raiser that this could happen with just two servers going down. Doesn’t sound like good capacity planning. I’d have thought they’d be load balanced with plenty of headroom on each server that would allow for such an eventuality.

Can’t be right unless there’s something specific re security for such systems that doesn’t allow them to do that.

One wonders what would have caused two servers to go down at the same time. Rack outage maybe? No generator bup? Suspect we won’t find out and I’m only mildly interested.

The other observation relates to that comment by the reporter re people at petrol stations whose cards were rejected.  Unless they had alternative means of payment they had to wait at the petrol station until someone came along and paid for them.

Petrol stations in my experience can fail over to a manual card swipe using old fashioned slips of paper. Maybe not all of them. Or maybe because the card processing system was not “down” generally the specific Lloyds customers weren’t trusted.

That’s all.

PS no such thing as 100% uptime – see this post on Vodafone outage

Cloud Engineer servers

What will you do if you die before me?

note that came with Cisco UCS chassis

If I die before u i gonna write your name on the twinkling stars to show the world how much i love u

I could just leave it at that but I’m sure you want to know more.  This was a note found in a Cisco UCS  chassis when it was delivered to the new Timico data centre in Newark. Funny  huh?!

Well with the same shipment came a UCS B200 blade that was DOA (Dead On Arrival).  Not so funny huh? Fortunately BT iNet replaced it with another in short order – well done BT iNet.

I would expect there is an investigation under way at Cisco. It is too much of a coincidence for it to be accidental.

It takes me back to my time in the semiconductor industry where chip designers used to leave little messages etched in metal in spare areas of silicon. One guy I know, who shall remain nameless, etched “live fast die young”. This was only discovered after they made the chip. Unfortunately it was on a high reliability product expected to last a very long time. They had to redo the metal mask. Expensive.

Being intelligent enough to hold down a job as a semiconductor designer he did get his grammar and spelling right unlike whoever wrote this note:)

I do quite like the idea of writing things in the stars though. The concept is doable. You just have to blank out the stars that you don’t want to appear in the writing. I leave it to you to decide how you go about doing that (micro-managing ain’t my thing).

The message would only be viewable from a certain footprint on earth but I’m assuming here that it is intended for a loved one whose location would be known to you. Let us know.  When you’re dead you can do anything, or not as the case may be.

If I were you I’d forget about it, live long and enjoy it while you can 🙂

Cloud Engineer servers

Cisco UCS B200 blade server with 96Gigs of RAM #itsoktogetexcited

Cisco UCS B200 blade serverIf you have ever stood in PC World and wondered whether to Cisco UCS B200 blade server
go for the the 4Gigs of RAM with 1Terabyte hard drive or the 6 Gigs which is a little bit more expensive then cop this baby.

Personally I opted for the 96Gig of RAM with dual 10 Terabyte hard drive (expandable) and dual 6 core Xeon processors.  You get my drift?

We have ten of these in two chasses of 5 just to get going. Note the Cisco UCS B200 blade server does come with two onboard hard drives but the discerning host will of course ignore these if he is at all interested in MTBF. Everything else on the board is solid state.
Cisco UCS B200 server blade in a chassis at the Timico Newark data centre

The next picture is of the chassis front and back.

Front view just has the one blade in it at the time the picture was taken.

You can see the four power supplies (N+2) in a line below the blade (circled on the larger photo – click to see). The last two photos are the rear of the chassis fully populated with fans – cool I thought ( 🙂 ) and the schematic that is printed on top of each blade – it will interest someone.

I think these fans in the back of the Cisco UCS B200 blade server chassis at the Timico Newark data centre look really cool :)schematic diag of Cisco UCS B200 blade server t the Timico Newark Data centre

Cloud Engineer servers

This networking kit is good enough for Jehovah

Timico,data center,Juniper,MX80,SRX 3400The word Juniper always makes me think back to the Monty Python and The Life of Brian movie where a hermit has been living on juniper berries for years and Brian’s followers trample all over his bush.

Juniper has a totally different connotation these days, at least when we talk networking. Clicking on the header photo will reveal some equipment we are currently playing with in the lab before it goes live in the new data center core network.

Timico’s core network actually encompasses multiple vendor equipment but there are a few neat things

dns Engineer online safety security servers

Telegraph Register and UPS DNS servers hacked

The Register DNS hackedIf you have been trying to access the telegraph online or TheRegister tonight you might come in for a bit of a surprise as the sites look as if they have been hacked.  More specifically it looks like some  Domain Name Servers have been hacked, diverting traffic to other pages.  Many people will not notice.

Click on the header to see more of what the Register site currently looks like. At this point in time the hack is less than 30 minutes old so I don’t have any more info but if I get a chance I’ll update the post as news comes in. Or just Google it. I saw it first on Twitter.

Business mobile connectivity servers

New Blackberry Enterprise Server

RIM has announced its latest upgrade to the Blackberry Enterprise Server. BES5 notionally provides a number of improvements (one might reasonably expect! 🙂 ) but one in particular caught my eye.

A BES sits LAN side of a corporate network and access to it is via an encrypted 3DES (or higher) path. Being LAN side is allows useful access to a company’s intranet. However what it didn’t do, or at least not without the involvement of a third party application, was to give access to computers on that LAN.  This meant that accessing data on corporate servers was not straightforward. 

With BES5 you can also access attachments within calendars. This is very useful in my mind. I often store location information for meetings in my calendar but my Nokia E Series phones don’t provide me with access to any of the notes. At least if they do I can’t see how.

I am indebted to one of our Blackberry gurus Will Curtis for the BES update. He has his own mobile gadget oriented blog with a post on this subject if you want to know more.

Engineer servers

Virtual Server Virtuosity

At Timico we recently installed a complete network solution for a customer in the UK. The requirement included installation of a domain controller, file and print server, Microsoft Exchange 2007, Microsoft SQL server various databases and for their document management system and a Citrix ZenApp for home workers to run the document management system remotely.

The company also needed to store lots of documents. They have a paperless office and all documents are scanned in by the document management system which required a redundant Storage Area Network (SAN).

100% uptime or as near to this as possible was also wanted but this came in tandem with a fairly tight budget which isn’t always consistent with high reliability.

The architecture that the Timico team came up with involved running all servers and the SAN in a virtualised environment. In this way the design challenge could be met by using only two physical servers called nodes that provided a fully load balanced and virtually clustered redundant solution.

By doing it this way we saved rackspace (5U) and power and 2 servers – we would otherwise have been looking at a pair of virtual servers and a pair of SAN servers.

Did it work? In the first week a hardware problem caused one of the 2 server nodes to temporarily fail. This was picked up by Timico’s monitoring desk but the customer, however, did not notice or experience any loss of service.

I’m Virtually Certain that this is the way forward.