UnorthodoxY Archive It's wasn't what you thought

Telewest managed to get the TV Drives back up and running after a seven hour outage yesterday. The official line says:

A configuration issue on our network caused this outage.

Sounds fishy to me, non TV Drive boxes weren’t affected so the majority of the network was intact. I guess it’s possible that they tried to push a upgrade to the boxes but it’s bizarre to do it nationwide in the middle of a Saturday. If they did to that and their update server died in the middle of it then I suppose that might hose the boxes until they got the server back.

But who knows what really happened? The problem with these sorts of things is that the customers are rarely really told what went on. In fact the customer services department are rarely told so the end users have no hope. Does it really matter? Do we benefit, as customers, from knowing the details after the fact? Is there any harm in putting the information out there?

I have the Telewest TV Drive system, basically their cable-PVR unit. I’ve been very pleased with it despite some slightly flaky software.

At half past one or so today it started displaying a message that say “Do not switch off your set top box as it is being updated”. Funny time for an update, I thought but I left it as I was supposed to be cleaning the flat anyway. A few hours of cleaning and present wrapping later I come back and it’s still the same. Some investigation on the web reveals that every box that was powered on at the time is similarly affected.

Now I don’t actually have much of an issue with this, it’s slightly annoying and it’ll be more annoying if it nukes all my recorded programs but I’ll live. I have plenty of other ways to distract myself (not least an old non-TV Drive cable box that I can use if I have to). The interesting part for me is the speculation that something has gone so badly wrong that all the boxes are screwed beyond help and will need swapped out (or at least an engineer visit).

So, I’m wondering, imagine you’re in charge of fixing this for Telewest, a good majority of your engineers are probably on holiday, you’ve got tens of thousands of boxes needing replaced or repaired, you may not even have the spares available to do it and if you don’t manage it within the next five days or so you’re the company that ruined Christmas, at least for some people. Let’s, for now, ignore the fact that there are much worse hardships than just having no cable on Christmas day.

What do you do?

Hire more engineers? Send out boxes by post? Start sending out the older boxes? Just start paying people compensation?

What if we go more unorthodox? Start streaming TV-On-Demand for free to people with broadband that are happy with that? Send DVDs (and players) of the big films that were on? Buy everyone Sky? The conspiracy theorist in me is forced to wonder about the timing, if you were a malicious individual acting in the interests of Sky you’d take the network out today — the deadline to order Sky in time for Christmas is the 18th.

What if the boxes are fixable but it’s an “open the case and fiddle” fix? Do you publish the fix instructions online and let people have a bash? Pay people to fix their neighbours’ boxes if they think they can?

All companies should have disaster recover plans, they should cover everything from the mundane to the ridiculous. I’d love to know if “what if we accidentally goose all the set top boxes on the network” is in Telewest’s DRP, and what’s written under it.

I’m guessing it says “Resign”, or maybe just “Pray”.