Ahahahahaha, Ahahaha, it was the vacuum cleaner at BA!

The incident is thought to concern an uninterruptible power supply (UPS) which delivers a smooth flow of power from the mains, with a fall-back to a battery back-up and a diesel generator.

This week BA’s parent company, International Airlines Group (IAG) admitted that the supply to Boadicea House, a data centre, was temporarily lost. An internal investigation found that the UPS, believed to have been supplied by Hitec Power Protection, was functioning correctly at the time.

One BA source said it was rumoured that a contractor doing maintenance inadvertently switched the supply off, although this has not been confirmed.

An internal email from Bill Francis, head of group IT at IAG, appeared to confirm this version of events. The email, leaked to the Press Association, said: “This resulted in the total immediate loss of power to the facility, bypassing the backup generators and batteries . . . It was turned back on in an unplanned and uncontrolled fashion, which created physical damage to the system.”

Snigger.

29 comments on “Ahahahahaha, Ahahaha, it was the vacuum cleaner at BA!

  1. I hope the full story comes out, they are always fascinating. El Reg is full of stories and comments about such things. Emergency stops next to light switches, diesel pumps for backup generators being powered by the now failed mains and so on.

  2. “…an uninterruptible power supply (UPS) ….

    …a contractor doing maintenance inadvertently switched the supply off…”

    or….an uninterruptible power supply was interrupted.

  3. Something similar happened to me way back in the early 90s.

    Small company, new fangled PC, new software & modem so the PC could actually talk to the bank overnight. We spent a whole week wondering why the overnight communications kept failing until we discovered the cleaner was unplugging it.

    I was the office junior – don’t look at me.

  4. The legendary Digital Equipment Corp had a similar (though non-destructive) event happen in their DEC Park facility in Reading. A building maintenance guy pressed the red button to exit the computer room, except of course it was the emergency power-off button. The plastic covers meant to prevent such accidents were promptly fitted following this outage, after previously just lying around in a box.

  5. The classic story–which must be an urban legend as even the NHS couldn’t be that bad–is of the ICU where patients are found dead in the morning on a far too regular basis.

    They eventually discover that the cleaning woman unplugs the life support unit while she uses the plug to power her hoover, floorwasher etc–and then plugs the life support back in back in and leaves.

  6. The story seems to have originated in South Africa. Here‘s a report from the New Scientist, which used the term “apparently true” to mean “we couldn’t be bothered to check”.

  7. Any proper data centre will have two sources of power. Each source of power will come from a separate electrical sub-station and have its own UPS & diesel backup.

    Then every device in the DC has two power supply units, one plugged into each source.

    If one substation goes down, one UPS fails, one idiot presses the big red button, the systems keep running on the other supply. Hence the term redundant.

    Another point – I’ve never seen a big red button in a DC. Normally it’s a massive breaker switch you have to turn, and you don’t do that by accident. The only device that can normally cut the whole supply to a DC is the fire control system.

    Having said all that, anyone remember the story of how a vacuum cleaner that was accidently left turned on took down a nuclear power station in Gothenburg?

  8. “a contractor doing maintenance inadvertently switched the supply off”: it wasn’t us, officer, it was the boys from the next village.

  9. “a contractor doing maintenance inadvertently switched the supply off”:
    As they have subcontracted IT isn’t the blame always going to be pointed at a contractor, can’t blame your staff when you don’t have any.

  10. DEC Park certainly had big red buttons by the machine rooms’ exit doors, though it was an engineering etc. site not a data centre. They did indeed get pressed by accident occasionally instead of the little green exit buttons, and they did cut off all the computers.

    Re a comment on the other thread, the aircon will have to be on the UPS, the generator if not the battery, if you’re going to keep the machines running. The DEC Park 2 aircon frequently failed, so temperature sensors hit the big red button. When this also failed, we’d arrive to tropical conditions in the machine rooms, and (some models of) the disks would all die one by one over the next few weeks.

  11. “Good to see the IT industry has robust Permit To Work procedures”: golly that brings it back. Permit to work, Reg 57, etc etc. Proper technology, not boys’ toys.

  12. Sure I’ve told this story here before….

    Many years ago, I worked for a major IT firm’s helpdesk in Brussels. Massive storm, lightning strikes local substation and power goes down. We lose lights, phones and of course our front-end PCs. This was before mobiles were in common usage. We called our local head office in Paris on the boss’s pre-GSM mobile, told them the situation and they said that amazingly our network link was still running: our servers and routers were on a UPS and the Belgacom exchange must have been on a separate circuit.

    Magic, we said, we’ll switch everything off in the server room, except the mail system and the router, run some analogue phone lines from the switch and lash up some engineers with battery-driven laptops so that we’ll at least have a skeleton service for a couple of hours.

    The server room had an electronic lock on the door. We couldn’t get in from the outside.

    I went home for the afternoon, I was just a contractor and didn’t need to face the shitstorm that was brewing in the boss’s office.

  13. The server room had an electronic lock on the door. We couldn’t get in from the outside.

    Every place I’ve worked, going back to the eighties, that has security doors, they were held shut by big mains powered electromagnets. No power, no lock. Obviously the Belgians do things differently. Anyway, suspended ceiling by any chance? Worked for the Aliens.

  14. In my experience cleaners will unplug anything – and I mean *ANYTHING* – in order to plug in their vacuum cleaners.

  15. RleJ

    You’d think that would be obvious, wouldn’t it ? I think that many were programmable “stay open/stay locked” and no one had bothered to check – well why would one ?

  16. Raffles:

    “Fire control” has specific meaning, unrelated to avoidance or suppression of combustion. The term actually refers to the control of devices involved in “aiming” of equipment employed to subject (enemy) personnel and equipment to destruction via bombardment, i.e., (optical) rifle sights, binoculars and monoculars, infrared (night vision, including FLIR: forward-looking infrared), telescopes, periscopes, borescopes.

    What is most amusing is that, although there are, literally,
    thousands of such devices (including component parts and associated equipment) comprising (numerically) identifiable categories, few militarypersonnel–apart from those involved with use and maintenance of such–have any idea as to the meaning of “fire control.”

    I’ve been involved with items and equipment of such sort over 50 years, as a dealer specializing in sales of just such items (acquired as military surplus), and it’s never ceased to amaze me that folks involved in buying such stuff for years have no idea as to the meaning of “fire control.” Very typically (and this has occurred hundreds of times), I’d be asked–usually by someone in a purchasing capacity–“What has this telescope got to do with putting out a fire?”

  17. “that has security doors, they were held shut by big mains powered electromagnets. No power, no lock. ”

    The way to avoid disaster is to keep the doors open. Fire Extinguishers were designed for this purpose.

  18. @Gene Berman,

    If terminology is a sufficient barrier to communication that it baffles even experts, the terminology needs changing.

  19. Switches by doors; heard of people hitting the switch to dump the halon by mistake, although that normally sets an alarm off and a timer, so they’ve got time to hit the correct button, run back to the ops room and cancel the gas.

  20. In my (not IT) experience it takes two mechanical failures and three stupid reactions, human error, to create a major balls up.
    I doubt if the BA incident had one simple cause.

  21. So someone, either intentionally or accidentally, hits the master switch. Then, to compound the problem, realise that everything has gone awfully dark and quiet and try to turn it on again without following the proper phased startup procedure. So far, so Murphy.

    But why didn’t the second parallel data centre (BA has one) just keep everything running as normal? It sounds like there were at least three major cock-ups going on here.

  22. In one job 25-odd years ago I ended up sellotaping (other brands are available) the server plug to the socket to stop the cleaners pulling it out.

  23. It sounds more like maintenance on the UPS. UPSs have a static bypass which allows you to switch the UPS and batteries out of circuit without interrupting the output power. This has to be done regularly for battery checks, etc, and would often be done by a specialist contractor (unless you have the requisite skills on staff, unlikely for a DC). If you forget to switch it back to supply from UPS or isolate the supply while the UPS is bypassed, oops..

    An unnamed control centre I used to work at had exactly this problem once. They have two incoming supplies with an auto transfer switch, about a one second interruption when it switched supplies. The UPS was there to cover that one second. They had a power failure one day which shut them down for hours while they got everything back up and running in a controlled fashion. I wasn’t there at the time but from reading the press release (one second outage etc) it was pretty obvious that the UPS had been accidentally left in bypass, confirmed with a couple of discreet calls. This is what checklists are for, of course, but people are people and get complacent.

    What Raffles said though, in a situation like this there should be two UPSs and the equipment should all be dual supply. In really mission critical systems, the dual (or more) backups will be different makes to further reduce the chance of them failing simultaneously.

    I’m rather less surprised that the backup data centre failed to work independently, it probably relies on some sort of controlled transfer to take over. The sort of thing you would do to shut down the primary site for major maintenance.

  24. Gene Berman,

    Surely ‘Fire Control’ does what its plain meaning suggests: exerts control over the firing of the pointy-bangy stuff? The equipment you mention merely provide the inputs by which Fire Control does its job. I mean, how much control does an acquisition radar require?

  25. BiG:

    You’re right!(at least in my estimation). But to whom do you tell
    (or give such advice)? (Sum’pn I never did learn.)

    NEMO: My point is that the overwhelming number of the very personnel most needing to understand such meaning are
    almost uniformly remiss in such understanding.

    The only misunderstanding I can think of that might be seen in similar light is if personnel in the field were prone to think of
    the “Battery Commander” as the guy in charge of “C” and “D”
    cells and “triple A’s”, etc.

    I almost shudder to think what might befall the unit who ran out of the “Form for Ordering Forms.”

Leave a Reply

Name and email are required. Your email address will not be published.