Reply to: BA contractor screw up

[NotAllThere]

computer equipment do not switch off from Safety Sign Supplies

BA say that they can prevent a similar occurrence at minimal cost

[motoukenin]

computer equipment do not switch off from Safety Sign Supplies

BA say that they can prevent a similar occurrence at minimal cost

[BrilloPad]

What is the likelihood that the external company carrying out the inquiry are the same external company who were contracted in to carry out the work/support in the first place?

Reminds me of the Alex cartoon.
Alex to Penny : There is to be a BoE enquiry into derivatives
Penny to Alex: What are those things?
Alex to Penny : A way of offsetting your liabilities
Penny to Alex: So that what a derivative is!
Alex to Penny : No. Thats what a BoE enquiry is. No-one in the city has any idea what a derivative is.

[ascender]

I noticed this line too....

The airline's UK IT infrastructure is said to span more than 500 cabinets in six halls across two different data centres, both of which are no more than a mile from the eastern end of Heathrow's two runways.

I've worked for clients where the fact they're under a designated flight path or even within 10 miles of an airport means they will identify a geographically-distant location for a second data hall. BA have both within a mile from the end of a runway at one of the world's busiest airports?!

Other than that, I concur, it still all smells very fishy indeed.

[WTFH]

What is the likelihood that the external company carrying out the inquiry are the same external company who were contracted in to carry out the work/support in the first place?

[MrMarkyMark]

The information I have second hand from an insider is that an IT partner was told to apply security patches. They were given 24 hours to do this. The patches were applied to both streams (computer systems running in parallel), instead of one and then the other. Then the data centre was powered down, and a power up (all at once) was attempted - so everything went to pot. All done in ten minutes, so they over-achieved the target.

Note that TCS (just as a hypothetical example) can be referred to as "a contractor".

This is exactly what I read on a flyers forum.

[NotAllThere]

...So yes my money is completely on TCS being in the frame for this...

The information I have second hand from an insider is that an IT partner was told to apply security patches. They were given 24 hours to do this. The patches were applied to both streams (computer systems running in parallel), instead of one and then the other. Then the data centre was powered down, and a power up (all at once) was attempted - so everything went to pot. All done in ten minutes, so they over-achieved the target.

Note that TCS (just as a hypothetical example) can be referred to as "a contractor".

[Eirikur]

Contractor in this context mean most likely BT Openreach engineer

Looks like my post was largely right

[saptastic]

Thanks Bobspud for all the insight.

https://www.theguardian.com/business...lsh-passengers

"The actions of a technician who switched off and then reconnected the power supply to BA’s data centre are expected to form the centre of the inquiry, which will be carried out by an as-yet-unnamed external company".

Just cant get my head round this story. Its like a sitcom. Wonder what the BA IT Security is like.

[bobspud]

JCI now CBRE I think would actually use an FM plus Electrical staff/Contractors for any HV/PDU/UPS work probably a small herd of them
They also are at pain to go through method statements H&S docs etc. ad nauseum before even picking their noses.

So whatever happened I don't think we are seeing the full story yet.

...

You only need one Cockwomble to create an Outage.

Yep DC's are full of trained trades to handle this and I have never seen a cowboy amongst them because of the strict site rules and the fact that the electricity voltages are way higher than home so getting it wrong for a sparky is a one time thing.

Back to the BA thing if they over fed the UPS with power draw it should still only blow that rack not the whole room so something very broken and Im not convinced that it would have got signed off for its commissioning certificates had they spotted it when they refitted the system a few years back.

More fish than Billingsgate...

[original PM]

El Reg has more detail.

https://www.theregister.co.uk/2017/0...configuration/




EPO??

Apparently the UPS units were recent from Socomec

So in effect they tried to turn it off and on again to fix a bug and failed.

I imagine someone logged a call saying x was not working and first line guy asked if you had tried turning it off and on again to see if that solves the problem

👀

[DallasDad]

Bobspud describes a modern DC well.
Diseasex may be nearer the mark if indeed the site is positively ancient but I can't imagine why it would be.

I am not convinced an "IT Engineer" would be responsible though.
JCI now CBRE I think would actually use an FM plus Electrical staff/Contractors for any HV/PDU/UPS work probably a small herd of them
They also are at pain to go through method statements H&S docs etc. ad nauseum before even picking their noses.

So whatever happened I don't think we are seeing the full story yet.
No criminal act has occurred share prices need protecting so I think a suitable story is being spun.

However maybe, just maybe they had a big red physical or virtual button!

An ex client had a big red button on the wall in their DC in Henley.
It was alongside a ramp up to a higher level in the room, it was not a clever place to put it

One day apparently someone 'protected' it with a polystyrene cup.
If you hit a polystyrene cup they squash up don't they.
A few day later, yep visiting copier engineer gave it a wallop!
Room went dark.

Same client this time in London
JCI FM who should have known better let in a bunch of sparks to carry out an earth bonding exercise on some new racks is a very large machine room.
Everything thing went fine until oops.
Room went very dark.

Then there is the total dick scenario which I bet you have all seen.
I worked for a while for F***x at Stanstead
All their comms and local Netware 4.1 servers ran via multiple extension leads dangling over a metal separation wall to a three way adapter using a single 13A socket.
before it died it had smoldered for probably years. and you could see how warm the bakelite had been getting.
It went pop on my day one!

You only need one Cockwomble to create an Outage.

[vetran]

El Reg has more detail.

https://www.theregister.co.uk/2017/0...configuration/

Bill Francis, Head of Group IT at BA's owner International Airlines Group (IAG), has sent an email to staff saying an investigation so far had found that an Uninterruptible Power Supply to a core data centre at Heathrow was over-ridden on Saturday morning. He said: "This resulted in the total immediate loss of power to the facility, bypassing the backup generators and batteries. This in turn meant that the controlled contingency migration to other facilities could not be applied. "After a few minutes of this shutdown of power, it was turned back on in an unplanned and uncontrolled fashion, which created physical damage to the system, and significantly exacerbated the problem.

EPO??

Apparently the UPS units were recent from Socomec

BoHo's uninterruptible power supplies (UPSes) were replaced three years ago with equipment from electrical firm Socomec, which refused to comment for this article.

[MrMarkyMark]

So were most nuclear bunkers? I thought most of them still required 5.25 inch floppy drives?

I reinstalled <A Large Telecom Providers> Production Reporting systems reports (200+) with a (possibly illegal for me to have) personal USB back up once, their back up tape machine had "broken" .

The IT director was extremely grateful.

[Paddy]

"The system was designed and installed there in the mid-1980s. "