So who did BA outsource their IT to?

[WTFH]

I'm supposed to be flying with them on Tuesday.

[darmstadt]

Being attributed to a "power failure".

Those in power failed by the looks of it, will heads roll?

[darmstadt]

Friends flying out to play a gig today:

PSK, Sweden. We're really sorry but unfortunately we are unable to get to Sweden today for our show in Stockholm. This is the first time in 45 years that we have missed a show and we're absolutely gutted. It's been a really stressful day at the airport. British Airways ended up cancelling ALL their flights due to a problem with their systems worldwide. However, we had already managed to get checked in and our bags and guitars taken from us and put on the plane ready to leave. The plane had a full crew and the pilot wanted to take off. They even sent 2 buses to drive us to the plane, so we thought our flight would make it. However, at the last minute they cancelled EVERYTHING. Then came the real problems of trying to get out of the airport. Suddenly thousands and thousands of people needed to leave and it was chaos. There was no help, no information and a lot of stress. No staff knew what was happening. We were getting updates from the BBC News channels who knew more than the people in the airport! For all our friends in Sweden, we are truly sorry to have let you down. We will make it up to you. The festival are working on a solution. At the moment it's early days, however they have said to keep your wristbands and await further news. To British Airways, we appreciate that this is not a usual situation, however you judge a company on how it deals with these things and frankly it was at best inadequate and at worst negligent. You cancelled flights and sent people to exits before the exits were even staffed, just leading to more confusion and delay. You had no-one on the ground giving any guidance or information. We appreciate that it is a 'live' situation which can change rapidly, however the BBC announced that all flights were cancelled a full hour before any of your staff knew at the gates. We were all showing British Airways staff what was going on with their own company and the statements you'd made to the press BEFORE it had filtered down to the people on the frontline. Disgraceful. All our bags and instruments were sitting on one plane but you couldn't work out how to get them off and on to a belt for us to pick up!? Again the default answer was that you didn't know what was going on. But that is the point. Where are your contingency plans? What disaster recovery do you have in place? To say "go home and put in a claim for your bags" is no good to us. We need the equipment tomorrow as we're flying again to Las Vegas. The lack of procedure, communication and information was very very poor. We are beyond frustrated and you have let a lot of people down

[AtW]

Those in power failed by the looks of it, will heads roll?

Whoever was in charge will be fired...

[jamesbrown]

Those in power failed by the looks of it, will heads roll?

An experienced CIO would've known to turn it off and on again.

[Andy Hallett]

An experienced CIO would've known to turn it off and on again.

It's now known as 'power cycling'

[RSoles]

It's now known as 'power cycling'

Or 'slope shouldering'.

[NotAllThere]

Being attributed to a "power failure".

A few years ago, I flew to the UK for a meeting for a couple of days with the client manager and his team. A couple of hours after arrival, there was a "major incident", so we were all pulled into the "war room", where there was a conference call going on with the data centre in Bangalore. The database server had stopped most of its communication with the outside world, but since it was still pingable the fail-over detector didn't notice. The client manager told the chap in the data centre "switch off the database server". (Then the fail-over detector wouldn't get a response to ping, and would properly fail over to the backup).

"I can't do that sir".
"Yes you can. It's my responsibility. Now do it."
"I can't do that sir".
"Just unplug it!":
"I can't do that sir".
...
and so on. Eventually the data centre numpty chap cracked, and agreed to switch off the database server. At least, that's what we heard him agree to. The next minute, everything went down.

"What have you done?".
"Just what you told me sir. I've turned the power off."

He'd hit the emergency power off*. Thus neatly turning what should have been a thirty minute outage affecting a few thousand users into over five hours affecting a few tens of thousands of users, as network cards and other components failed at boot up and needed replacing...

The official cause of the issue? Power outage.



*At this point I diplomatically left the room, as it was becoming increasingly difficult to stifle my laughter.

[vetran]

hmm seems to be a week for it..

https://www.theregister.co.uk/2017/0...a_data_centre/

A major outage at a Capita data centre has knocked out multiple services for customers – including a number of councils' online services – for the last 36 hours.

Some of the sites affected include the NHS Business Services Authority, which apologised on its website for the continuing disruption and said it hoped its systems would be available by noon on Friday.

However, the impact appears to be much wider. One customer who asked not to be named said: "According to rumour, there was a power failure in West Malling and the generators failed, shutting the whole data centre down."

He added Capita has a virtualisation platform which hosts at least 30 clients and many internal Capita systems, and it still appears to be down. "They have probably had to fly parts in from out of the country as the infrastructure is so old," he said.

the cloud is somebody else's knackered old computer apparently..

[jamesbrown]

At this point I diplomatically left the room, as it was becoming increasingly difficult to stifle my laughter.