What is the Biggest IT Cockup You've Ever Made?

[quackhandle]

Realising that SQL Server 2000 object dependency is a bag o tulipe whilst fending off 100s of calls from warehouse managers around the country asking why the parcel management system is still offline after three hours when it was only supposed to be for 30 mins.

Oh we had a chuckle over that one. (once I'd taken a few beta blockers, natch).

qh

[MrMarkyMark]

I wrote on another post that I once restored <A Large Telecom Providers> Production Reports via my own personal back up on a pen drive.

There was a little more to the story in that I was helping a user at the time and was viewing a report in Production.
On exiting the report file I then noticed the whole folder of 200 odd Production reports was missing .

It seemed the user involved was less than honest and fingers were being pointed my way, the head of IT even said "Are you sure you didn't do it?".

As I had set up the whole system, unknowingly to the user, full auditing was switched on.
It was then a simple matter of running a report to show the date and time of the deletion, complete with the users name

Back up tape machine was broken

We then restored the whole lot of 200 Prod reports from my own personal pen drive

[psychocandy]

I was working late night at a Tambrands site transferring all their drawings in to DXF format using various scripts and batch files. On one of the trips to the server room to set the next load off I though I could improve the throughput by killing some threads off. Worked a treat, the drawings were copying much faster but it was eerily silent. Went back on to the shop floor and some of the automated tampon making machines seemed to have stopped. (gulp).
Had to call the out of hours guy out to restart it all. Very uncomfortable day next day on site. Got my one and only written warning for that.

Quite apt you used to work in a tampon factory. ;-)

[psychocandy]

Can't remember any huge screw ups apart from one. Managed to overwrite part of a DB of an app managing a global network. Customer couldnt manage their network (or reroute it) for hours until it was fixed. Luckily no outages so nothing lost otherwise it would have cost them (they reckon) £1million an hour in costs if they lost links.

My only ever verbal warning as an employee was a bit of a stitch up. They gave me an expensive sparc laptop (about £10K) when I started. I hardly ever used it so locked it in desk where I was. Moved desk a week later and there was no lockable desk so I left it where I was.

Week later went to retrieve it - gone. Plus a few of my manuals etc. Of course, an investigation took place and it never turned up. I was a bit miffed to get a verbal warning for "losing it". I even asked HR in the meeting - "So what exactly should I have done in this circumstance?" and they just shrugged their shoulders. I should have appealed but didnt bother in the end.

[northernladyuk]

Quite apt you used to work in a tampon factory. ;-)

His lifetime discount on manpons comes in handy.

[geoffreywhereveryoumaybe]

COBOL Cockup

Back in the eighties I put my first release live and it all tested fine in the test environment and was to be released the evening before I took annual leave.

Came back from my holidays to discover that I hadn't release the PSB to accompany my code change and discovered that the live policy system had crashed causing mayhem with nobody having a clue what had gone wrong.

Learnt one important lesson - never deliver anything important prior to a holiday.

[vetran]

I ran rm-rf at the wrong level on a call logger. I said great we can restore from the tape as the system was installed 6 months ago. Oh we wanted you to fix that while you were there the installer never got the tape working.

Cue a long night building the system again from scratch something that the installer took 3 days doing.

I then drove back from Dorset about 4am and they wanted me in London at 8.

[BrilloPad]

I ran rm-rf at the wrong level on a call logger. I said great we can restore from the tape as the system was installed 6 months ago. Oh we wanted you to fix that while you were there the installer never got the tape working.

Cue a long night building the system again from scratch something that the installer took 3 days doing.

I then drove back from Dorset about 4am and they wanted me in London at 8.

Reminds me about the time at GS when a manager complained about lack of prod disk space. Some youngster suggested \rm -rf * (rm was aliased to ensure you were always prompted). They really got a bawling out for that.....

[northernladyuk]

I'm not an IT person at all but I did once bring down 'something' on a national scale (Mail or NHS Network - I didn't ask too closely) after creating an auto-reply loop between a GP practice and a hospital which multiplied the number of emails exponentially. It was a really neat piece of clinical service redesign.

[WTFH]

What is the Biggest IT Cockup You've Ever Made?

I'm not an IT person at all but I've had a cockup or two.

FTFY