What is the Biggest IT Cockup You've Ever Made?

[SeededLoaf]

In my earlier days I did a software upgrade of a date centre management network and took down a large chunk of prod. The upgrade process, as per the manual, stated no downtime but it lied. Prod was far too dependent on the backend and there rapid re-design after that incident.

[BlasterBates]

The best one I experienced was when we were installing a computer controlled computer system for a hot strip steel mill in South Korea. During commissioning the mill was run up and a few tests, the rollers went up and down, after the tests were finished, the Lead Engineer switched off the computer, as you do , after which we heard a rather odd crunching sound, followed by the swift entry of a Korean Engineeer into the Computer room who uttered something to the effect of WTF. The sight from the control room was rather dramatic with the rollers skewed at a funny angle, unable to rotate and the mill covered in white hot steel, alas the hydraulic valves installed had lost power from the computer and defaulted to wide open thus crushing the rollers and causing the hot metal to pile up like spaghetti.

Having repaired the mill the tests were repeated. The lead Engineer spotted a slight problem with the control electronics, did a couple of measuerments, having realised the resistor on the card was wrong, pulled the card out to replace the resistor with his soldering iron, and....exactly ...., you pull the card out it's like switching the computer off.

He was not a happy lead Engineer, and it was not a happy Steel company.

I earnt a lot of money in overtime fixing the steel mill, but gaw was I tired.

[m0n1k3r]

In the very late eighties, with 2400 bps modems the norm rather than the exception. I was asked to get a Usenet feed with only a few groups. I managed to get the subscription wrong, causing half of the modems to be occupied permanently and racking up quite large phone bills, while at the same time running out of space on the 340 MB Fujitsu HDD (proper Winchester physical size) that was supposed to store it all. I was on holiday when disaster struck. Oops.