Reply to: What is the Biggest IT Cockup You've Ever Made?

[m0n1k3r]

In the very late eighties, with 2400 bps modems the norm rather than the exception. I was asked to get a Usenet feed with only a few groups. I managed to get the subscription wrong, causing half of the modems to be occupied permanently and racking up quite large phone bills, while at the same time running out of space on the 340 MB Fujitsu HDD (proper Winchester physical size) that was supposed to store it all. I was on holiday when disaster struck. Oops.

[BlasterBates]

The best one I experienced was when we were installing a computer controlled computer system for a hot strip steel mill in South Korea. During commissioning the mill was run up and a few tests, the rollers went up and down, after the tests were finished, the Lead Engineer switched off the computer, as you do , after which we heard a rather odd crunching sound, followed by the swift entry of a Korean Engineeer into the Computer room who uttered something to the effect of WTF. The sight from the control room was rather dramatic with the rollers skewed at a funny angle, unable to rotate and the mill covered in white hot steel, alas the hydraulic valves installed had lost power from the computer and defaulted to wide open thus crushing the rollers and causing the hot metal to pile up like spaghetti.

Having repaired the mill the tests were repeated. The lead Engineer spotted a slight problem with the control electronics, did a couple of measuerments, having realised the resistor on the card was wrong, pulled the card out to replace the resistor with his soldering iron, and....exactly ...., you pull the card out it's like switching the computer off.

He was not a happy lead Engineer, and it was not a happy Steel company.

I earnt a lot of money in overtime fixing the steel mill, but gaw was I tired.

[SeededLoaf]

In my earlier days I did a software upgrade of a date centre management network and took down a large chunk of prod. The upgrade process, as per the manual, stated no downtime but it lied. Prod was far too dependent on the backend and there rapid re-design after that incident.

[fool]

Correct and its funny you have just reminded me about something I did when I had just started in IT .
I deleted a folder on a NW drive by being clumsy

Luckily was a small IT Dept, at the time, so got it restored without any issue.
I was always a lot more careful after that

Yeah it actually crippled my productivity when dealing with production systems for a good wee while. Now it serves as a story I tell the kids when they **** up, so they don't feel like it's over.

At the same place a guy deleted the main volume on a netapp that had all the LUNs for all the customer VMs that was sold as "Europes first cloud". No backups either. Had to call the vendor and ask them how to extract LUNs direct from the block storage.

[northernladyuk]

The thread was about IT.

HTH

MF

Please don't tell us you're a real person.

[MarillionFan]

Apart from your life, of course.

The thread was about IT.

HTH

MF

[MrMarkyMark]

Indeed, but it was a good time to learn the lesson that anyone can screw up...

Correct and its funny you have just reminded me about something I did when I had just started in IT .
I deleted a folder on a NW drive by being clumsy

Luckily was a small IT Dept, at the time, so got it restored without any issue.
I was always a lot more careful after that

[northernladyuk]

I am afraid to,admit I don't recall anytime I've fecked something up, minor or major. I'm the one who tends to catch everyone's elses possible feckups and don't make them myself.

Apart from your life, of course.

[MarillionFan]

I am afraid to,admit I don't recall anytime I've fecked something up, minor or major. I'm the one who tends to catch everyone's elses possible feckups and don't make them myself.

[fool]

Seems like you lived up to your user name on this occasion

Indeed, but it was a good time to learn the lesson that anyone can screw up...

[MrMarkyMark]

In my first dev role about 10 years ago, the Lead (only other) Dev was editing files in production and expected that I would do the same.

I said, sod that I'm making a development environment. I exported production (my first job lads and I had no other access to the schema or any way of producing test data quickly) and imported into a dev version.

It failed because of some MySQL configuration values which after a quick google suggested I needed to add a flag, fair enough, I'll just drop this...

CEO - "Why is production down"
ME - *Gulp*

To be fair we restored from backup and I finished setting up dev environments.

Seems like you lived up to your user name on this occasion

[fool]

In my first dev role about 10 years ago, the Lead (only other) Dev was editing files in production and expected that I would do the same.

I said, sod that I'm making a development environment. I exported production (my first job lads and I had no other access to the schema or any way of producing test data quickly) and imported into a dev version.

It failed because of some MySQL configuration values which after a quick google suggested I needed to add a flag, fair enough, I'll just drop this...

CEO - "Why is production down"
ME - *Gulp*

To be fair we restored from backup and I finished setting up dev environments, SCM, etc.

[OwlHoot]

Ages ago I wrote a small web app for a local sandwich company which was essentially ordering a customised sandwich. The data was then emailed to a third party who would fax it to the kitchen. No database involved since they wanted it done cheaply and quickly.

Anyway, all done, tested on all environments and before go live I wanted to run some automated tests and decided that the best way was to comment out the line of code that sent the email, since I did not want the kitchen to start making sandwiches that I was testing.

On go live, I completely forgot about this and the site was launched and appeared to be busy. But no orders reached the kitchen and the phones went off the hook with customers angry that their sandwich never was delivered.

Was it you who tested a direct mail promotion to customers of a prestigious bank, and when it went live forgot to change the salutation from "Dear Rich Bastard" ?

[xoggoth]

Not had any major ones

Not that have come out yet anyway. Done quite a bit of work on aircraft control systems. If you are on a 747 just make sure you don't flush the toilet at exactly 24301ft 7.5inches

[VectraMan]

Not an IT person, but I remember whilst working for a small company we'd got a new server with RAID5 which we hadn't seen before. I thought it was a good idea to test it by pulling a drive out whilst the system was running, which apparently you weren't meant to do. No data was lost or anything, but I think we ended up returning the drive and getting a warranty replacement.

I can remember leaving something that logged to c:\log.txt in a release build that went out to customers. And I'm sure I've once or twice made the mistake of commenting out a line that enabled some major piece of functionality, forgetting, committing it and then getting back a bug report that major-new-piece-of-functionality wasn't working. Doh.