anyone made a massive mistake on a contract?

[BoredBloke]

Formatted the IT directors PC on my one and only job for a bank many, many moons ago! Doing a DOS upgrade from 3.x to 5 and IBM PC DOS suggested moving the partition. Thinking that would be good and that it would tell me if it was going to delete anything I figured I'd give it a go. Much chugging and then Formatting 1%, 2% as my face went white, and I prepared to be turfed out of the building. A contract that lasted less than half a day!!

[Sysman]

First job as a grad, I deleted all the files on a UNIX directory. It was the entire source repo.

Now, I am uber paranoid.

It was that time of year when a load of contractors left and we got a fresh bunch in.

Yours truly deleted the old accounts.

Screams from project manager.

Me: Eh?
PM: YOU DELETED OUR FILES!
Me: But these people have left.
PM: I GAVE THE OLD USERS' LOGIN DETAILS TO THE NEW FOLKS!!!
Me. That's against bank policy (starting to smirk at this stage)
PM: YOU DELETED OUR DATA. DON'T YOU EVER DO THAT AGAIN!!!!!!!

Me: Hangs up. Sends polite memo, CC-ing my boss, and his boss too.

OK, I restored the data in super quick time, but I managed to score some brownie points. Both bosses were extremely keen on security.

PS: The PM concerned was a real battle axe and was trying to throw her weight (of which there was plenty - bitchy snigger) around. Some people simply deserve having someone stand up to them for a change.

PPS: From then on I adopted a policy of renaming account directories to somewhere inaccessible rather than deleting them outright.

[seyre1972]

Those late Friday requests - "Ah - you're in the office tomorrow - can you ......"

Long time ago ........

Sorry V long story.

Friday afternoon - rota'd to be onsite on the Saturday for several changes to be implemented (in the days when SA's did everything HW/OS/App support etc)

Got asked as a favour for the SA who was meant to be doing it as it was his project - could I run an implementation (which had run succesfully for previous 18 months or so) a new release of the Companies major earning application (with several hundred external financial clients) Bit put out -but as we were on an hourly rate - not a problem.

Started Saturday 8am on other work.
4pm -Saturday afternoon - Time to run the automated, checked with ops, quiesced the application etc.

Backout for this app was a Copy A and Copy B each on shared storage of the application. Script would offline/remount as Copy B Copy A of the app, and do opposite for Copy B to Copy A.

Kicked it off - and left it running as did loads of stuff in background.

1 hrs later - still running - seems a bit odd - tailing log files - still updating.
Tried a few commands - got permission/ownership errors.

Transpired that the script which rolled out put the new app in place - had been modifed during the week. you're going to like this bit.

cd $VAR
chown -R appid:groupid *

Which all worked a treat on UAT - as the $VAR was set. However the ENV file on PROD had not been updated by the developers -

As this script ran as root - with $VAR not set, cd took it back to / - and a recursive chown was done. In an Active Active cluster......

Long story short - left the office following Monday morning after explaining to the new IT director (in his 2nd week) what the problem was, and why it had taken nearly 30 hrs to restore the system from the backup system they used (5% restored - fail, restart, 7% restored, fail and so on)

Later found out that the development director had asked for me to be sacked - as it was my fault for running their supplied script

Thankfully new IT director told him where to go.

My overtime for that month was a new record - which I don't think has been broken since

[DiscoStu]

Lots of stuff...

Care to rephrase in English for those of us who don't speak Unix?

[Cr1spy]

Tales from the crypt.

The worst "Get Latest from Source Control" error I've managed (so far!) overrode a whole week’s work. It's surprising how quickly you can recreate a week’s work. I suppose most of the initial development involved feeling out the solution and running down dead ends.

While working at Numpty Perm Co many years ago, a colleague received a written warning when he emptied the bosses deleted email folder in an attempt to free some disk space. Apparently, the boss used the deleted folder to store emails that needed further action!

Best one was another friend of mine who inadvertently downed the entire south east air ambulance fleet. He was worked on a mapping GPS system used in the cockpit. The system run on a Windows NT 4with a custom physical interface, providing routing information overlaid on a map. All was well and good until he received a call later that year...

All the systems had displayed the "Switch to British summer time" message box, but the physical interface didn't provide any way to click the ok button!

[xoggoth]

I had the best paid contract I ever had, only a few days unfortunately, to write a report on best options for recovery after the MOD somehow lost most of an umpteen hundred thousand pound training simulator I had worked on years before.

PS Not IT, but as a student I did once break a rubber mill just when they needed it to fill an urgent contract.

[stek]

I once did tar -cvf /home stek.tar and ended up with everyones homes totally twatted and and an empty stek.tar file....

f is FILENAME, f is FILENAME....

Not the other way round. Never did it again and a panic enfolds me every tar I do.

[kingcook]

Did an rsync with --delete, but got the source and destination hosts the wrong way 'round

[Freamon]

Some good candidates for the daily WTF on this thread:

The Daily WTF: Curious Perversions in Information Technology

[Sysman]

Did an rsync with --delete, but got the source and destination hosts the wrong way 'round

rsync is a wonderful tool, but it's also a good demonstration of the saying:

"To err is human.
For a serious cockup you need a computer."

rsync --dry-run is my friend here.