Reply to: anyone made a massive mistake on a contract?

[lukemg]

2 spring to mind -
In an early job as support monkey, turned a bit quick with a new 26" CRT monitor on a trolley. The monitor carried straight on and smashed....
In the old days, server room at client co had a monitor and k/b for each server on the bench above. However, monitors tended to be a bit wider than the server they serviced which I found out to my cost when turning off a backup server only to see the next monitor to the left go off instead of the one I was looking at.
It was the main email server with 2000 people working on it.....

[seyre1972]

For the non techies

Care to rephrase in English for those of us who don't speak Unix?

Sure - I'll try and put it in simplistic terms ....

It was royally tulliped.

[Sysman]

Did an rsync with --delete, but got the source and destination hosts the wrong way 'round

rsync is a wonderful tool, but it's also a good demonstration of the saying:

"To err is human.
For a serious cockup you need a computer."

rsync --dry-run is my friend here.

[Freamon]

Some good candidates for the daily WTF on this thread:

The Daily WTF: Curious Perversions in Information Technology

[kingcook]

Did an rsync with --delete, but got the source and destination hosts the wrong way 'round

[stek]

I once did tar -cvf /home stek.tar and ended up with everyones homes totally twatted and and an empty stek.tar file....

f is FILENAME, f is FILENAME....

Not the other way round. Never did it again and a panic enfolds me every tar I do.

[xoggoth]

I had the best paid contract I ever had, only a few days unfortunately, to write a report on best options for recovery after the MOD somehow lost most of an umpteen hundred thousand pound training simulator I had worked on years before.

PS Not IT, but as a student I did once break a rubber mill just when they needed it to fill an urgent contract.

[Cr1spy]

Tales from the crypt.

The worst "Get Latest from Source Control" error I've managed (so far!) overrode a whole week’s work. It's surprising how quickly you can recreate a week’s work. I suppose most of the initial development involved feeling out the solution and running down dead ends.

While working at Numpty Perm Co many years ago, a colleague received a written warning when he emptied the bosses deleted email folder in an attempt to free some disk space. Apparently, the boss used the deleted folder to store emails that needed further action!

Best one was another friend of mine who inadvertently downed the entire south east air ambulance fleet. He was worked on a mapping GPS system used in the cockpit. The system run on a Windows NT 4with a custom physical interface, providing routing information overlaid on a map. All was well and good until he received a call later that year...

All the systems had displayed the "Switch to British summer time" message box, but the physical interface didn't provide any way to click the ok button!

[DiscoStu]

Lots of stuff...

Care to rephrase in English for those of us who don't speak Unix?

[seyre1972]

Those late Friday requests - "Ah - you're in the office tomorrow - can you ......"

Long time ago ........

Sorry V long story.

Friday afternoon - rota'd to be onsite on the Saturday for several changes to be implemented (in the days when SA's did everything HW/OS/App support etc)

Got asked as a favour for the SA who was meant to be doing it as it was his project - could I run an implementation (which had run succesfully for previous 18 months or so) a new release of the Companies major earning application (with several hundred external financial clients) Bit put out -but as we were on an hourly rate - not a problem.

Started Saturday 8am on other work.
4pm -Saturday afternoon - Time to run the automated, checked with ops, quiesced the application etc.

Backout for this app was a Copy A and Copy B each on shared storage of the application. Script would offline/remount as Copy B Copy A of the app, and do opposite for Copy B to Copy A.

Kicked it off - and left it running as did loads of stuff in background.

1 hrs later - still running - seems a bit odd - tailing log files - still updating.
Tried a few commands - got permission/ownership errors.

Transpired that the script which rolled out put the new app in place - had been modifed during the week. you're going to like this bit.

cd $VAR
chown -R appid:groupid *

Which all worked a treat on UAT - as the $VAR was set. However the ENV file on PROD had not been updated by the developers -

As this script ran as root - with $VAR not set, cd took it back to / - and a recursive chown was done. In an Active Active cluster......

Long story short - left the office following Monday morning after explaining to the new IT director (in his 2nd week) what the problem was, and why it had taken nearly 30 hrs to restore the system from the backup system they used (5% restored - fail, restart, 7% restored, fail and so on)

Later found out that the development director had asked for me to be sacked - as it was my fault for running their supplied script

Thankfully new IT director told him where to go.

My overtime for that month was a new record - which I don't think has been broken since

[Sysman]

First job as a grad, I deleted all the files on a UNIX directory. It was the entire source repo.

Now, I am uber paranoid.

It was that time of year when a load of contractors left and we got a fresh bunch in.

Yours truly deleted the old accounts.

Screams from project manager.

Me: Eh?
PM: YOU DELETED OUR FILES!
Me: But these people have left.
PM: I GAVE THE OLD USERS' LOGIN DETAILS TO THE NEW FOLKS!!!
Me. That's against bank policy (starting to smirk at this stage)
PM: YOU DELETED OUR DATA. DON'T YOU EVER DO THAT AGAIN!!!!!!!

Me: Hangs up. Sends polite memo, CC-ing my boss, and his boss too.

OK, I restored the data in super quick time, but I managed to score some brownie points. Both bosses were extremely keen on security.

PS: The PM concerned was a real battle axe and was trying to throw her weight (of which there was plenty - bitchy snigger) around. Some people simply deserve having someone stand up to them for a change.

PPS: From then on I adopted a policy of renaming account directories to somewhere inaccessible rather than deleting them outright.

[BoredBloke]

Formatted the IT directors PC on my one and only job for a bank many, many moons ago! Doing a DOS upgrade from 3.x to 5 and IBM PC DOS suggested moving the partition. Thinking that would be good and that it would tell me if it was going to delete anything I figured I'd give it a go. Much chugging and then Formatting 1%, 2% as my face went white, and I prepared to be turfed out of the building. A contract that lasted less than half a day!!

[conned tractor]

First job as a grad, I deleted all the files on a UNIX directory. It was the entire source repo.

Now, I am uber paranoid.

i worked with someone who did that....ha was named delboy after that incident.

[RasputinDude]

First job as a grad, I deleted all the files on a UNIX directory. It was the entire source repo.

Now, I am uber paranoid.

[TestMangler]

Back in the days when big players in banking and insurance ran their servers on OS/2 LAN Server, then Warp Server, I was a server techie. Was in a server room with a new guy (contractor) who was brought in as he was an 'expert' in oS/2 and the newly emerging NT4

While in a server farm, i was doing something on an NT server and he started poking at the one next to where i was. I told him, 'Don't touch that, it's an OS/2 box' he replied , 'No it isn't, it's NT4' and CTRL-ALT-DEL'd the thing. It was the days of inter departmental servers and he flattened the DC for Aegon's investment dept. Oh, how we laughed

In the end, I got a kicking for letting him touch it