Clusterf***

[MarillionFan]

I'll need more than a packet of bourbons.

She'll want a packet of bourbons and a packet of crisps. But no kissing.

[OwlHoot]

Ask for a copy of all security camera footage for the last two weeks, and run a face recognition scan on it until you see a guilty perp sneaking out of the server room.

(If you watched Spooks, like me, you wouldn't have to ask us.)

[Zippy]

She'll want a packet of bourbons and a packet of crisps. But no kissing.

A packet of Hobnobs?

Seriously SY you've had some good advice. I don't see the point in turning it into a witch hunt though.

[Spacecadet]

In the good old days we just blamed the hardware guys.

"Network faults" has served me well for the past 6 or more years

[norrahe]

Firstly: check what was delivered and what is at fault
Secondly : check against the business spec
Thirdly: check if system testing and UAT was actually carried out correctly
Fourthly: check if said developer who is "fixing on the spot" delivered to said spec

You will often find that a developer have decided that what was asked and what he delivered were two different things. you may also find that whomever sorted out the system and UAT were a tad lapsed in their testing

If all else fails.

Duck and cover or sit back and laugh.

[suityou01]

Good feedback.

To clarify a couple of points :

The area of the system in question has been running for 10 months without problems.
The area of the system in question has had no changes in 10 months.
The guy with amnesia is not a developer. He is support. They (developers) are not allowed near the live system.
This is not a witch hunt. Admittedly the guy with amnesia has pissed me off, but I have brushed that to one side as the only witch I am hunting is the technical root cause.
The system will now be replaced I suspect as the politics dictate but this is not my concern. My concern is to troubleshoot, fix, resurrect and give guarantees. Then they can replace at leisure.

I think given that the people I interviewed have changed their story the only choice I have is to coordinate load tests on the UAT environment and meanwhile do a forensic level check on the transaction logs.

The truth will out.

Stay tuned folks.

[NotAllThere]

If the MySQL binary transaction log (exhibit a) records session information then he won't be able to wriggle out of anything. I have not taken "sabbotage" - deliberate or mistaken - out of the equation. It is on my list of possibilies. Transaction log analysis is high up my list of next tasks.

Can't help you there. I know people and business, not databases. I'm just an 'umble programmer.

[suityou01]

Good work and thanks for your swift reply. If the MySQL binary transaction log (exhibit a) records session information then he won't be able to wriggle out of anything. I have not taken "sabbotage" - deliberate or mistaken - out of the equation. It is on my list of possibilies. Transaction log analysis is high up my list of next tasks.

And how true that turned out to be. The transaction log made very interesting reading

Taxi for the dba

[NotAllThere]

Oo do tell. You can change the names to protect the guilty.

[suityou01]

Oo do tell. You can change the names to protect the guilty.

DBAs story

1) I took a live backup which failed.
2) I noticed a lock on a table which I cleared.
3) I took the live backup which worked.

Transaction log says

1) I dropped the whole database and tried recreating it, badly. (While the system was live)
2) I failed to repopulate the table in question. (While the system was live)
3) The system limped along like this for 25 hours, and this caused further problems.
4) I then tried to rebuild the table in question again and it worked this time. (While the system was live)

So when we started investigating the data all looked hunky dory, well in the most part anyway.