Reply to: Clusterf***

[suityou01]

Oi, where's my tags gone?

[NotAllThere]

No, it was bob who couldn't work it out. Not Uhuru.

[MarillionFan]

I recall when our DB server stopped responding. But the failover wouldn't trigger, because the controller was trying to do a clean shutdown, and couldn't because a process was hanging. So we told our good friend Bob to force a shutdown of the DB server, which Bob was reluctant to do, as it wasn't in the SOP. Eventually, the op manager persuaded him - "Pull the bloody plug out if you have to - JFDI".

Only Bob, didn't. He shut down ALL the servers.

Oh, and naturally, when we tried to bring it all back, the failover db server wouldn't come up. Uhuru just couldn't work it out. Fortunately, Scotty worked out that one of the network cards had failed, knew how to get in via one of the others, and got that back online somewhat faster than the four hours the datacentre were quoting.

13000 users, around the world, unable to log on for an hour. How we laughed.

WTF. Are you Captain Kirk?

[NotAllThere]

I recall when our DB server stopped responding. But the failover wouldn't trigger, because the controller was trying to do a clean shutdown, and couldn't because a process was hanging. So we told our good friend Bob to force a shutdown of the DB server, which Bob was reluctant to do, as it wasn't in the SOP. Eventually, the op manager persuaded him - "Pull the bloody plug out if you have to - JFDI".

Only Bob, didn't. He shut down ALL the servers.

Oh, and naturally, when we tried to bring it all back, the failover db server wouldn't come up. Bob just couldn't work it out. Fortunately, Scotty worked out that one of the network cards had failed, knew how to get in via one of the others, and got that back online somewhat faster than the four hours the datacentre were quoting.

13000 users, around the world, unable to log on for an hour. How we laughed.

[MarillionFan]

You patronising b******

HTH BIDI

No worries.

I'm giving a lesson on how to suck eggs later next week. If you have a granny you'd like to enroll?

[suityou01]

With databases SY it is normally necessary to update the Live server.

But if you do, the process is to backup right before, notify and disconnect all users, take down application services, update, then bring it all back up after you've run through the process on test.

Anyway that's what I made one client attempt to do this week. They're test box is still down because they can't work out how to restart the service. I'm bloody glad i didn't go gung ho and try my changes on live first.

You patronising b******

HTH BIDI

[MarillionFan]

With databases SY it is normally necessary to update the Live server.

But if you do, the process is to backup right before, notify and disconnect all users, take down application services, update, then bring it all back up after you've run through the process on test.

Anyway that's what I made one client attempt to do this week. They're test box is still down because they can't work out how to restart the service. I'm bloody glad i didn't go gung ho and try my changes on live first.

[MarillionFan]

Oo do tell. You can change the names to protect the guilty.

Oh, your terrible.

[suityou01]

Oo do tell. You can change the names to protect the guilty.

DBAs story

1) I took a live backup which failed.
2) I noticed a lock on a table which I cleared.
3) I took the live backup which worked.

Transaction log says

1) I dropped the whole database and tried recreating it, badly. (While the system was live)
2) I failed to repopulate the table in question. (While the system was live)
3) The system limped along like this for 25 hours, and this caused further problems.
4) I then tried to rebuild the table in question again and it worked this time. (While the system was live)

So when we started investigating the data all looked hunky dory, well in the most part anyway.

[NotAllThere]

Oo do tell. You can change the names to protect the guilty.

[suityou01]

Good work and thanks for your swift reply. If the MySQL binary transaction log (exhibit a) records session information then he won't be able to wriggle out of anything. I have not taken "sabbotage" - deliberate or mistaken - out of the equation. It is on my list of possibilies. Transaction log analysis is high up my list of next tasks.

And how true that turned out to be. The transaction log made very interesting reading

Taxi for the dba

[NotAllThere]

If the MySQL binary transaction log (exhibit a) records session information then he won't be able to wriggle out of anything. I have not taken "sabbotage" - deliberate or mistaken - out of the equation. It is on my list of possibilies. Transaction log analysis is high up my list of next tasks.

Can't help you there. I know people and business, not databases. I'm just an 'umble programmer.

[suityou01]

Good feedback.

To clarify a couple of points :

The area of the system in question has been running for 10 months without problems.
The area of the system in question has had no changes in 10 months.
The guy with amnesia is not a developer. He is support. They (developers) are not allowed near the live system.
This is not a witch hunt. Admittedly the guy with amnesia has pissed me off, but I have brushed that to one side as the only witch I am hunting is the technical root cause.
The system will now be replaced I suspect as the politics dictate but this is not my concern. My concern is to troubleshoot, fix, resurrect and give guarantees. Then they can replace at leisure.

I think given that the people I interviewed have changed their story the only choice I have is to coordinate load tests on the UAT environment and meanwhile do a forensic level check on the transaction logs.

The truth will out.

Stay tuned folks.

[norrahe]

Firstly: check what was delivered and what is at fault
Secondly : check against the business spec
Thirdly: check if system testing and UAT was actually carried out correctly
Fourthly: check if said developer who is "fixing on the spot" delivered to said spec

You will often find that a developer have decided that what was asked and what he delivered were two different things. you may also find that whomever sorted out the system and UAT were a tad lapsed in their testing

If all else fails.

Duck and cover or sit back and laugh.

[Spacecadet]

In the good old days we just blamed the hardware guys.

"Network faults" has served me well for the past 6 or more years