Accidentally nuked production database

[BrilloPad]

https://www.theregister.co.uk/2017/0...stantly_fired/

Who would be that stupid? Accidentally nuking prod - we all make mistakes. Once you make a mistake, STFU. Don't go onto Reddit.....

[MrMarkyMark]

https://www.theregister.co.uk/2017/0...stantly_fired/

Who would be that stupid? Accidentally nuking prod - we all make mistakes. Once you make a mistake, STFU. Don't go onto Reddit.....

[bobspud]

Thats not their fault...

The system admin team and especially networks need 100 lashes for abject ******* stupidity.

1) Segregate DEV, Test and Production environments completely.

2) Dev team should never know production passwords.

3) Dev team should not have live user accounts.

everything after the test environment should be handled by a deployment team NOT the developer team.

The fact that they could sit on the Dev systems and still hit live shows a total lack of understanding...

[MrMarkyMark]

Thats not their fault...

The system admin team and especially networks need 100 lashes for abject ******* stupidity.

1) Segregate DEV, Test and Production environments completely.

2) Dev team should never know production passwords.

3) Dev team should not have live user accounts.

everything after the test environment should be handled by a deployment team NOT the developer team.

The fact that they could sit on the Dev systems and still hit live shows a total lack of understanding...

Very true, true segregation is key in any well designed system.

I generally deal with enterprise reporting software right across the board so have worked with most databases at some stage.

One thing I always stress is having the right number of environments procedures and DR etc.
I usually manage to "sell" it to be people that get it.

i.e.

Less risk
Less stress
Less down time
Less money on expensive major incidents

Funny though you see some of the most ludicrous things and not just in the smaller Blue Chips either.

I wasn't particularly surprised to find a major gambling business with one "Prod" system complete with a centralized spreadsheet to patch in the data holes in their reporting.
Obviously some clown left the spreddy open for the weekend causing 600 reports to have failed by the Monday.

They then had to pay me to set up optimised batch jobs to play catch up during the week.
Got a few more days billing out of that one

[vetran]

Production DB access should be controlled more tightly than the bank accounts for many reasons.

DB creation etc should be automated and only done by DBAs not Devs.

Poor management

[woohoo]

Production DB access should be controlled more tightly than the bank accounts for many reasons.

DB creation etc should be automated and only done by DBAs not Devs.

Poor management

Lots of Devs have excellent knowledge of databases and are perfectly capable of maintaining a database. It's nothing special that can't be learnt.

[AtW]

1) Segregate DEV, Test and Production environments completely.

Racist!!!

[bobspud]

Lots of Devs have excellent knowledge of databases and are perfectly capable of maintaining a database. It's nothing special that can't be learnt.

No thats not the point. The point is the DB is a crown jewel and no one especially a ******* first day dev should be able to get anywhere near it.

Scripts that run against it should be peer reviewed to death and it should never be accessible from anywhere on the dev teams floor.

Im not saying the dev team should not be able to do hourly release cycles or that Jenkins can't be made to run like the wind. Just that after you have the release candidate, the upgrade to live is handled by someone that respects the need for maximum up time.

even with the best code team on the planet, we have all seen deployment scripts arrive with the wrong configuration settings and so for that reason you are an utter and complete moron if your dev team can ping a live instance from their terminals, let alone have the login details and permissions to execute code...

[quackhandle]

Lots of Devs have a little knowledge of databases and are perfectly capable of ballsing up a database. It's nothing special that they fail to learn.

FTFY.

qh

[vetran]

Lots of Devs have excellent knowledge of databases and are perfectly capable of maintaining a database. It's nothing special that can't be learnt.

They may have, however if you have a DBA you should segregate the duties. The only people who should create DBs in production are DBAs following detailed written instructions written on Dev, tested in QA and then migrated securely to Production. Its the only way to make sure migration goes well - trust me I have tried other ways.

No Dev should create a DB on Production they tend to 'play'. Anyway the production DB is owned by Support not Development.

No real need for a Dev to create one on QA. This is test for production.

A dev can have access to Dev and create DBs with pleasure. In fact I would expect them to do so and script them so the scripts can be tested by DBAs in QA & Production.

But smaller teams may do slightly different.