Go, Suity!

[minestrone]

This is pretty much how most financials operate, everything is spread too thin and nobody gives fook about what you think of it, they just want a result and they often want it that day.

You have to develop ways to work to make sure impact is low if it goes wrong. Releasing stuff early and keeping it hidden from the user goes on, then you can test in in prod in isolation. Stuff like that. You have to change your mind set, "doing a proper job" approach is going to get laughed out the building.

[Ticktock]

From my point of view, what I'd look at is:

1. What does this system actually do? Is it customer facing? If internal only, is it critical? Would your "fix" affect the whole system, a critical part, or a "nice to have" bit?

2. Can you clearly write an email explaining what you propose to do, and the risks if it goes tits up? If so, can you get someone to sign off on it so that they carry the can if it fails?

3. Can you easily do a roll-back if it fails, preferably before the users realise what you've done?

4. If you don't get the new release out, does this cause more problems than going ahead and it failing?

If you can pass those tests then go ahead, as the worst case is that you're back to where you are now, someone else takes the blame, or something is out of action that annoys users, but doesn't stop them doing their job / customers are unaffected.

Then you can put together the plan for how to do things "properly" for next time, and don't let them get away with saying "we managed it last time, so can do the same again".

[minestrone]

1. What does this system actually do? Is it customer facing? If internal only, is it critical? Would your "fix" affect the whole system, a critical part, or a "nice to have" bit?

It is the vending machine on level 4. Geoff in accounts has not been able to get a cup a soup for days.

[MarillionFan]

It is the vending machine on level 4. Geoff in accounts has not been able to get a cup a soup for days.

But if Geoffs VP cannot get his hot chocolate heads will roll.

[vetran]

Actually on the Intermediate environment I agree totally with Suity. You have to have one to test deployment. It should be a perfect copy of live. You then have one for the developers to play on called Dev.

They can muck about as much as they like on Dev but they have to write down everything you need to do to get from Dev to Staging so a non technical person can do it. On Staging / QA you have real data and the admins are cleared to see it. Until you can deploy, run overnights & EOM & roll backs etc it doesn't go anywhere near live. Timing of processes and loading on staging versus live should be done before & after.

In your case I would insist they wipe dev & reload with a recent copy of live. Turn the logging on and have the dev's write a document for promotion. Don't let them touch the keyboard!

Once you have a good deployment thoroughly tested clean the confidential data and give the Devs access to their new Dev Environment.

You can use it as a DR test as well. Maybe use your DR server as the staging server? Make sure you have good backups and a spare set of disks so you can get DR back up in time.

Of course you can Nike it and screw up live if you want as suggested here. Then you can ask if RBS have any jobs.

Virtual servers are your friends in a lot of cases.

[greenlake]

It is the vending machine on level 4.

Uh oh....BSOD.....

[Bunk]

It is the vending machine on level 4. Geoff in accounts has not been able to get a cup a soup for days.

But if Geoffs VP cannot get his hot chocolate heads will roll.

Oh FFS! Suity, get a notepad, take down their order, then go up to level 5 and get the drinks. Once the urgent stuff is out of the way you can start planning how you're going to switch your broken machine with the nice shiny one upstairs.

Oh and you better take the lift, otherwise you'll need to add "Risk of hot spillages" to the RAID log

[minestrone]

The system is probably so noddy that they wont give him a SIT env. By the sound of it he has walked in and done the changes himself so it hardly sounds complicated.

As usual the problem is suity. Nobody else on here messes up every contract like he does, he is the constant.

[Ticktock]

It is the vending machine on level 4. Geoff in accounts has not been able to get a cup a soup for days.

Which technician is he?

[Old Greg]

Which technician is he?

They're tough. They're mean.
Suity's Z team gets things clean.