Reply to: Software Testing

[domcobb180]

You'll never get perfect documentation, but as a contractor neither will you be downing tools. You're (presumably) being hired for your expertise in a particular area - you'll have all sorts of references to create test scenarios from, whether this is use case scenarios from the business, or your own domain experience as the test oracle. Raise issues and risks in your strategy/plan, create test cases where possible, note that you'll be doing exploratory testing to complement your structured approach.


You're not going to be able to capture all defects. What you can do, though, is design tests that capture as many permutations and scenarios as possible for your test coverage. If the execution time of your tests is not possible within the time allocated for testing, you'll need to work with your PM/Dev/business to analyse for risk and priority.
Data type tests are good candidates for automation - many tools are free or included within dev apps.

Thanks. Agree wrt your suggested approach and also on the exploratory testing - that's how I'd look on any perm test assignment. (Wasn't serious on the downing tools suggestion!)

In terms of working as a test contractor, is it just a case of working through a limited company, getting contracts reviewed for possible liability and getting some PI then or do I need to print out and archive all documents/e-mails/test plans/approvals/marked up docs/results daily so I have an archive on leaving the client so that I can justify any future issues......(And I couldn't do this with many organisations as it would breach security rules.....) I've worked in IBs in London where vendors are treated well and have a great collaberative approach but I've also seen a lot of vendor companies having to expend considerable effort on negotiation where issues are found.

[meridian]

Agree in theory but in practise you find on a lot of projects this isn't always the case and you have to roll with a sub optimal documentation set or a set that's fairly high level with a lot of ambiguity - is it practical to refuse to work on test cases until a set of perfect or near perfect documentation exists.... As a perm you can negotiate on this but mostly there's a compromise. How do contract testers find this goes down - is it a case of downing tools until a perfect doc set is produced?

You'll never get perfect documentation, but as a contractor neither will you be downing tools. You're (presumably) being hired for your expertise in a particular area - you'll have all sorts of references to create test scenarios from, whether this is use case scenarios from the business, or your own domain experience as the test oracle. Raise issues and risks in your strategy/plan, create test cases where possible, note that you'll be doing exploratory testing to complement your structured approach.

In addition, you can test something and it works with one dataset or at one time slot (or in conjunction with other processes) but won't work with another but to test all possible data instances/process combinations would be impossible. Then you'd end up with a requirement failing under a certain context but probably out of 100 testers, none would find the issue or think to specify anything other than a fairly generic test case.

You're not going to be able to capture all defects. What you can do, though, is design tests that capture as many permutations and scenarios as possible for your test coverage. If the execution time of your tests is not possible within the time allocated for testing, you'll need to work with your PM/Dev/business to analyse for risk and priority.
Data type tests are good candidates for automation - many tools are free or included within dev apps.

[Freamon]

Agree in theory but in practise you find on a lot of projects this isn't always the case and you have to roll with a sub optimal documentation set or a set that's fairly high level with a lot of ambiguity - is it practical to refuse to work on test cases until a set of perfect or near perfect documentation exists.... As a perm you can negotiate on this but mostly there's a compromise. How do contract testers find this goes down - is it a case of downing tools until a perfect doc set is produced? In addition, you can test something and it works with one dataset or at one time slot (or in conjunction with other processes) but won't work with another but to test all possible data instances/process combinations would be impossible. Then you'd end up with a requirement failing under a certain context but probably out of 100 testers, none would find the issue or think to specify anything other than a fairly generic test case.

When you write your test cases, document any assumptions you've had to make about the prior state of the application and then get those signed off. The assumption could be "application will be set up with X set of test data and test will be executed at A,B and C times". These should all be documented in the behaviours, but if they are not then you just have to make assumptions (and get them agreed upon).

[russell]

Current work fits nicely with lifestyle / family. Work 3 or 4 days a week, mostly from home and getting paid well for it. Finishes at end of June, so might actually have to get proper work after that.

Ah sorry, I thought you were in a developer role and they had asked you to do testing. Sounds good.

[mudskipper]

Why are you doing this? Would you also clean the toilets? Or do a shift in the coffee shop.

Current work fits nicely with lifestyle / family. Work 3 or 4 days a week, mostly from home and getting paid well for it. Finishes at end of June, so might actually have to get proper work after that.

[russell]

It is a toughie. As a developer who's currently testing (and bored tulipless) I know I find defects that someone who hasn't got the technical background wouldn't pick up.

Having said that, most of them are obscure scenarios that would be unlikely to happen in the real world. And ClientCo seem to have a very tolerant approach to even quite major defects getting through to their customer facing websites and accept it as 'one of those things'.

Why are you doing this? Would you also clean the toilets? Or do a shift in the coffee shop.

[mudskipper]

It is a toughie. As a developer who's currently testing (and bored tulipless) I know I find defects that someone who hasn't got the technical background wouldn't pick up.

Having said that, most of them are obscure scenarios that would be unlikely to happen in the real world. And ClientCo seem to have a very tolerant approach to even quite major defects getting through to their customer facing websites and accept it as 'one of those things'.

[EternalOptimist]

Does few issues from UAT = a good system or just a poor UAT..... (or course you could have both... !)

Agree on the benefits of automated tools but not available in many places and not available to run through many combinations as various constraints in practise.

Thanks for the replies all - be interested to hear how any seasoned testers approach things and if you find it any different from perm testing?

well, I am the developer, not the tester. so few user complaints, issues, change requests or bugs, is good , but its probably more luck and tolerant users than anything else

[EternalOptimist]

It went live so wearing my consultant hat that's sign off. Everything is now a chargeable change request (even if it is a bug).

thats nothing, they asked me on wednesday to document the system. So I quoted 3 months

they had brown adrenalin running down their legs. hehhe

that'll learn them to get a system build with no spec, no test area, no uat and no plan.

anyway, they must like it, they asked me to do another

[domcobb180]

The infratructure dudes here finally set up a test area at ClientCos client, yippee, we now have a test system. three months after the system went live with bugger all UAT

Does few issues from UAT = a good system or just a poor UAT..... (or course you could have both... !)

Agree on the benefits of automated tools but not available in many places and not available to run through many combinations as various constraints in practise.

Thanks for the replies all - be interested to hear how any seasoned testers approach things and if you find it any different from perm testing?

[eek]

The infratructure dudes here finally set up a test area at ClientCos client, yippee, we now have a test system. three months after the system went live with bugger all UAT

It went live so wearing my consultant hat that's sign off. Everything is now a chargeable change request (even if it is a bug).

[EternalOptimist]

The infratructure dudes here finally set up a test area at ClientCos client, yippee, we now have a test system. three months after the system went live with bugger all UAT

[eek]

Agree in theory but in practise you find on a lot of projects this isn't always the case and you have to roll with a sub optimal documentation set or a set that's fairly high level with a lot of ambiguity - is it practical to refuse to work on test cases until a set of perfect or near perfect documentation exists.... As a perm you can negotiate on this but mostly there's a compromise. How do contract testers find this goes down - is it a case of downing tools until a perfect doc set is produced? In addition, you can test something and it works with one dataset or at one time slot (or in conjunction with other processes) but won't work with another but to test all possible data instances/process combinations would be impossible. Then you'd end up with a requirement failing under a certain context but probably out of 100 testers, none would find the issue or think to specify anything other than a fairly generic test case.

So you write you test specification based on the incomplete specification and record a brief set of areas which cannot be tested due to ambiguities within the specifications.

As for data sets and process combinations that is what automated testing is for. schedule to work through all the variations you can think of and ensure they work.

[domcobb180]

Are these issues cases where the product does not meet the business requirements (as documented)?

Or where the product does not exhibit a certain (previously documented) behaviour?

You can only test against the requirements/behaviours that are specified.

If the issue is some unexpected behaviour that the customer notices in live, but does not contradict any of the previously documented requirements/behaviours, then testing isn't the problem.

(A good way to spot this kind of problem early on is to review the specified requirements/behaviours to ensure they are written in such a way that it is very obvious exactly how you would test the product against them).

Agree in theory but in practise you find on a lot of projects this isn't always the case and you have to roll with a sub optimal documentation set or a set that's fairly high level with a lot of ambiguity - is it practical to refuse to work on test cases until a set of perfect or near perfect documentation exists.... As a perm you can negotiate on this but mostly there's a compromise. How do contract testers find this goes down - is it a case of downing tools until a perfect doc set is produced? In addition, you can test something and it works with one dataset or at one time slot (or in conjunction with other processes) but won't work with another but to test all possible data instances/process combinations would be impossible. Then you'd end up with a requirement failing under a certain context but probably out of 100 testers, none would find the issue or think to specify anything other than a fairly generic test case.

[AtW]

I enjoy failing developers when testing their tulipy software.

Also I love slurping coffee.

Mich the Bastard.