False positives

[NotAllThere]

The Inland Revenue have a new machine that can detect IR35-caught contractors. It's good, at 90% accurate (10% false positives). As the machine scans this years self-assessment papers 3,000 contractors are tested and the first 30 escape detection, while the 31st fails. How sure are you that this person is due a lengthy legal battle?

I get you - it's a logic/maths problem.

The first 30 escape detection - do you mean "the first thirty who are tax evading scumbags", or "the first thirty tested"?

If the former, then I'm a 100% sure - unless they roll-over and admit guilt immediately.

If the latter (which I guess you really mean), then it's all about expectation probability. What is the expectation that the 31st person will be flagged up by the machine?

Now, one in ten of those flagged up will be innocent. The chance of it being a specific individual of the 3000 is 1/3000. So, the chances that the 31st person is guilty, is:

1/3000 * 9/10 = 9/30000 = 3/10000 = 0.03%.

Which is probably wrong, especially as it isn't one of the options, but also expectation probability makes my head hurt.

[TimberWolf]

I get you - it's a logic/maths problem.

The first 30 escape detection - do you mean "the first thirty who are tax evading scumbags", or "the first thirty tested"?

If the former, then I'm a 100% sure - unless they roll-over and admit guilt immediately.

If the latter (which I guess you really mean), then it's all about expectation probability. What is the expectation that the 31st person will be flagged up by the machine?

Now, one in ten of those flagged up will be innocent. The chance of it being a specific individual of the 3000 is 1/3000. So, the chances that the 31st person is guilty, is:

1/3000 * 9/10 = 9/30000 = 3/10000 = 0.03%.

Which is probably wrong, especially as it isn't one of the options, but also expectation probability makes my head hurt.

That's the general idea. You could probably come up with any number of answers, depending on wording and interpretation of the problem. I can't say I'm particularly happy with the conclusion and wording of the one I sourced mine from, specifically that by 'accuracy' they implicitly mean false positives without saying so and where the assumption that 1 of the 3000 is definitely a terrorist comes from. As you say it makes your head hurt.

Original text http://news.bbc.co.uk/1/hi/magazine/8153539.stm

[RichardCranium]

As you say it makes your head hurt.

Only because of the "90% accurate" figure which is insufficient to draw a meaningful conclusion. What does it mean?

If 1 in 3000 people is a terrorist, there are 20,000 terrorists in our 60,000,000 population.

Test 100 terrorists with the machine. How many does it say are terrorists? If less than 100 (and it probably will be) that gives you a rate for false negatives.

Test 100 non-terrorists (which is in itself a silly term, we all have the potential if driven). How many does it say are terrorists. If more than 0 (and it probably will be) that gives you the rate for false positives.

Different kinds of tests are sensitive in different ways. Some are great for capturing those with a disease (for example); others are good for capturing those without a disease. They have varying rates for false negatives and false positives.

It is by conducting more than one test that one's confidence can rise enormously; 4 or 5 dodgy tests can produce very high confidence when all are undertaken ... for some of the results. For a large %age there will still only be varying degrees of confidence.

So, your terrorism machine. What does 90% accurate mean?

Does it mean that of the 2,999 innocents, 299 will be identified as terrorists?

Does it mean that 1 in 10 terrorists get through?

Ready for the nasty bit?

If your machine NEVER identifies terrorists as terrorists, but says 299 out of every 3000 innocents ARE terrorists, it is 90% accurate.


This is yet another classic case of: the question is broken.