High Consequence Testing

By Jim Porto

On 5 December 2006 I attended the Science Community Meeting on Security, a FBI sponsored conference in Washington, DC as UNC-Chapel Hill’s representative. It was clear that there were differences between the academic community and the FBI on how to detect criminal activity on campus. The effectiveness of criminal activity “tests” can be measured by its sensitivity and its specificity. The perfect test will be highly effective if test results yield positive results for all the guilty parties (high sensitivity), and negative results for all the innocent parties (high specificity). Unfortunately, no test can do this. In practice, there is a trade off. The more sensitive the test, the more likely it will ensnare the innocent. The more specific the test, the more likely the guilty will not be detected.

After the meeting, I graphed this trade-off.

The FBI wants a low threshold for cutoff of suspicious activity to make the test more sensitive, even though many innocent victims will be arrested. Academics want a high threshold to make the test less sensitive, even though some of the guilty will not be caught. In other words the FBI wants to minimize false negatives (missing the bad guys) and academics want to minimize false negatives (ensnaring the good guys). As a general rule, it appears to me that the higher the consequence of the act, the more sensitive we would want to make the test. However actions on these high consequence events need to greatly outweigh the threats to individual liberties. The determination for highly sensitive tests should be made through a rational process that explicitly gives a high weight to civil liberties.