[Dailydave] Debugging the false alarm problem.

[dan at geer.org]

Dave, et al.,

The answer to (almost) any testing problem is to 
do something that is multi-stage.  The reason is
that, as you were suggesting, it is impossible to
eliminate all errors but -- and this is the good
part -- you can bias one stage to err in a known
direction and bias a subsequent stage to err in
the opposite direction.  This is what, at the
hardware level, many signal filters do so as to
get "band-pass" outputs, and it is likewise how 
we (they) can afford to screen all the blood supply
for HIV.

It works like this: Stage 1 misses nothing but has
a high false positive rate.  Because it misses 
nothing, that which it declares to be negative can
be discarded as of no further interest.  Stage 2
has the reverse characteristic or, as the more
frequent alternate, Stage 2 is much better but 
much more expensive.  In either case, you've used
Stage 1 to rule-out and Stage 2 to rule-in.

Not advertising, but this is in somewhat greater
detail in slides 75-100 at

http://geer.tinho.net/usenix/measuringsecurity.tutorialv2.pdf

which also includes some of the standard terminology
used by diagnostic testing and information retrieval
folks, which terminology I suggest that we in the
security field adopt.

--dan