[Dailydave] [fuzzing] Coverage and a recent paper by L. Suto

[Alexander Sotirov]

On Sat, Oct 27, 2007 at 09:25:47AM +0200, Nicolas RUFF wrote:
> > Using the following perl script two buffer overflows are detected:
> > cat vuln.c | perl -ne '/rnd\[i\]/ and print "Buffer overflow!\n"'
> > This post does have a point. Discuss among yourselves.
> 
> Is this vendor bashing, maybe ? ;)

Not at all. I've written static analysis tools myself and I know how hard a
problem it is, so I have nothing but respect for the people trying to solve it.

My point is that comparing static analysis tools by testing them on a single
vulnerable function is a very poor way to test their performance. It is very
easy to construct samples that will show the strenghts of one tool and the
weaknesses of the others (see my vulncheck paper for a great example of that),
as well as to write a Perl one-liner that will beat all commercial tools when
run on a single program.

It's a very similar situation to compiler benchmarking. Microbenchmarks are a
great way to test specific types of optimizations, but they don't reflect the
real-world preformance of the compiler.

The only way to compare static analysis tools is to use a large sample set of
real vulnerabilties and measure the false positive and false negative rates.
Everything else is a waste of time.

Alex
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 186 bytes
Desc: not available
Url : http://lists.immunitysec.com/pipermail/dailydave/attachments/20071028/f5865156/attachment.pgp