[Dailydave] [fuzzing] Coverage and a recent paper by L. Suto

[J.M. Seitz]

Andre,

> Developers can also improve automation through continuous 
> integration, automated static code analysis security 
> scanners, automated model-checkers, and build-integrated 
> system tests (e.g. using Canoo WebTest, Jameleon, staf.sf.net, etc).

Continuous integration is a definite plus, if you can couple a per-build
smoke test with a nightly build robustness test (fuzzing), you are sitting
pretty. And generally it's a great first warning that something is up. The
real problem is in designing an automation framework that is extensible
enough to not only allow custom fuzzers, but is able to adapt to changes in
the test targets, network parameters, protocol/file format changes, etc.

> Design and code review is difficult to automate, but we can "automate"
> processes through workflow tools.  Blueinfy has some free 
> tools that help with automating aspects of code review, and 
> Atlassian has a product called Crucible to improve code 
> review workflow.  Atlassian and ThoughtWorks/OpenQA are great 
> resources for continuous integration developer testing improvements.

Atlassian's JIRA also has some excellent workflow tools that can extend into
any one of their products and SVN, etc. It is easily automatable through
build tools like Tinderbox, Hudson, etc.
 
> More advanced operations/maintenance 
> security testing can be done using EFS and/or 
> WI/AppScan/Hailstorm against weak areas of code coverage (or 
> areas which represent more
> complexity) before adversaries do the same.  I wouldn't say 
> that these solutions are "custom" built in the same way as 
> complete protocol dissection, reverse engineering, or other 
> generated test scenarios.
> However, each of these tools does require some 
> "configuration", usually by an expert.

How do you know where your weak areas of code are? Is EFS going to
successfully dissect a binary protocol and give you metrics that are
useable? Not really. In my experience, and I love Jared like a brother, EFS
works very well against ASCII protocols, with a known set of seeds. If you
turn it loose against an unknown binary protocol, it doesn't do so well.
This is where we need to advance the whole notion of automated binary
protocol dissection, and then couple that will a block-based (or genetic if
you want) fuzzer that can then utilize the results that your dissection
yielded. Look for something like this in December, by some goofy looking
Canadian dude :)

 
> Unfortunately, the largest problem we're facing with the 
> above strategies is the growing rate of code vs. the growing 
> rate of qualified experts.  Make sure to check out Felix 
> Linder (FX)'s talk from HITB 2007, available here - 
> http://conference.hackinthebox.org/hitbsecconf2007kl/materials
> /D2T1%20-%20Felix%20Lindner%20-%20%20%09%20Attack%20Surface%20
> of%20Modern%20Applications.pdf

You got that right, in a development shop you are lucky to have one person
in the whole company that can actually test the corner cases, use a debugger
to track down problems (without source), demonstrate attack surface,
yadayada. Problem is there are all kinds of jobs for security engineers,
pentesters, app-pen testers, etc. But how often do you see a job posting
like: "Hax0r In the QA Department Needed". That's where the demand _has_ to
start moving towards. And please folks, pay your QA hackers well :)

JS