[Dailydave] [Fwd: Why I love Spike..]

[Matt Hargett]

Mike Bailey wrote:
> 
>>I tried hard to make SPIKE as unusable as possible, and yet 
>>still people use it. QA teams, for the most part, don't 
>>however. I believe this is because most of them don't have 
>>anyone on staff who knows C, which is probably the original 
>>idea behind Greg Hoglund putting a GUI on a fuzzer and trying 
>>to resell it to them. I still think this would work, if you 
>>priced it right. Something simple, but customizable.
> 
> 
> It would certainly sell.  Make it really intuitive to help coddle the
> process along and relatively inexpensive so as not to choke the budgets and
> people would use it.  From QA staff to sysadmins to security folk.   It's
> already powerful, It just needs "simple" to get into the hands of the
> masses.  

I beg to differ. After working in QA for 6 years (mostly in the security
space), I did a lot of "fuzzing" testing manually, built custom tools to
do it, combined it with runtime analysis, etc. When I went to
ClickToSecure to work on Hailstorm, I thought it was the perfect tool
for security QA people. I was wrong.

The approach is intrinsically flawed, far too involved given the amount 
of time people are generally given to test, has no good way to do fault 
detection, has no good way to measure code coverage (and therefore the 
effectiveness of the fault injection), etc, etc. This is all detailed in 
a talk I have yet to give. (Maybe I should resubmit to Blackhat?)

You need more than an easy UI or "people who know C" to make this 
palatable. You have to understand processes (and lack thereof) people 
generally have in their organizations, and the real world needs and 
wants. Also, "simple" and "customizable" turn out to be at odds with 
eachother. It's amusing to see companies like "Imperfect Networks" and 
others trying to do this over again, but "better". They're making the 
same mistakes, but correcting the things that weren't wrong in the first 
place.

Hailstorm did indeed have an audience, one with money, but it was not 
with QA people or developers. It was security researchers and IT 
security folks. I may eat my words some day, but I am going to say that 
the current black box approaches will never be palatable to a market 
large enough to sustain a company for any length of time while there are 
other/better methods available.

Hailstorm was a great bunch of lessons that I learned from and put into 
BugScan's business and technology. It has worked out really well for me, 
but figuring out the real problems and being objective about it was more 
difficult than it might sound at first.

</rant>