Re: Fwd: Testing for Shellshock...

[Eduardo A . Bustamante López]

> I believe it would be interesting to measure the combinatorial coverage of
> the fuzz tests.
You already asked this... why didn't you follow up in the thread that
you started?

> [...] So we would be able to say what percentage of 2-way,
> 3-way, etc. combinations are covered,  which would be useful in
> understanding the difficulty of finding the bug using tests.  For example,
> it may be that the test script produces a high level of 4-way combination
> coverage, suggesting that the bug is complex and that a high proportion of
> the possible input space needs to be covered by tests to detect the problem
> (that we would need 4-way or 5-way testing).
Uh, what? This seems like a school/research project for a course that
I obviously don't know, but would you mind explainin what n-way
combinations are and why should we care?

> This may depend on how tailored the fuzz tests are for finding this
> particular bug.  Obviously they could be very narrowly tailored and not
> cover much of the input space.  We would have to measure the coverage to
> see.
Uh, but, you see... they already found bugs with that approach.
Either we fiddle with that approach to increase the number of tests
and possibly uncover more bugs, or make the approach more general,
but I'm not sure this is feasable. What I remember from the person that
run a fuzzer on bash, sie specifically reduced the possibilities by
testing just a small part of bash (exported functions).

> I’m asking this group for feedback... do you think the fuzz test scripts
> that have been developed could be analyzed in this way?   To measure the
> combination coverage, we just need test values in a matrix or spreadsheet
> format, where each row is a test and each column represents a parameter.
Uh, again, what? Also, again, follow up in the original thread.

> Could we run the fuzz test scripts and produce such a matrix?
Who is 'we'? If you're willing to invest some CPU cycles on that,
please do it :)