How my DOM fuzzer ignores known bugs
When my DOM fuzzer finds a new bug, I want it to make a reduced testcase and notify me so I can file a bug report. To keep it from wasting time finding duplicates of known bugs, I maintain several ignore lists:
- 23 known crash signatures for 12 bugs
- 435 known assertions for 280 bugs
- 12 known types of small memory leaks for 10 bugs
- Known valgrind warnings for 11 bugs and a bunch of library/compiler bugs.
Some bugs are harder to distinguish based on output. In those cases, I use suppressions based on the fuzzer-generated input to Firefox:
Fixing any bug on those lists improves the fuzzer's ability to find additional bugs. But I'd like to point out a few that I'd especially like fixed:
- Bug 531550 has me ignoring the "invalid array index" assertion for nsTArrays.
- Bug 580790 has me ignoring "recursion level" assertions for PLDHashes.
- Bug 588237 has me ignoring a large number of crashes and assertions in layout.
In rare cases, I'll temporarily tell the fuzzer to skip a feature entirely:
- Bug 571613 has me not testing with accessibility enabled.
- Bug 576927 has me not testing text zoom.
- Bug 605271 has me not testing QueryInterface.
Several bugs interfere with my ability to distinguish bugs. Luckily, they're all platform-specific, so they don't prevent me from finding cross-platform bugs.
- Bug 610311 makes it difficult to distinguish crashes on Linux, so I ignore crashes there.
- Bug 612093 makes it difficult to distinguish PR_Asserts and abnormal exits on Windows. (It's fixed in NSPR and needs to be merged to mozilla-central.)
- Bug 507876 makes it difficult to distinguish too-much-recursion crashes on Mac. (But I don't currently know of any, so I'm not ignoring them at the moment!)
November 21st, 2010 at 11:10 pm
Nice! But it’s sad to have to implement these kinds of mechanisms– people should just fix reported bugs quickly :). We don’t have any of this kind of stuff and basically just stop testing a compiler when reported bugs don’t get fixed in a timely fashion…
November 22nd, 2010 at 3:08 am
Do you try fuzz testing builds with the proposed patches?
November 22nd, 2010 at 3:29 am
I occasionally pre-test a big patch (e.g. harfbuzz) or pref flip (e.g. html5.enable). But mostly I test mozilla-central debug builds straight from Tinderbox.
November 22nd, 2010 at 4:01 am
It’s sad to see all those bugs assigned to nobody…
November 22nd, 2010 at 4:53 am
A full list of bugs found through fuzzing would show that about 80% have been fixed. It’s only depressing if you focus on the unfixed bugs ;)
November 24th, 2010 at 5:26 am
First bug on the list now fixed! :)