When my DOM fuzzer finds a new bug, I want it to make a reduced testcase and notify me so I can file a bug report. To keep it from wasting time finding duplicates of known bugs, I maintain several ignore lists:
- 23 known crash signatures for 12 bugs
- 435 known assertions for 280 bugs
- 12 known types of small memory leaks for 10 bugs
- Known valgrind warnings for 11 bugs and a bunch of library/compiler bugs.
Some bugs are harder to distinguish based on output. In those cases, I use suppressions based on the fuzzer-generated input to Firefox:
Fixing any bug on those lists improves the fuzzer's ability to find additional bugs. But I'd like to point out a few that I'd especially like fixed:
- Bug 531550 has me ignoring the "invalid array index" assertion for nsTArrays.
- Bug 580790 has me ignoring "recursion level" assertions for PLDHashes.
- Bug 588237 has me ignoring a large number of crashes and assertions in layout.
In rare cases, I'll temporarily tell the fuzzer to skip a feature entirely:
- Bug 571613 has me not testing with accessibility enabled.
- Bug 576927 has me not testing text zoom.
- Bug 605271 has me not testing QueryInterface.
Several bugs interfere with my ability to distinguish bugs. Luckily, they're all platform-specific, so they don't prevent me from finding cross-platform bugs.
- Bug 610311 makes it difficult to distinguish crashes on Linux, so I ignore crashes there.
- Bug 612093 makes it difficult to distinguish PR_Asserts and abnormal exits on Windows. (It's fixed in NSPR and needs to be merged to mozilla-central.)
- Bug 507876 makes it difficult to distinguish too-much-recursion crashes on Mac. (But I don't currently know of any, so I'm not ignoring them at the moment!)