How my DOM fuzzer ignores known bugs

When my DOM fuzzer finds a new bug, I want it to make a reduced testcase and notify me so I can file a bug report. To keep it from wasting time finding duplicates of known bugs, I maintain several ignore lists:

Some bugs are harder to distinguish based on output. In those cases, I use suppressions based on the fuzzer-generated input to Firefox:

Fixing any bug on those lists improves the fuzzer's ability to find additional bugs. But I'd like to point out a few that I'd especially like fixed:

In rare cases, I'll temporarily tell the fuzzer to skip a feature entirely:

Several bugs interfere with my ability to distinguish bugs. Luckily, they're all platform-specific, so they don't prevent me from finding cross-platform bugs.

  • Bug 610311 makes it difficult to distinguish crashes on Linux, so I ignore crashes there.
  • Bug 612093 makes it difficult to distinguish PR_Asserts and abnormal exits on Windows. (It's fixed in NSPR and needs to be merged to mozilla-central.)
  • Bug 507876 makes it difficult to distinguish too-much-recursion crashes on Mac. (But I don't currently know of any, so I'm not ignoring them at the moment!)

6 Responses to “How my DOM fuzzer ignores known bugs”

  1. John Regehr Says:

    Nice! But it’s sad to have to implement these kinds of mechanisms– people should just fix reported bugs quickly :). We don’t have any of this kind of stuff and basically just stop testing a compiler when reported bugs don’t get fixed in a timely fashion…

  2. Neil Rashbrook Says:

    Do you try fuzz testing builds with the proposed patches?

  3. Jesse Ruderman Says:

    I occasionally pre-test a big patch (e.g. harfbuzz) or pref flip (e.g. html5.enable). But mostly I test mozilla-central debug builds straight from Tinderbox.

  4. mmc Says:

    It’s sad to see all those bugs assigned to nobody…

  5. Jesse Ruderman Says:

    A full list of bugs found through fuzzing would show that about 80% have been fixed. It’s only depressing if you focus on the unfixed bugs ;)

  6. Brian Says:

    First bug on the list now fixed! :)