Indistinguishable from Jesse

Assertions frequently help me find bugs that would be difficult to catch otherwise. I've filed 479 bugs on assertion failures, excluding bugs that also causes crashes or hangs, and 223 of those have already been fixed. (Thanks to Brendan, I can safely file bugs when I see assertion failures, knowing that my bugs aren't invalid.)

Many assertion failures indicate that assumptions were violated in a relatively harmless way, merely causing incorrect layout or slow performance in an edge case. But sometimes they indicate the presence of a bug that could lead to a serious memory safety violation. For example, "ASSERTION: Some objects allocated with AllocateFrame were not freed" in the FrameArena destructor has helped find dozens of bugs that lead to leaks and/or dangling pointers.

Gecko developers have added many useful assertions lately, and it looks like more are on the way. Regression tests for assertions are improving, too.

I have a list of assertions that I ignore during automated testing. Whenever a developer fixes a bug on that list, I remove the bug and the corresponding assertion from the file, so if I hit the assertion again after that point, I'll notice it.

In addition to helping to catch bugs, assertions also serve as "living documentation" about the code's invariants. We notice if they become out of date, unlike Wiki pages and comments. In my dream world, assertions would also serve as waypoints for an automated theorem prover ;)

Apple employee Greg Parker has ported Valgrind to Mac, and plans to release his work soon after Leopard is released in October. He's been working on it for quite a while.

I'm excited about being able to use Valgrind on Mac. Valgrind's "Memcheck" is much better at catching dangling-pointer bugs and heap buffer-overflow bugs than simply watching for crashes (even with MallocScribble enabled). Running a fuzzer with Memcheck can reveal exploitable memory safety bugs that would not have triggered crashes otherwise.

I wrote a tool called Lithium that automatically reduces large testcases, such as real-world web pages or testcases produced by jsfunfuzz. It can usually reduce a 3000-line jsfunfuzz crash testcase to 3-10 lines in several minutes, considerably faster than I can reduce by hand. Perhaps more importantly, I can do something else while it reduces the testcase.

• Documentation for using Lithium
• Description and analysis of Lithium's algorithm
• Download Lithium (0.9.1)

There are two (related) reasons I'm not calling it "Lithium 1.0" yet. First, I'm hoping to improve the way "interestingness tests" are written. Currently, they're separate programs that communicate to Lithium using their exit code, which limits error handling and might slow Lithium down. I'd like to make the interestingness tests be Python files, but I'm not sure what the best way to do that is. (Should Lithium __import__ the interestingness test? Or should the interestingness test import Lithium and be renamed to e.g. "reduce_crash.py"?)

Second, it would be useful to be able to pass extra arguments to the program being tested. For example, it would be useful to be able to pass a profile name to Firefox, or to pass a Firefox path to Valgrind. One possibility is to put the program being tested last on the command line, so extra positional arguments become options to that program. This solution would only work for interestingness tests that launch a single program (so it wouldn't work for a "renders differently in these two Firefox builds" test, for example), but maybe that's okay. Another possibility is to require the use of a config file for passing arguments to programs being tested (so you don't end up typing all of ".../firefox-bin -P foo" on Lithium's command line).

I'll probably use the MIT license for Lithium (but not for timed_run.py, which was mostly written by Chris Cooper and Bob Clary).

Opera has posted a build with fixes for several crashes found by jsfunfuzz. Cool!

Opera community members posted dozens of comments about it, and I replied to several.

Before the presentation:

• ComputerWorld: Mozilla to give away own security testing tools
• InternetNews: Black Hat Cometh; You Afraid?
• Datamation: Will Mozilla's Fuzzer Break The Web?
• eWeek Channel Insider: Mozilla to Release JavaScript Fuzzer for Firefox

After the presentation:

• PC World Staff Blog: Mozilla Releases Hacker Tools
• InternetNews: Mozilla Puts The Fun in Fuzz
• cnet news.com: Mozilla releases browser testing tools
• cybernetnews: Both Opera and Firefox Benefit from Mozilla’s jsfunfuzz
• InfoWorld: Mozilla shares scanning tool, Firefox 3 features
• Mozilla Delivers Security Tools, Previews Firefox 3 At Black Hat
• TechSpot: Mozilla releases browser-abusing tool
• Avencius: Mozilla helps Opera in fixing JavaScript bugs
• Digg: Mozilla Releases Hacker Tools

Fuzz-testing is usually only used to find crashes and assertion failures, but my JavaScript engine fuzzer goes beyond catastrophic failures when it tests the decompiler. It checks the decompiled code for signs of incorrectness in two ways.

First, it checks that the decompiled code compiles without giving syntax errors. This finds fun bugs like bug 346904 where the decompiler screwed up in an understandable way, as well as bugs like bug 351496 where the decompilation is complete nonsense.

Second, it checks that the decompiled code is canonical -- compiling and decompiling again should give the exact same representation as the original decompilation. This helps find bugs like bug 381196 where decompilation changes the meaning of the code without introducing a syntax error.

Some decompilation changes, such as bug 352068, did not change the meaning of the code and simply reflected varying amounts of optimization in the compiler. Early in the fuzzer's life, I was able to convince convince Brendan that it was worth fixing many of those otherwise harmless "round-trip changes" in order to make it possible to find other bugs with this method.

This pair of checks doesn't find all decompiler bugs, of course, but it finds quite a few of them. jsfunfuzz has a few other correctness checks for things like unnecessary parentheses in decompiled code and bogus results from object uneval.

Can you think of other ways to use fuzz-testing to find "correctness" bugs?

I wrote a fuzzer called jsfunfuzz for testing the JavaScript engine in Firefox. Window, Shaver, and I announced it at Black Hat earlier today, as part of Mozilla's presentation, "Building and Breaking the Browser".

It tests the JavaScript language engine itself, not the DOM. (That means that it works with language features such as functions, objects, operators, garbage collection rather than DOM objects accessed through "window" or "document".)

It has found about 280 bugs in Firefox's JavaScript engine, over two-thirds of which have already been fixed (go Brendan!). About two dozen were memory safety bugs that we believe were likely to be exploitable to run arbitrary code.

In the presentation, I speculated as why it has been able to find so many bugs:

• It knows the rules of the JavaScript language, allowing it to get decent coverage of combinations of language features.
• It breaks the rules, allowing it to find errors in syntax error handling such as bug 350415 and more generally helping the fuzzer avoid having "blind spots".
• It isn't afraid to nest JavaScript constructs in fairly complicated ways, like when it found bug 353079.
• It allows state to accumulate by creating and running functions in a loop. (See bug 361346 for an example of a bug that would be hard to find otherwise.)
• It tests for correctness, not just crashes and assertions. (Since I didn't talk about this aspect much during the security-focused Black Hat presentation, I've made it a separate blog post.)

If you want to test it out, grab jsfunfuzz.js and multi_timed_run.py from jsfunfuzz and build the standalone JavaScript Shell. (It can be tested inside a web browser, but it works better in a standalone shell where output is simple and restarting isn't painful.)

Mike Connor's cat is a natrual fuzz tester.