Introducing jsfunfuzz
I wrote a fuzzer called jsfunfuzz for testing the JavaScript engine in Firefox. Window, Shaver, and I announced it at Black Hat earlier today, as part of Mozilla's presentation, "Building and Breaking the Browser".
It tests the JavaScript language engine itself, not the DOM. (That means that it works with language features such as functions, objects, operators, and garbage collection rather than DOM objects accessed through "window" or "document".)
It has found about 280 bugs in Firefox's JavaScript engine, over two-thirds of which have already been fixed (go Brendan!). About two dozen were memory safety bugs that we believe were likely to be exploitable to run arbitrary code.
In the presentation, I speculated as why it has been able to find so many bugs:
- It knows the rules of the JavaScript language, allowing it to get decent coverage of combinations of language features.
- It breaks the rules, allowing it to find errors in syntax error handling such as bug 350415 and more generally helping the fuzzer avoid having "blind spots".
- It isn't afraid to nest JavaScript constructs in fairly complicated ways, like when it found bug 353079.
- It allows state to accumulate by creating and running functions in a loop. (See bug 361346 for an example of a bug that would be hard to find otherwise.)
- It tests for correctness, not just crashes and assertions. (Since I didn't talk about this aspect much during the security-focused Black Hat presentation, I've made it a separate blog post.)
If you want to test it out, grab it from the bug report. I recommend running it in a standalone JavaScript Shell, as it is much faster to start and shut down than a whole browser.
Update (2015): newer versions are available in the funfuzz repository on GitHub
August 2nd, 2007 at 8:47 pm
I’ll have to play around with that on Rhino. Yay!
August 3rd, 2007 at 12:59 am
[…] brings us to Jesse’s fuzzer. I’m curious if Jesse’s fuzzer just unleashed hell on the browser teams. Based on this, […]
August 3rd, 2007 at 2:10 am
Does this mean you have found ways to run arbitrary code through old version of flash?
This is the tamarine JS engine isn’t it?
I think this may deserve a bit more press than it’s getting. I mean, one or two compromised flash adverts could create a giant bot-net.
monk.e.boy
August 3rd, 2007 at 2:16 am
Flash and Firefox don’t share any part of their ECMAScript implementation yet. Even in the future, only the JIT will be shared; the ES-to-bytecode compilers (and bytecode-to-ES decompilers?) will remain separate.
That said, I’d expect the Flash team to be as interested in this fuzzer as any browser vendor, since it might find memory safety bugs in any ECMAScript implementation.
August 3rd, 2007 at 11:26 am
[…] developed by Mozilla, and so you would expect it to help them the most…and it has. Using it they’ve found 280 bugs in Firefox’s JavaScript engine with about two-dozen of those that could have been […]
August 4th, 2007 at 5:10 pm
Can this be used by other browsers (Safari, IE, etc) to validate their JS implementations?
August 4th, 2007 at 7:31 pm
“Can this be used by other browsers (Safari, IE, etc)”
Yes. Opera has already put out a build for testing that includes fixes for four crashes (one of which might have been exploitable). Safari has already fixed several decompilation bugs.
“to validate their JS implementations?”
Fuzzers don’t really validate. They find bugs but they can’t tell you “there are no more bugs”.
August 6th, 2007 at 1:30 am
I tried accessing “jsfunfuzz.js” from “https://bugzilla.mozilla.org/show_bug.cgi?id=jsfunfuzz”. However I got the error
“Sorry, you are not authorized to access this attachment.”.
When are you planning to make it available to every one?
August 6th, 2007 at 1:49 am
Grab the last attachment with the name “jsparsefuzz.js”. That’s the most recent public version. I’ve shared the most recent version in the bug with other browser vendors and I’m planning to make it public in a few weeks.
August 6th, 2007 at 7:59 pm
[…] Ruderman just introduced a fantastic new utility for probing JavaScript implementations called jsfunfuzz. Specifically, this […]
August 8th, 2007 at 7:48 pm
This is all very well and good for JavaScript, but what about other relatively new bits of the engine like SVG? I am not a security expert but it seems obvious to me given the success here that any point at which the browser accepts external input should be fuzz tested.
Also can you go into more detail about how “it knows the rules” and “it breaks the rules”? It seems to me that it would be easy to miss possible vulnerabilities by having incorrectness in the former or incompleteness in the latter.
As fuzzy testers are undoubtably now an important part of the black hat arsenal too, I think every effort should be made to make the ones used by white hats as comprehensive as possible, and to come up with guidelines for best practice.
August 8th, 2007 at 9:31 pm
Why just the new bits? ;)
I think it’s best to focus on external input whose handling is (1) complicated and (2) done in an unsafe manner (e.g. C++ with raw pointers flying around). JavaScript and SVG certainly meet those criteria.
Sure. It “knows” how to use most of the JavaScript syntax: loops, properties, operators. It also “knows” that certain things are functions, and certain things want functions (e.g. setters, the argument to Array.map). It “breaks” the rules by substituting one syntax element for another (see the “totallyRandom” bits) and by screwing up the syntax slightly (see the function called “cat”).
You’re right: incorrectness or incompleteness in the former, and incompleteness in the latter, can lead to it missing bugs (including bugs that can be found by small modifications to the fuzzer — fuzzing is never about guaranteeing that you don’t have any bugs). I’m not sure what to do about that, but having More Eyes on the fuzzer probably helps :)
I’d prefer smart static analysis, but if fuzzing is the best we can do, sure :)
This could be tricky, since there are many fuzzing strategies and I’m not sure there’s one that’s “best”. “Mutation fuzzers” that only make small changes to existing files are easy to make; “generating fuzzers” have to know the syntax and tend to find more bugs; “genetic fuzzers” are clever and have interesting behaviors.
I usually write “generating fuzzers”. For those, my guidelines might be the ones above under “why it has been able to find so many bugs”, plus one additional guideline: make sure you can make a reduced testcase easily when your fuzzer finds a bug.
August 9th, 2007 at 3:34 am
[…] por pilsen on 9/08/07 Fue una persona que desarrollo esta herramienta para testear el motor javascript del Firefox y asÃ, con esta […]
August 9th, 2007 at 11:52 pm
[…] Ruderman has released jsfunfuzz, a fuzz tester that hunts for edge cases that can break things. It tests the JavaScript […]
August 10th, 2007 at 12:32 pm
It sounds like more of a Model-Based Tester than a fuzz tester. (http://en.wikipedia.org/wiki/Model-based_testing)
August 23rd, 2007 at 2:59 pm
[…] the last few years which tries to address the “state explosion” of real-world programs (applied successfully to the Firefox Javascript engine to find 280 bugs). This, of course, places even more burden on the […]