Introducing jsfunfuzz

I wrote a fuzzer called jsfunfuzz for testing the JavaScript engine in Firefox. Window, Shaver, and I announced it at Black Hat earlier today, as part of Mozilla's presentation, "Building and Breaking the Browser".

It tests the JavaScript language engine itself, not the DOM. (That means that it works with language features such as functions, objects, operators, and garbage collection rather than DOM objects accessed through "window" or "document".)

It has found about 280 bugs in Firefox's JavaScript engine, over two-thirds of which have already been fixed (go Brendan!). About two dozen were memory safety bugs that we believe were likely to be exploitable to run arbitrary code.

In the presentation, I speculated as why it has been able to find so many bugs:

  • It knows the rules of the JavaScript language, allowing it to get decent coverage of combinations of language features.
  • It breaks the rules, allowing it to find errors in syntax error handling such as bug 350415 and more generally helping the fuzzer avoid having "blind spots".
  • It isn't afraid to nest JavaScript constructs in fairly complicated ways, like when it found bug 353079.
  • It allows state to accumulate by creating and running functions in a loop. (See bug 361346 for an example of a bug that would be hard to find otherwise.)
  • It tests for correctness, not just crashes and assertions. (Since I didn't talk about this aspect much during the security-focused Black Hat presentation, I've made it a separate blog post.)

If you want to test it out, grab jsfunfuzz.js and multi_timed_run.py from jsfunfuzz and build the standalone JavaScript Shell. (It can be tested inside a web browser, but it works better in a standalone shell where output is simple and restarting isn't painful.)

16 Responses to “Introducing jsfunfuzz”

  1. Erika Says:

    I’ll have to play around with that on Rhino. Yay!

  2. Inside Firefox » Blog Archive » So, did Jesse’s fuzzer break the other browsers… Says:

    […] brings us to Jesse’s fuzzer. I’m curious if Jesse’s fuzzer just unleashed hell on the browser teams. Based on this, […]

  3. monk.e.boy Says:

    Does this mean you have found ways to run arbitrary code through old version of flash?

    This is the tamarine JS engine isn’t it?

    I think this may deserve a bit more press than it’s getting. I mean, one or two compromised flash adverts could create a giant bot-net.

    monk.e.boy

  4. Jesse Ruderman Says:

    Flash and Firefox don’t share any part of their ECMAScript implementation yet. Even in the future, only the JIT will be shared; the ES-to-bytecode compilers (and bytecode-to-ES decompilers?) will remain separate.

    That said, I’d expect the Flash team to be as interested in this fuzzer as any browser vendor, since it might find memory safety bugs in any ECMAScript implementation.

  5. Both Opera and Firefox Benefit from Mozilla’s jsfunfuzz - CyberNet News Says:

    […] developed by Mozilla, and so you would expect it to help them the most…and it has. Using it they’ve found 280 bugs in Firefox’s JavaScript engine with about two-dozen of those that could have been […]

  6. Manoj Says:

    Can this be used by other browsers (Safari, IE, etc) to validate their JS implementations?

  7. Jesse Ruderman Says:

    “Can this be used by other browsers (Safari, IE, etc)”

    Yes. Opera has already put out a build for testing that includes fixes for four crashes (one of which might have been exploitable). Safari has already fixed several decompilation bugs.

    “to validate their JS implementations?”

    Fuzzers don’t really validate. They find bugs but they can’t tell you “there are no more bugs”.

  8. Ashish Yadav Says:

    I tried accessing “jsfunfuzz.js” from “https://bugzilla.mozilla.org/show_bug.cgi?id=jsfunfuzz”. However I got the error
    “Sorry, you are not authorized to access this attachment.”.

    When are you planning to make it available to every one?

  9. Jesse Ruderman Says:

    Grab the last attachment with the name “jsparsefuzz.js”. That’s the most recent public version. I’ve shared the most recent version in the bug with other browser vendors and I’m planning to make it public in a few weeks.

  10. John Resig - JavaScript Fuzz Testing Says:

    […] Ruderman just introduced a fantastic new utility for probing JavaScript implementations called jsfunfuzz. Specifically, this […]

  11. anon Says:

    This is all very well and good for JavaScript, but what about other relatively new bits of the engine like SVG? I am not a security expert but it seems obvious to me given the success here that any point at which the browser accepts external input should be fuzz tested.

    Also can you go into more detail about how “it knows the rules” and “it breaks the rules”? It seems to me that it would be easy to miss possible vulnerabilities by having incorrectness in the former or incompleteness in the latter.

    As fuzzy testers are undoubtably now an important part of the black hat arsenal too, I think every effort should be made to make the ones used by white hats as comprehensive as possible, and to come up with guidelines for best practice.

  12. Jesse Ruderman Says:

    This is all very well and good for JavaScript, but what about other relatively new bits of the engine like SVG?

    Why just the new bits? ;)

    I am not a security expert but it seems obvious to me given the success here that any point at which the browser accepts external input should be fuzz tested.

    I think it’s best to focus on external input whose handling is (1) complicated and (2) done in an unsafe manner (e.g. C++ with raw pointers flying around). JavaScript and SVG certainly meet those criteria.

    Also can you go into more detail about how “it knows the rules” and “it breaks the rules”? It seems to me that it would be easy to miss possible vulnerabilities by having incorrectness in the former or incompleteness in the latter.

    Sure. It “knows” how to use most of the JavaScript syntax: loops, properties, operators. It also “knows” that certain things are functions, and certain things want functions (e.g. setters, the argument to Array.map). It “breaks” the rules by substituting one syntax element for another (see the “totallyRandom” bits) and by screwing up the syntax slightly (see the function called “cat”).

    You’re right: incorrectness or incompleteness in the former, and incompleteness in the latter, can lead to it missing bugs (including bugs that can be found by small modifications to the fuzzer — fuzzing is never about guaranteeing that you don’t have any bugs). I’m not sure what to do about that, but having More Eyes on the fuzzer probably helps :)

    As fuzzy testers are undoubtably now an important part of the black hat arsenal too, I think every effort should be made to make the ones used by white hats as comprehensive as possible

    I’d prefer smart static analysis, but if fuzzing is the best we can do, sure :)

    and to come up with guidelines for best practice.

    This could be tricky, since there are many fuzzing strategies and I’m not sure there’s one that’s “best”. “Mutation fuzzers” that only make small changes to existing files are easy to make; “generating fuzzers” have to know the syntax and tend to find more bugs; “genetic fuzzers” are clever and have interesting behaviors.

    I usually write “generating fuzzers”. For those, my guidelines might be the ones above under “why it has been able to find so many bugs”, plus one additional guideline: make sure you can make a reduced testcase easily when your fuzzer finds a bug.

  13. Testear bugs javascript en navegadores, resultados en Opera y Firefox « Dígito Binario - Tecnología Informática Says:

    […] por pilsen on 9/08/07 Fue una persona que desarrollo esta herramienta para testear el motor javascript del Firefox y así, con esta […]

  14. Ajaxian » Getting Fuzzy with jsfunfuzz Says:

    […] Ruderman has released jsfunfuzz, a fuzz tester that hunts for edge cases that can break things. It tests the JavaScript […]

  15. Adam Blinkinsop Says:

    It sounds like more of a Model-Based Tester than a fuzz tester. (http://en.wikipedia.org/wiki/Model-based_testing)

  16. Are you confident in your test coverage? Says:

    […] the last few years which tries to address the “state explosion” of real-world programs (applied successfully to the Firefox Javascript engine to find 280 bugs). This, of course, places even more burden on the […]