This is all very well and good for JavaScript, but what about other relatively new bits of the engine like SVG?

Why just the new bits? ;)

I am not a security expert but it seems obvious to me given the success here that any point at which the browser accepts external input should be fuzz tested.

I think it’s best to focus on external input whose handling is (1) complicated and (2) done in an unsafe manner (e.g. C++ with raw pointers flying around). JavaScript and SVG certainly meet those criteria.

Also can you go into more detail about how “it knows the rules” and “it breaks the rules”? It seems to me that it would be easy to miss possible vulnerabilities by having incorrectness in the former or incompleteness in the latter.

Sure. It “knows” how to use most of the JavaScript syntax: loops, properties, operators. It also “knows” that certain things are functions, and certain things want functions (e.g. setters, the argument to Array.map). It “breaks” the rules by substituting one syntax element for another (see the “totallyRandom” bits) and by screwing up the syntax slightly (see the function called “cat”).

You’re right: incorrectness or incompleteness in the former, and incompleteness in the latter, can lead to it missing bugs (including bugs that can be found by small modifications to the fuzzer — fuzzing is never about guaranteeing that you don’t have any bugs). I’m not sure what to do about that, but having More Eyes on the fuzzer probably helps :)

As fuzzy testers are undoubtably now an important part of the black hat arsenal too, I think every effort should be made to make the ones used by white hats as comprehensive as possible

I’d prefer smart static analysis, but if fuzzing is the best we can do, sure :)

and to come up with guidelines for best practice.

This could be tricky, since there are many fuzzing strategies and I’m not sure there’s one that’s “best”. “Mutation fuzzers” that only make small changes to existing files are easy to make; “generating fuzzers” have to know the syntax and tend to find more bugs; “genetic fuzzers” are clever and have interesting behaviors.

I usually write “generating fuzzers”. For those, my guidelines might be the ones above under “why it has been able to find so many bugs”, plus one additional guideline: make sure you can make a reduced testcase easily when your fuzzer finds a bug.

Also can you go into more detail about how “it knows the rules” and “it breaks the rules”? It seems to me that it would be easy to miss possible vulnerabilities by having incorrectness in the former or incompleteness in the latter.

As fuzzy testers are undoubtably now an important part of the black hat arsenal too, I think every effort should be made to make the ones used by white hats as comprehensive as possible, and to come up with guidelines for best practice.