Indistinguishable from Jesse

Fuzzers make things go wrong.
Assertions make sure we find out.

Assertions can improve code quality in many ways, but they truly shine when combined with fuzzing. Fuzzing is normally limited to finding obvious symptoms like crashes, because it's rare to be able to tell correct behavior from incorrect behavior when the input is generated randomly. Assertions expand the scope of fuzzing to include everything they check.

Assertions can even help find crash bugs: some bugs are relatively easy for fuzzers to trigger, but only lead to crashes when additional conditions are met. A well-placed assertion can let us know every time we trigger the bug.

Fuzzing JS and DOM has found about 4000 assertion bugs, including about 300 security bugs.

Asserting safe use of generic data structures

Assertions in widely-used data structures can find bugs in many callers.

• Array indices must be within bounds. This simple precondition assert in nsTArray has caught about 90 bugs.
• Hash tables must not be modified during enumeration. If the modification happened to resize the hash table, it would leave stack pointers dangling. This PLDHashTable assertion has caught over 50 bugs.
• Cached values should not be out of date. When a cache's get method takes a key and a closure for computing values in the case of a cache miss, debug builds can check whether the cached values are still correct. This is effectively a form of differential testing that notices bugs in cache-invalidation logic.

Asserting module invariants

When an entire module must maintain an invariant, a single assertion can catch dozens of bugs.

• Compartment mismatches. When a JS object in one page's compartment references an object in another, it must do so through a wrappers that enforces security policies. Without these assertions, we would have missed over 25 violations of Firefox's script security model.
• Phases of layout. These assertions have strings like "Should be in an update while creating frames" and "reflowing in the middle of frame construction". More phase and nesting assertions are wanted, but sometimes special cases like plugins get in the way.

Making the frame arena safer

Gecko's CSS box objects, called "frames", are created and destroyed manually. They are allocated within an arena to reduce malloc overhead and fragmentation. The arena also made it possible to reduce the risk associated with manual memory management. A combination of assertions (in debug builds) and runtime mitigations (in all builds) mitigates dangling pointer bugs that involve frames.

• When the arena is destroyed, debug builds assert that all objects in the arena were also destroyed. Over 60 bugs have been caught by the assertion. About half of the bugs that trigger the assertion can lead to exploitable crashes, but without a specially crafted testcase, they will not crash at all.
• While the arena is still alive, deleted frames are overwritten with a special poison pattern. If any code uses a pointer from a deleted frame, the browser will segfault safely. This mitigation, called frame poisoning, has prevented dozens of bugs from being exploitable.
• Writing over the poison trips another assertion. This assertion is actually more prone to catch hardware errors than software bugs, so it has been modified to help distinguish between the two.

Requests for Gecko developers

Please add assertions, especially when:

• A bug would be a security hole
• Crashing is not guaranteed
• Many callers must fulfill a precondition
• Complex, extensive code must maintain an invariant

Also consider:

• Ensure assertions in third-party libraries are enabled in debug builds of Firefox.
• Fix bugs requesting new assertions.
• Fix my assertion bugs to allow my fuzzers to find more.

This entry was posted on Monday, February 3rd, 2014 at 1:02 am and is filed under Fuzzing, Mozilla. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.