Indistinguishable from Jesse

Most of Mozilla's leak-detection tools work on the premise that when the application exits, no objects should remain. This strategy finds many types of leak bugs: I've used tools such as trace-refcnt to find over a hundred. But it misses bugs where an object lives longer than it should.

The worst of these are bugs where an object lives until shutdown, but is destroyed during shutdown. These leaks affect users as much as any other leak, but most of our tools don't detect them.

After reading about an SVG leak-until-shutdown bug that the traditional tools missed, I wondered if I could find more bugs of that type.

A new detector

I started with the premise that if I close all my browser windows (but open a new one so Firefox doesn't exit), the number of objects held alive should not depend on what I did in the other windows. I retrofitted my DOM fuzzer with a special exit sequence:

1. Open a new, empty window
2. Close all other windows
3. Until memory use stabilizes
   • Flush all in-memory caches
   • Collect all garbage (XPCOM CC + JS GC)
   • Briefly return to the event loop
4. Count the remaining objects (should be constant)
5. Continue with the normal shutdown sequence
6. Count the remaining objects (should be 0)

If the first count of remaining objects depends on what I did earlier in the session, and the second count is 0, I've probably found a leak-until-shutdown bug.

To reduce noise, I had to disable the XUL cache and restrict the counting to GlobalWindow and nsDocument objects. On Linux, I normally count 4 nsGlobalWindows and 4 nsDocuments.

So far, I've found two bugs where additional objects remain:

• An easy-to-hit leak with the <video> element
• An obscure leak with the InstallTrigger API

I'm glad we found the <video> leak before shipping Firefox 4!

Note that this tool can't find all types of leaks. It won't catch leak-until-page-close bugs or other leaks with relatively short lifetimes. It can't tell you if a cache is misbehaving or if cycle collection isn't being run often enough.

Next steps

Depending on how promising we think this approach is, we could:

• Use it in more types of testing
  • Package it into a more user-friendly extension for Firefox debug builds
  • Make it a regular part of fuzzing
  • Use it for regression tests
• Add something to Gecko that's similar but less kludgy
• Expand the classes it will complain about
• Debug the flakiness with smaller objects
• Make the XUL cache respond to memory-pressure notifications

It's also possible that DEBUG_CC, and in particular its "expected to be garbage" feature, will prove itself able to find a superset of leaks that my tool can find.

User experience designers Alex Faaborg and Alexander Limi are looking to broaden the scope of efforts to make Firefox faster. Until recently, most of the effort has involved reducing the computation time needed to launch Firefox or render a web page. Faaborg and Limi argue that we should also look for ways to make computation time matter less.

The wiki page Perceived Performance contains a long list of ideas. Don't be fooled by the "perceived" part of the name: although a few of the ideas would merely make Firefox users happier, most of the ideas are aimed at reducing the amount of time users spend waiting for Firefox to do various things.

Common themes include:

Improve handling and feel during input. For example, letting scrolling "accelerate" makes rapid scrolling easier without harming the ability to make fine adjustments.

Give instant feedback in response to input, even if an operation has to continue in the background. In addition to reducing human waiting time, these fixes make the experience feel more like direct manipulation and less like telling Firefox what to do. (See also: "snappiness" bugs.)

Allow users to interact with partially loaded pages sooner. This is especially important with slow connections, which are becoming common again as computing goes mobile. Boris Zbarsky's interruptible reflow work is likely to help, and I added some ideas about making better use of the cache.

Reduce the number of clicks and keypresses needed for the most common interactions.

Give users new tools that let them avoid waiting. The introduction of tabbed browsing made it matter less when pages loaded slowly, by letting users load links in background tabs while continuing to read, but also introduced new problems.

Diminish frustration when slowness can't be helped by making changes to activity indicators, progress bars, and other messages. Give users the impression that Firefox is working hard, and help users make better choices about whether to wait.

What kinds of slowness do you encounter while using Firefox? Where should we focus performance efforts, whether by reducing computation time or through more clever means? Can you think of new ways to apply the themes above, or any other ways to make Firefox faster where it matters?

Lately, I've been running debug builds of Firefox with trace-refcnt summaries enabled and reporting any leaks I can reproduce. Frequently, someone else quickly attaches a patch, but in the case of bug 401393, nobody else could reproduce the leak I was seeing. For me, simply loading a page with Flash and scrolling down could leak several objects.

Boris helped me use more detailed trace-refcnt logging options to figure out what caused the leak, and then Jonas assigned the bug to me:

Jesse, here's your first blocker :)

Let me know if you need help and I'll do my best.

I was able to fix the leak, although I still don't understand much of the code for the class I modified. Boris and I also figured out why not everyone could reproduce the leak: it only happened because I was using Greasemonkey, which implements nsIContentPolicy using JavaScript. (Adblock Plus does the same, and it is apparently the most popular Firefox extension, so I'm glad this leak is fixed!)

Knowing how to debug leaks with trace-refcount logging turned out to be useful: I was able to fix a few editor leaks and debug another.

I turned on trace-refcnt a few days ago in order to find leak bugs. I found three yesterday and three today.

Four of the six bugs already have patches :)

Many Mozilla community members, including both volunteers and Mozilla Corporation employees, have been helping to reduce Firefox's memory usage and fix memory leak bugs lately. Hopefully, the result of this effort will be that Firefox 3 uses less memory than Firefox 2 did, especially after it has been used for several hours.

Memory usage

Federico Mena-Quintero submitted a patch to make Firefox discard decompressed image data after five seconds (bug 296818). ImageLib module owner Stuart Parmenter experimented with a competing idea in bug 386377, but now he plans to help with Federico's patch. The patch will make image data join text runs in using time-based caching rather than traditional space-bounded caching.

Aaron suggested having an "about:memory" page showing a breakdown of Firefox's memory use (bug 392351). When I pointed out the bug to Brendan Eich, he excitedly assigned the bug to himself.

Eli Friedman discovered that nsFloatCache was no longer necessary and eliminated most of it (bug 381385).

Memory leaks

David Baron checked in a patch for the last bug that contributed to RLk on Linux, bringing the number of XPCOM objects leaked during this test to zero. Since this test runs on Tinderbox, it's likely that regressions will be noticed quickly, even if they don't turn Tinderbox orange.

Robert Sayre created a script to load random pages and see whether they cause leaks. The random URLs come from the Yahoo directory (biased toward older, top-level pages), del.icio.us (biased toward newer, geeky pages), and AltaVista (biased toward pornography). The script detects leaks using trace-refcnt, the same test used by RL; future versions might use trace-malloc in order to catch additional leaks. Robert has caught at least 6 distinct leak bugs using this script, 3 of which have already been fixed. See LeakingPages and bug 394517 for details.

David Baron created a series of patches to the cycle collector to aid in debugging leaks. With this code, DEBUG_CC builds of Firefox can notice when an object "expected to be garbage" is not collected and then explain in detail why it was not collected.

Steve England tested the top 500 web sites, finding two leaks. Later, he tested the top 20 Firefox extensions and found leaks in several of them.

David Baron recorded seven leak debugging screencasts, which you can watch to see how David Baron debugs real leaks.

Kris Zyp found a leak in the JavaScript Engine when using the watch method (bug 394709). Igor Bukanov responded quickly with not only a patch for the bug but also a leak detection patch to enable regression testing of JavaScript Engine leaks. I asked him to modify his patch so I could use jsfunfuzz to test for JavaScript Engine leaks, and he did. (This led me to find several bugs in evalcx, but no additional leaks.)

David Baron got the stack walking code and the stack fixer working on Mac, making it possible to use trace-malloc and the refcount balancer on Mac (bug 336517, bug 392118).

How to help

You don't have to be a C++ programmer to help find leaks in Firefox.

If you're a Firefox user, an easy way to help is to browse with a trunk nightly build wrapped in a script that calls leak-gauge.pl when Firefox exits. If it reports that documents or windows leaked, try to figure out how to reproduce the leak and then file a bug report.

If you're an advanced user, you can do something similar with with trace-refcnt, which detects leaks of all reference-counted objects, not just windows and documents. Build Firefox with the .mozconfig option "--enable-logrefcnt" (or build debug) and run your build with XPCOM_MEM_LEAK_LOG=2. When Firefox exits, it will print a detailed but understandable summary of what types of objects leaked.

If you're are a C++ programmer and want to help diagnose or fix bugs, check out Performance:Leak_Tools along with David Baron's screencasts, and hang out in #developers on irc.mozilla.org.

Update 2007-11-06: see also my more recent post on this topic, Firefox memory usage and memory leak news, from September 2007.

One of the more common complaints about Firefox 1.5 was that it leaked a lot of memory. I don't know how often these leaks were severe enough to cause noticeable slowdowns or thrashing, but enough Firefox users seem to care about memory leaks that I thought I'd post something about what we've done and what we're doing.

Two memory leak bugs were fixed in Firefox 1.5.0.1, released earlier this week:

• 316775 - Leak when selecting.
• 317478 - Leaks due to global scope polluter being removed from not enough (?) prototype chains.

In addition, the following memory leaks have been fixed on the trunk since the branch point for Firefox 1.5:

• 241518 - Calling addEventListener with a closure holding a content node leaks the document. (Many Greasemonkey scripts use addEventListener, and they each live within implied closures. Otherwise, this leak pattern isn't too common, mostly because Internet Explorer 6 leaks under the same circumstances.)
• 321283 - Using Find causes documents to leak.
• 315708 - Should release found link and current window object from nsTypeAheadFind.cpp.
• 315951 - Unknown content type dialog leaks domwindow.
• 319123 - Leak with contentsink-parser-dtd cycle involving document.write().
• 323441 - Memory leak if a site sets location and then document.writes (e.g. when visiting www.economist.com).
• 323377 - Lots of leaks in nsInternetSearchService.
• 323532 - Leak when using history autocomplete.
• 315427 - Bad nullcheck in nsXULPrototypeElement::ReleaseSubtree, causing the Options window to leak if the remembered panel is the cache panel.
• 323534 - CreateTreeWalker can cause leaks due to cycles created by closures.

The trunk seems close to not leaking huge object graphs during my typical browsing sessions (YMMV). The only big leak I still see all the time is a leak at Gmail. The leak might be related to the "unused XMLHttpRequest" leak that David Baron is working on, or it might not -- when I asked whether Gmail is considering working around the "unused XMLHttpRequest" leak bug, a Gmail coder said he didn't think Gmail created any XMLHttpRequest objects it didn't use.

David Baron's leak-gauge has been allowing more people to test for large memory leaks with their own browsing habits, extensions, and favorite sites. At least 19 people filed 30 memory leak bug reports found using leak-gauge, accounting for more than half of the memory leak bug reports (and most of the memory leak bug reporters) since its introduction. The bug reports look pretty useful, and are definitely more useful than general complaints about high memory usage after using Firefox for a few hours.

If you're still suffering a lot from memory leaks in Firefox 1.5.0.1 and/or want to help find memory leak bugs, head over to The Burning Edge and try out a trunk build. If you prefer to stick with stable versions of Firefox, don't worry -- many of the fixes that are currently only on the trunk will be included in the next scheduled update (currently planned to be called Firefox 1.5.0.2 and released in March), most of them will be included in Firefox 2, and all of them will be included in Firefox 3.

Leaks in extensions

Recognizing that memory leaks aren't always due to bugs in Firefox itself, Steve England and others have tested popular Firefox extensions for memory leaks. They found that Session saver, NoScript, IE Tab, and the combination of FlashGot and Filterset.G Updater cause leaks.

Giorgio Maone, the author of NoScript and FlashGot, has already fixed the biggest leak in NoScript thanks to Steve's bug report. He is now working on threading issues in FlashGot that prevent debugging and might be responsible for the leaks. Authors of the other extensions haven't been as fast to fix their leaks.

I also fixed several minor memory leaks in my own extensions recently, some found through code inspection and some found using leak-gauge. Search Keys 0.7.2 and below kept parts of search result pages alive for the lifetime of the window due to an unintentionally global variable (a missing "var"). Thumbs 0.6.1 and below kept the DOMWindow for the page Thumbs was last used on alive until shutdown.

Other causes of high memory usage

Memory leaks aren't the only thing that can cause Firefox to (appear to) use a lot of memory. See MozillaZine Fourms: Memory usage FQA and MozillaZine KB: Reducing memory usage.

David Baron recently wrote a tool that testers can use to help reduce memory leaks in Firefox. With this tool, you can find out what leaks you encounter during your normal browsing patterns and report useful bugs when you encounter leaks.

It detects many large leaks even though it only tracks three types of objects (documents, docshells, and domwindows) because many large leaks entrain at least one of those objects. It tries to report the URLs associated with the leaks it detects, allowing you to go back and figure out the steps needed to reproduce the bug.

I have found three leaks so far with this tool. David Baron has already fixed one of them on the trunk.

To use leak-gauge.pl, you need a version of Firefox with the logging code, such as a recent trunk nightly build. If you're using Windows, you'll also need to download Perl. leak-gauge.pl contains instructions for making Firefox log the required information and for running leak-gauge.pl to analyze the log.

Update 2006-01-14: There's also a JavaScript version, which might be easier to use if you use Windows and don't have Perl.

Further reading:

• David Baron: How to file useful leak bugs
• David Baron: How to debug leaks
• David Baron: Using XPCOM in JavaScript without leaking
• Bug 320192, where leak-gauge.pl was introduced and where the associated patches lived.
• Bug 323075, which lists David Baron's ideas for an extension that would monitor JavaScript memory use.