Archive for the 'Mozilla' Category

Secure and compatible

Thursday, August 25th, 2011

Previously, I discussed some of the ways Firefox's new rapid release process improves its security. But improving Firefox's security only helps users who actually update, and some people have expressed concern that rapid releases could make it more difficult for users to keep up.

There is some consensus on how to make the update process smoother and how to reduce the number of regressions that reach users. But incompatibilities pose a tougher problem.

We don't want users to have to choose between insecure and incompatible.

There are three ways to avoid facing Firefox users with this dilemma:

  • Prevent incompatibilities from arising in the first place.
  • Hasten the discovery of incompatibilities, and fix them quickly.
  • Overlap releases by delaying the end-of-life for the old version.

Overlapping releases

For many years, Mozilla tried to prevent this dilemma by providing overlap between releases. For example, Firefox 2 was supported for six months after Firefox 3.

We observed two security-related patterns that made us reluctant to continue providing overlapping releases:

First, some fixes were never backported, so users on the "old but supported" version were not as safe as they believed. Sometimes backports didn't happen because security patches required additional backports of architectural changes. Sometimes we were scared to backport large patches because we did not have large testing audiences for old branches. Some backports didn't happen because they affected web or add-on compatibility. And sometimes backports didn't happen simply because developers aren't focused on 2-year-old code. Providing old versions gave users a false sense of security.

Second, a feedback cycle developed between users lagging behind and add-ons lagging behind. Many stayed on the old version until the end-of-life date, and then encountered surprises when they were finally forced to update. Providing old versions did not actually shield users from urgent update dilemmas.

I feel we can only seriously consider returning to overlapping releases if we can first overcome these two problems.

Improving add-on compatibility

Everything we do to improve add-on compatibility helps to break the lag cycle.

The most ambitious compatibility project is Jetpack, which aims to create a more stable API for simple add-ons. Jetpack also has a permission model that promises to reduce the load on add-on reviewers, especially around release time.

Add-ons hosted on AMO now have their maxVersion bumped automatically based on source code scans. Authors of non-AMO-hosted add-ons can use the validator by itself or run the validator locally.

This month, Fligtar announced a plan of assuming compatibility. Under this proposal, only add-ons with obvious, fatal incompatibilities (such as binary XPCOM components compiled against the wrong version) will be required to bump maxVersion.

With assumed compatibility, we will be able to make more interesting use of crowdsourced compatibility data from early adopters. Meanwhile, the successful Firefox Feedback program may expand to cover add-ons.

Breaking the add-on lag cycle

Even with improvements to add-on compatibility, some add-ons will be updated late or abandoned. In these cases, we should seek to minimize the impact on users.

In Aurora 8, opt-in for third-party add-ons allows users to shed unwanted add-ons that had been holding them back.

When users have known-incompatible add-ons, Firefox should probabilistically delay updates for a few days. Many add-ons that are incompatible on the day of the release quickly become compatible. Users of those add-ons don't need to suffer through confusing UI.

But after a week, when it is likely that the add-on has been abandoned, Firefox should disable the add-on in order to update itself. It would be dishonest to ask users to choose between security and functionality once we know the latter is temporary.

Once we've solved the largest compatibility problems, we can have a more reasonable discussion about the benefits and opportunity costs of maintaining overlapping releases.

Rapid releases and security

Thursday, August 25th, 2011

Several people have asked me whether Mozilla's move to rapid releases has helped or hurt Firefox's security.

I think the new cadence has helped security overall, but it is interesting to look at both the ways it has helped and the ways it has hurt.

Security of new features

With release trains every 6 weeks, developers feel less pressure to rush half-baked features in just before each freeze.

Combined with Curtis's leadership, the rapid release cycle has made it possible for security reviews to happen earlier and more consistently. When we need in-depth reviews or meetings, we aren't overwhelmed by needing twenty in one month.

The rapid release process also necessitated new types of coordination, such as the use of team roadmaps and feature pages. The security team is able to take advantage of the new planning process to track which features might need review, even if the developers don't come to the security team and ask.

Security reviews are also more effective now. When we feel a new feature should be held back until a security concern is fixed, we create less controversy when the delay is only 6 weeks.

Security improvements and backports

Many security improvements require architectural changes that are difficult to backport to old versions. For example:

  • Firefox 3.6 added frame poisoning, making it impossible to exploit many crashes in layout code. Before frame poisoning, the layout module was one of the largest sources of security holes.
  • Firefox 4 fixed the long-standing :visited privacy hole. Mozilla was the first to create a patch, but due to Firefox's long development cycle, several other browsers shipped fixes first.
  • Firefox 6 introduced WeakMap, making it possible for extensions to store document-specific data without leaking memory and without allowing sites to see or corrupt the information. Extension authors might not be comfortable using WeakMap until most users are on Firefox 6 or higher.

Contrary to the hopes of some Linux distros and IT departments, it is no longer possible to backport "security fixes only" and have a browser that is safe, stable, and compatible.

We're constantly adding security features and mitigations for memory safety bugs. We're constantly reducing Firefox's attack surface by rewriting components from C++ into JavaScript (and soon Rust).

Disclosure windows

One area where rapid releases may hurt is the secrecy of security patches.

We keep security bug reports private until we've shipped a fix, but we check security patches into a public repository. Checking in patches allows us to test the fixes well, but opens the possibility of an attacker reverse-engineering the bug.

We check security patches into a public repository for two reasons:

  • We cannot rely entirely on in-house testing. Because of the variety of websites and extensions, our experience is that some regressions introduced by security patches are only found by volunteer testers.
  • We cannot ship public binaries, even just for testing, based on private security patches. This would violate both the spirit and letter of some open-source licenses. It also wouldn't be effective for secrecy: attackers have been using binary patch analysis to attack unpatched Microsoft software. (And that's without access to the to the old source!)

But we can shorten and combine the windows between patch disclosure and release. Instead of landing security patches on mozilla-central and letting them reach users in the normal 12-18 weeks, we can:

  • Land security fixes to all active branches at once. This may be safer now that the oldest active branch is at most 18 weeks old compared to mozilla-central. (In contrast, the Firefox 3.6.x series is based on a branch cut 2009-08-13, making it 24 months old.) But even 18 weeks is enough to create some regression risk, and it is not clear which audiences would test fixes on each branch.
  • Accelerate the channel flow for security bugs. For example, hold security fixes in a private repository until 3 weeks before each release. Then, give them a week on nightly, a week on aurora, and a week on beta. The same channel audiences that make rapid release possible would test security fixes, just for a shorter period of time. Something like this already happens on an ad-hoc basis in many bugs; formalizing the process could make it smoother.

There are significant security wins with rapid releases, and a few new challenges. Overall, it's a much better place to be than where we were a year ago.

(I posted followups about add-on compatibility and intranet compatibility.)

How Mozilla could improve search engine competition

Friday, August 12th, 2011

Browsers have been improving rapidly now that there are multiple strong players with significant market share. Wouldn't it be great if the search engine market were more like that?

Mozilla is in a unique position to be able to improve competition among search engines. We're not a search engine company and we stand for meaningful user choice.

If Mozilla were to decide that improving search competition was important to us, here are some things we could do:

Measurements and surveys

We could run Test Pilot studies to learn about search engine choice, speed, and result quality. A study could even encourage users to try multiple search engines and report their experiences.

The content and order of Firefox's search bar have always been determined exclusively based on what we believe to be best for Firefox users. We should strive to have the best data possible when we make these decisions.

Ease of experimentation

We could make it easier for users to compare search engines. For example, after I enter a query into the search bar, it would be nice if I could run the same query on another search engine in one or two clicks. The Search Tabs extension is one example of how this could be done.

Ease of migration

We could let users share their search history with a new search provider, so the search engine they've used in the past doesn't have a hidden personalization advantage.

Helping users make informed choices is core to the Mozilla mission. Users should be aware if their current search engine is collecting search history, and be able to make an informed decisions about the tradeoffs. For example, they should be able to take their history to an engine with a better privacy policy.

We could require special promises from search engines about how they will use data shared through this feature. For example, we could require they only use the data for result personalization, and only on the search engine site itself.

There may be ways to use history to improve search quality that involve only partial history sharing, such as client-side relevance tweaks or RePriv. If Firefox supported one of these methods, users could comfortably ask their search engine not to store any search history.

Improving incentives for web advertisers

Wednesday, August 10th, 2011

When users install ad filters out of a desire to avoid unpleasant ads, they usually end up blocking all ads. This does little to incentivize individual sites to clean up their ads, because from the perspective of any web site owner, users decide whether to block ads mostly based on their experiences on other sites.

As more users turn to ad filters, debates about ad filtering are becoming increasingly polarized. A pro-ad faction screams “blocking ads is like stealing from web sites”. An anti-ad faction screams “showing me ads is like kicking me in the groin in the hope that a penny will fly out of my pocket”.

A better way?

What if a future version of Adblock Plus only tried to block bad ads by default? The immediate result would be negligible, because most ad networks today are somewhere between bad and terrible. But some users would feel more comfortable enabling the blocks, and web site owners would have a harder time blaming visitors for missed revenue.


Current Adblock Plus first run page

Ad filter set:
e.g. animations, sounds, interstitials
[?]e.g. plugins, scripts that block parsing
[?]
[?]
Proposed options and defaults

“Block distracting ads” would block ads that animate, ads with bright pink, gigantic ads, <audio> ads, ads that use absolute positioning to cover other content, and plugins.

“Block slow ads” would block ad scripts that do not use async or defer, any ad that uses more than 5 sequential or 10 total requests, any ad content that hasn't finished loading after 500ms, and plugins.

Note that these are all things that can be detected by the client, which already has a filter set that distinguishes ads from non-ads. Upon blocking an ad through one of these heuristics, the entire ad network should be blocked for a period of time, so that Firefox does not waste time downloading things it will not display. The filter set could also specify ad networks known to specialize in distracting ads, or ad networks that are slow in ways the heuristics miss.

Blocking manipulation

The heuristics above would block ads that interfere directly with your web experience, but what about ads that harm you in slightly subtler ways? Maybe activists would be inspired to curate subsets of existing filters, focusing on their causes:

Ad filter set:
[?]e.g. non-evidence-based medicine, “Free*”
[?]e.g. appeals to feelings of inadequacy
[?]e.g. sexual puns, gratuitous cleavage
Ad filter set:
e.g. lingerie, “adult dating”
e.g. action films, appeals to fear
e.g. junk food, tobacco
[?]e.g. toys, nag coaching

These would block almost every ad network today, assuming the curators err on the side of over-blocking when a network carries multiple types of ads. I can only name one ad network that demonstrates the slightest bit of competence at keeping out scams and one ad network that actively gathers feedback from viewers about individual ads.

With improved incentives, more ad networks would try to do the right thing.

I look forward to a future where advertising is a truly low-transaction-cost way to compensate free content providers.

I look forward to innovators and creators once again having a way to connect with people who might genuinely stand to benefit from their work, without having their voices drowned out by screaming scammers.

More skimmable diffs on hgweb

Sunday, June 19th, 2011

I've found it hard to skim diffs on hgweb (example). The lack of visual hierarchy meant I often didn't notice the jumps between files, even when I was looking for them.

I wrote a user stylesheet to fix this. Between files, it adds a gray bar, vertical space, and outdented filenames. Between sections of a file, it adds a small amount of vertical space. (screenshot before, screenshot after)

To install hgweb-visual-hierarchy.css, add it to chrome/userContent.css in your Firefox profile directory. You may need to create the subdirectory and file.

Tracking after-fix tasks

Friday, June 10th, 2011

I often come across a bug report and decide that I want to do something once it's fixed:

  • Once bug X is fixed, tweet joyfully.
  • Once bug X is fixed, update some documentation.
  • Once bug X is fixed, add a feature to a fuzzer.
  • Once bug X is fixed, retest a fuzz testcase that I assumed triggered bug X but might in fact trigger a different bug.
  • Once bug X is fixed, add a regression test for bug W.
  • Once bug X is fixed, see if it also fixed bug Y.

I could CC myself to the bug, but then I'll get lots of email and might forget my reason for being CCed. I could create a dependency, but that sends everyone else confusing bugspam and gives me notifications that are easy to miss.

The after-fix tool

I created a tool called after-fix to track these bug-dependent tasks for me. I have a large after-fix config file with entries like:

# Make use of new GC APIs in DOM fuzzer bug 661469

I'll see my note the next time I run after-fix after bug 661469 is fixed.

Pruning workarounds

I've also been using after-fix to ensure workarounds don't outlive their purpose:

Faster crash analysis with stack-blame

Wednesday, June 1st, 2011

When a new crash appears, we often want to know whether any code on the stack changed recently. Historically, this has required opening each source link in a new tab and waiting for hgweb to generate megabytes of HTML. Even if you only look at the top 10 frames of each stack, this gets boring quickly.

I created stack-blame to make this easier. It shows 5 lines of context around each line of the stack trace, and highlights fresh lines in green. In the stack-blame output for this crash report, it's easy to see that of the cairo functions on the stack, only _cairo_quartz_surface_finish was touched recently.

Fuzzing in the pool

Tuesday, November 23rd, 2010

In mid-2009, John O'Duinn offered to let my DOM fuzzer run on the same pool of machines as Firefox regression tests. I'd have an average of 20 computers running my fuzzer across a range of operating systems, and I wouldn't have to maintain the computers. All I had to do was tweak my script to play nicely with the scheduler, and not destroy the machines.

Playing nicely with the scheduler

Counter-intuitively, to maximize the amount of fuzzing, I had to minimize the duration of each fuzz job. The scheduler tries to avoid delays in the regression test jobs so developers don't go insane watching the tree. A low-priority job will be allowed to start much more often if it only takes 30 minutes.

Being limited to 30 minutes means the fuzz jobs don't have time to compile Firefox. Instead, fuzz jobs have to download Tinderbox builds like the regression test jobs do. I fixed several bugs in mozilla-central to make Tinderbox builds work for fuzzing.

I also modified the testcase reducer to split its work into 30-minute jobs. If the fuzzer finds a bug and the reducer takes longer than 30 minutes, it uploads the partially-reduced testcase, along with the reduction algorithm's state, for a subsequent job to continue reducing. To avoid race conditions between uploading and downloading, I use "ssh mv" synchronization.

Not destroying the test slaves

I wasn't trying to fill up the disks on the test slaves, really!

Early versions of my script filled up /tmp. I had incorrectly assumed that /tmp would be cleared on each reboot. Luckily, Nagios caught this before it caused serious damage.

Due to a security bug in some debug builds of Firefox, the fuzzer created randomly-named files in the home directory. This security bug has been fixed, but I'm afraid RelEng will be finding files named "undefined" and "[Object HTMLBodyElement]" for a while.

By restarting Firefox frequently, fuzzing accelerated the creation of gigantic console.log files on the slaves. We're trying to figure out whether to make debug-Firefox not create these files or make BuildBot delete them.

Results so far

Running in the test pool gets me a variety of operating systems. The fuzzer currently runs on Mac32 (10.5), Mac64 (10.6), Linux32, Linux64, and Win32. This allowed me to find a 64-bit-only bug and a Linux-only bug in October. Previously, I had mostly been testing on Mac.

The extra computational power also makes a difference. I can find regressions more quickly (which developers appreciate) and find harder-to-trigger bugs (which developers don't appreciate quite as much). I also get faster results when I change the fuzzer, such as the two convoluted testcases I got shortly after I added document.write fuzzing.

Unexpectedly, getting quick results from fuzzer changes makes me more inclined to tweak and improve to the fuzzer. I know that the change will still be fresh in my mind when I learn about its effects. This may turn out to be the most important win.

With cross-platform testing and the boost to agility, I suddenly feel a lot closer to being able to share and release the fuzzer.