Secure and compatible

Previously, I discussed some of the ways Firefox's new rapid release process improves its security. But improving Firefox's security only helps users who actually update, and some people have expressed concern that rapid releases could make it more difficult for users to keep up.

There is some consensus on how to make the update process smoother and how to reduce the number of regressions that reach users. But incompatibilities pose a tougher problem.

We don't want users to have to choose between insecure and incompatible.

There are three ways to avoid facing Firefox users with this dilemma:

  • Prevent incompatibilities from arising in the first place.
  • Hasten the discovery of incompatibilities, and fix them quickly.
  • Overlap releases by delaying the end-of-life for the old version.

Overlapping releases

For many years, Mozilla tried to prevent this dilemma by providing overlap between releases. For example, Firefox 2 was supported for six months after Firefox 3.

We observed two security-related patterns that made us reluctant to continue providing overlapping releases:

First, some fixes were never backported, so users on the "old but supported" version were not as safe as they believed. Sometimes backports didn't happen because security patches required additional backports of architectural changes. Sometimes we were scared to backport large patches because we did not have large testing audiences for old branches. Some backports didn't happen because they affected web or add-on compatibility. And sometimes backports didn't happen simply because developers aren't focused on 2-year-old code. Providing old versions gave users a false sense of security.

Second, a feedback cycle developed between users lagging behind and add-ons lagging behind. Many stayed on the old version until the end-of-life date, and then encountered surprises when they were finally forced to update. Providing old versions did not actually shield users from urgent update dilemmas.

I feel we can only seriously consider returning to overlapping releases if we can first overcome these two problems.

Improving add-on compatibility

Everything we do to improve add-on compatibility helps to break the lag cycle.

The most ambitious compatibility project is Jetpack, which aims to create a more stable API for simple add-ons. Jetpack also has a permission model that promises to reduce the load on add-on reviewers, especially around release time.

Add-ons hosted on AMO now have their maxVersion bumped automatically based on source code scans. Authors of non-AMO-hosted add-ons can use the validator by itself or run the validator locally.

This month, Fligtar announced a plan of assuming compatibility. Under this proposal, only add-ons with obvious, fatal incompatibilities (such as binary XPCOM components compiled against the wrong version) will be required to bump maxVersion.

With assumed compatibility, we will be able to make more interesting use of crowdsourced compatibility data from early adopters. Meanwhile, the successful Firefox Feedback program may expand to cover add-ons.

Breaking the add-on lag cycle

Even with improvements to add-on compatibility, some add-ons will be updated late or abandoned. In these cases, we should seek to minimize the impact on users.

In Aurora 8, opt-in for third-party add-ons allows users to shed unwanted add-ons that had been holding them back.

When users have known-incompatible add-ons, Firefox should probabilistically delay updates for a few days. Many add-ons that are incompatible on the day of the release quickly become compatible. Users of those add-ons don't need to suffer through confusing UI.

But after a week, when it is likely that the add-on has been abandoned, Firefox should disable the add-on in order to update itself. It would be dishonest to ask users to choose between security and functionality once we know the latter is temporary.

Once we've solved the largest compatibility problems, we can have a more reasonable discussion about the benefits and opportunity costs of maintaining overlapping releases.

13 Responses to “Secure and compatible”

  1. Michael Kaply Says:

    > But after a week, when it is likely that the add-on has been abandoned, Firefox should disable the add-on in order to update itself. It would be dishonest to ask users to choose between security and functionality once we know the latter is temporary.

    No. It can take longer than a week just to for the AMO editors to review something. So an add-on developer might discover something is broke when a new version of Firefox ships, then they have to fix it, then they have to get it reviewed. That can take a while. Remember that most add-on developers do this as a hobby.

    Unless your going to reenable the add-on when there is an update, it is wrong to disable a user’s add-on without their explicit permission.

  2. Zack Says:

    It’s not as much of a priority, but we should also consider the needs of users on Aurora and Beta (especially Beta). Specifically, we should do whatever it takes to make sure that addons popular among Beta users have updated on AMO before Aurora->Beta migrations.

    In particular, I’ve been trying to persuade my webdeveloper friends to use Beta, but they will not upgrade until Firebug is available (one of them compared the absence of Firebug to having a missing arm!) and right now for 7.0beta you need Firebug 1.9alpha, which is not yet on AMO. They would rather postpone the Firefox upgrade, indefinitely, than deal with getting Firebug from some third-party site and losing automatic upgrades for it.

  3. Jesse Ruderman Says:

    Zack – I agree that AMO needs better mechanisms for beta versions of add-ons, and for Firefox-version-specific versions of add-ons.

  4. sup Says:

    It’s not just addons and plugins though.

    IMHO the current release system, by not supporting the old version, forces me to:
    1- stop doing whatever i’m doing;
    2- update;
    3- check/search for new features – configure them;
    4- check for any missing old features, or changes – reconfigure;
    5- check if my addons are OK – if not, endure the pain;
    6- get back to whatever i was doing.

    Mozilla says it’s working on 1, 2 and 5, but they will never fix 3 and 4. These will always demand time from me. And i actually like digging in Firefox, but.. when i feel like it. (!)

    An example for 3: Geolocation. A privacy concerned user will want to look into it.
    And for 4: Firefox menu button. I’m certainly not using it, even though i kind of liked it. It just seems to miss a few items i use from Tools, to name one. I believe there was another problem with it, but i don’t remember – i don’t use it!

    Not to mention that 5 probably won’t be fixed in full.
    Example: should they bump Perspectives (currently not compatible)? Will it actually work properly as intended?
    The user can only guess that’s for sure. “I bumped it and it works just fine”. Without looking at the code, can he really be sure?

    Anyway, i type this on version 6, but i have to say i’m not comfortable with all this.
    I love Firefox and Mozilla, but i began looking for alternatives. I don’t know why, it’s just software after all, but i’m kind of sad.

  5. David Naylor Says:

    This all sounds like a great plan! A shame that users still will have to go through upgrades 4 -> 5 -> 6 -> 7 -> 8 -> 9 without these upcoming improvements.

  6. Jorge Says:

    Users should always be in control. It should be their choice if they want to update or not. It should always be their choice if their add-ons are disable or not. If they want to stay using an old, unsupported version so their add-ons continue to work, that’s their call.

    It’s sad that we have taken the rapid release route without thinking about all these things before. Users and add-on developers are hurting because of this, and this proposed solution only makes it worse by imposing our own decisions on them.

    While I appreciate the importance of security, it shouldn’t be preferred over explicit user actions. We also shouldn’t kid ourselves into believing that there’s no gain to be had in keeping a LTS branch and backporting security fixes to it. I don’t know the costs of doing something like this, but I doubt they’re as high as what these impositions will cost us in the long term.

  7. Jesse Ruderman Says:

    Jorge – I agree that we shouldn’t dismiss the possibility of LTS lightly. It certainly has potential upsides. I know some things about browser security and add-ons, so I’m bringing that knowledge to the discussion.

    But I don’t think “Users should always be in control” is useful framing for making decisions about how to deal with incompatible add-ons.

    If you’re using an insecure browser, you’re not in control of your computer.

    Firefox doesn’t ask you before upgrading if your favorite web sites aren’t compatible. What’s special about add-ons?

  8. Jorge Says:

    I think users should be free to downgrade if their favorite website breaks. I assume you disagree with that too, because you’re looking at the problem from a security standpoint, not from the user’s standpoint, where things are breaking for no explainable reason. And they *are* breaking for no explainable reason, because chances are that the website broke because of something unrelated to security fixes or new features they needed.

    The difference with add-ons is that there is forward knowledge of which will break during upgrade. That Firefox installation can know which add-ons will need to be disabled, and to empower the user to make a decision, it should tell the user before updating. I think it should be up to the user to make that call.

    If we could do the same with websites that break during an update, I would be totally in favor of the idea, but it’s clearly unfeasible, given the size and complexity of the web, as well as having to infer which websites will be needed after the upgrade.

  9. Danny Moules Says:

    “Firefox doesn’t ask you before upgrading if your favorite web sites aren’t compatible. What’s special about add-ons?”

    1) Firefox doesn’t go out its way to ‘disable’ web sites for being incompatible. Remember how developers felt about simple XML validation? That died a very painful death.

    2) Websites don’t (generally) go through a stringent third-party review process that makes pushing out updates hard. A lot of web sites do fail to be compatible – but they don’t have someone stopping them from addressing it promptly.

    3) They’re typically not incompatible – they just don’t have the right version number. Rapidly incrementing version numbers is _painful_ to add-on devs and whilst AMO have tried to mitigate this through automated processes, I’m not convinced it’s been solved.

    4) Can you imagine if you forced web sites to use ‘version numbers’ and only write against particular browser agents? That would be a very interesting day.

    5) Add-ons can delete your hard drive, et al. Incompatibility is rather a bigger issue than just broken CSS and forms that won’t submit.

    I’m really not sure what answer you expected to that question… but there’s just the ones that spring to mind…

  10. Nils Maier Says:

    I wanted to clarify some things about JetPack (AddonSDK):

    Most importantly: AddonSDK add-ons are NOT by-design more compatible as other add-ons. They are more or less subject to the same breaking changes as “regular” add-ons.
    This is because the AddonSDK core libs do not ship with the application (such as Firefox) but are shipped with the individual “compiled” XPIs and only upgrade when the add-on author updates her SDK, rebuilds the XPI and ships it. The AddonSDK is highly dependent on |require(“chrome”)| and other stuff that accesses “internals”. All that code may break at any new application release (at least under the current “no-stable-apis” rapid release regime). Furthermore many SDK add-ons use |require(“chrome”)| and buddies directly. Breaking changes would then require authors not only to rebuild the add-on with a new SDK but also require real code changes.
    I expect lots of SDK add-ons breaking e.g. once e10n get enabled, not only because of add-on code, but also because of SDK core code.

    Then the add-on SDK is pretty much “incomplete” (or: lacks a lot of major features). The top add-ons are all non-SDK add-ons and the vast majority of them are too complex to port to the AddonSDK in it’s current form.
    The add-on SDK is not an alternative for many add-ons.

    Also, there is not only Firefox: Thunderbird and Seamonkey are pretty much on “rapid release”, too. However the AddonSDK does not provide any meaningful functionality for e.g. the mail components.

    As for the “easier to review” part: Unfortunately AddonSDK add-ons are not easier/faster to review usually, at least in the short to mid term.
    While you can trust the SDK part (if amo-validator does not complain) you still have to review the custom code. Since many reviewers are still unfamiliar with SDK add-ons and correct/dangerous uses of SDK core functionality, leading to far fewer reviewers who are actually competent and available to perform SDK add-on reviews. Added to this there are some additional “new” things to consider when it comes to SDK add-ons, such as correct add-on shutdown (SDK add-ons are restartless).

    The AddonSDK might become a viable thing for most add-on needs, but judging from the development pace, this won’t happen within the next few years.

    As for the backports issues: What’s so different now? There are four branches now, release, beta, aurora, central (nightly). You still need to backport security fixes from central to at least aurora and beta and maybe even release. While in theory those branches shouldn’t be that far apart now (comparing to e.g. FX2/3) in practice I noticed that there are occasionally huge differences or even new architectures in individual components. If you landed a security fix on central, that fix would be at most 6 nightly + 6 aurora + 6 beta = 18 weeks away from shipping, so just waiting still isn’t an option.

    I’m favoring a new release process: Only ship breaking changes every 3 or better 4 release. The releases in between may ship actual changes, as long as those changes do not introduce actually necessary backwards compatibility issues.
    This would allow add-on authors to keep up better, and spare the users from a upsetting “not compatible” warnings and broken add-ons.
    While this requires some more work on mozilla’s end, mozila is far better suited and has far more man power than all those small-to-single-person teams around the world developing add-ons.

    If you look at the the past breaking changes (at least those the amo-validator checks for) most of them did not have a technical necessity, or even any apparent merit, that would have warranted or required shipping those changes with the next release, i.e. without a proper grace period.
    My favorite example: breaking nsIJSON, which I thankfully got reversed in time in https://bugzilla.mozilla.org/show_bug.cgi?id=672063

    The problem is, from where I’m standing, not the rapid release process per se, but mozilla not evaluating changes for add-on impact before shipping.

  11. Jake Says:

    I think one of the important things that users don’t seem to know is that Mozilla doesn’t make the add-ons that break with each release, and they really have no control over the code of these add-ons (especially for the add-ons that seem to break often, like the Norton toolbar).

    I saw a comment somewhere that read something like “My Norton toolbar broke with Firefox 6. I am really disappointed that you guys let this happen, and I expect it to be fixed in Firefox 7 when it comes out.” (He didn’t seem to realize that if it didn’t work with Firefox 6, it won’t work with Firefox 7, and Mozilla/Firefox wasn’t the culprit here.)

  12. Jesse Ruderman Says:

    There is now an “addon-compat” keyword in Bugzilla for changes that will affect add-on compatibility. Jorge will use it to compile lists at Aurora merge time.

    http://groups.google.com/group/mozilla.dev.planning/browse_thread/thread/dafd0bc59b587e27

  13. Jesse Ruderman Says:

    Blake Winton has a counter-proposal at http://weblog.latte.ca/blake/employment/mozilla/thunderbird/updates . I still prefer mine, but I do like the idea of https://bugzilla.mozilla.org/show_bug.cgi?id=548766 that was mentioned in his comments.