Thread about trunk alphas
There's an interesting thread on mozilla.dev.planning titled Scheduling a trunk alpha. With all the large changes for Gecko 1.9 and lack of alphas so far, there is concern that there aren't enough people testing trunk builds to find regressions quickly. Issues discussed in the thread include the frequency and naming of trunk alphas, milestones, or "known-good nightlies". If you're interested but don't want to read the entire thread, start with fantasai's summary. I'm curious what Burning Edge readers think of the various proposals.
April 17th, 2006 at 10:30 am
The problem isn’t not enough testers. The problem is not enough devs to fix the regressions that the testers find. Mozilla needs to spend all that money made last year to hire more full time employees rather then spend by giving away grants/donations like Mitchell proposed a few weeks back. I addressed that issue in her blog and she never responded about the need to hire more people which btw was the generall overall response from all the posters to her blog that day.
April 17th, 2006 at 10:35 am
Also forgot to say that most issues are unresolved or no bugs have been filed since most testers understand that a lot more work needs to go into cairo first and a lot of the would be filed bugs would be and have been fixed with big patches that have landed or will land in the future.
April 17th, 2006 at 11:29 am
Kurt, the problem with hiring is that it’s very hard to hire people who would:
1) Be able to make a serious contribution in a reasonable time frame
2) Would fit in well with the community.
Item 2 is very important — hiring developers who don’t “get” open source would be a disaster.
For what it’s worth, the Corporation _is_ working on hiring people as I understand. But anyone who’s hired now who’s not already a contributor won’t really get up to speed fast enough to affect the 1.9 alphas much, I suspect.
As for your problem characterization, I disagree. Not having enough developer is a problem, of course. But we were getting serious regression bugs filed in the 1.8 cycle well over a year after the regression had happened. Having all regression fixing compressed into a very short timeframe only compounds the lack of developer resources; the best way to get regressions fixed is to get them found, filed, and fixed early.
April 17th, 2006 at 11:58 am
The problem is the poor response sofar from those who mainly work on 1.9 (FF3.0).
If no one (1 or 2 exceptions at best) looks at the list of nominated bugs for 1.9a1 (in the past 4 month) , what would be the point in nominating them at all ?
I only nominated the absolute worst bugs, that make FF either constantly crash ormake it completely disfunctional (unless it’s shared with 2.0 development).
We really need a response like on branch… if something breaks… fix it, otherwise it’ll just cover the next series of regressions.
And tell testers what the priorities are… we haven’t got a clue….
April 17th, 2006 at 12:06 pm
oh… and one more suggestion:
If a building machine fails (for either Linux,Mac or Windows) or burns, the tree should be closed.
What happened after X-max , people checking in for 4 days while the win machine had died, costed many manweeks of work to find which bug caused what regression.
April 17th, 2006 at 12:24 pm
I understand what your saying Boris and am very happy to hear that there are some position to be possibly be filled soon but there are I am sure dozens of contirbutors around the community that would be more then willing to work for either the org or co. First position that needs filled is the sys admin or whatever chase was. The update system being broke for what was it 3 months was completely ridiculous and the biggest reasons for no one pinpointing regressions…way to much of a hassle to download dailys, let alone hourlies to pinpoint regressions. The update system should be priority and then what peter suggested and what used to be the “standard” (build machine is down, tree closes. period.)
Also about the nomination flags…flags are supposed to be used to nominate show stoppers (as it is always thrown around near release time), a broken webpage isn’t a showstopper generally unless it is one of the topp 100 sites on alexa (AFAIK that is what is always used as a reference), told not to go crazy filing bugs on cairo since it isn’t near completion. so now we are where peter says, “we haven’t got a clue”
April 17th, 2006 at 12:26 pm
About the hiring thing – if, as I understand it, mozilla is taking over a lot of the maintenance responsibilities for cairo backends and such, why not look at hiring competent cairo developers.. it will obviously take some time to get up to speed on layout issues and such, but it seems like a quick way of broadening the developer pool so that you don’t cannibalize existing moz contributors and yet still get someone who knows somethin’ ’bout somethin’ and “gets” open source.
Just a thought.
-woo
April 17th, 2006 at 3:07 pm
I think there are three main reasons why the trunk isn’t receiving as much testing attention as it might otherwise.
1) Testing is split between two branches.. the 2.0 branch and the main trunk. Since 2.0 is more imminent, then it is (and should be) receiving the more attention.
2) Cairo is on the trunk, and whilst it certainly has a good future ahead of it, it is currently slower than regular builds and also has many quirks/anoyances. There are still lots of cairo bugs to be squashed, and still cairo bugs being filed.
3) All the work done on the trunk that hasn’t landed on the 2.0 branch seems to have been under-the-hood stuff; stuff that regular testers can’t “see”. Coupled with point 2) then there is no apparent gain on trunk (features/speed/rendering/etc) and a slowdown with cairo. Which gives people a reason not to really test it.
Trunk testing will probably increase once something that “end-users” testers can appreciate lands, like the reflow branch (better (faster?) rendering), the ThreadManager stuff (faster) and ‘Fix the use of units in Gecko’ (improved rendering). These kind of landings always excite people (well, maybe just me? ;)) and attract more testers. Had one of these landed sooner (months ago) rather than later, then the Trunk may have received more testing attention from that point onwards.
But what to do now? Boris wants trunk regressions found sooner rather than later, and already 8 months have past without the trunk getting any proper attention, so I think his idea of releasing a trunk alpha is the only real way to go. Released as a Gecko 1.9 Preview or whatever will certainly get some eyes on the trunk. Maybe even release it as a non-cairo build so people who would be testing it aren’t going to be testing cairo-related things, but the other code changes that have been made and that will show up as regressions.
Other than that, when some of the bigger changes (reflow/threadmanager/etc) land I think this will attract tester attention, and when cairo gets to the stage where it’s equal or better than non-cairo builds, then it will grab even more testers. But of course such things are in the future for the moment. And also, such large landings will cause their own regressions, making regressions in the last 8 months perhaps harder to find.
The other thing to consider might be pushing back the planned release date of firefox 3.0. There are finite resources available and it is rightly so that 2.0 gets the most testing, so maybe 3.0’s aggressive schedule simply isn’t realistic.
Once the first 2.0beta is out the door then I think a lot of people will start looking towards 3.0 much more seriously. But until then, i don’t see (m)any reasons why web developers would choose to test 3.0 over 2.0.
April 17th, 2006 at 3:45 pm
They could hire out of the community couldn’t they? They have done this in the past IIRC?! I haven’t been paying much attention lately, been busy with other things, but are there any good coders in the community already making major contributions to M.O code that are up-to-speed on how things work?
April 17th, 2006 at 5:16 pm
Yeah right! Like I am going to install a software that might have a DATA-LOSS regression bug. Minefield was the right name for Fx 3.0 PS: Hire more developers.
April 17th, 2006 at 5:25 pm
Peter, I agree that the nominating and triage situation for 1.9a is silly. That’s why I started that thread.
As for priorities… The whole point of the current setup is that no changes to Gecko happen on branch without getting tested on trunk first. That’s the theory. So testing on trunk is much more useful than testing on branch for Gecko changes. Once we get closer to a beta on branch it may be worth testing some there to make sure nothing weird has happened, of course.
Kurt, I can’t speak for the Corporation on hiring stuff, as you might guess. ;) But it’s not that easy to find contributors who are not already employed or students (who can’t be hired for various reasons) or something. If you have suggestions for who should be hired for core work, please e-mail schrep; he’d love to hear them, I bet. As far as the build engineering situation, you’re right that more resources are needed there. A new guy started on that at the end of March, for what it’s worth; he’s getting up to speed now.
As far as nomination flags…. Flags should indeed be used to nominate showstoppers. As far as I’m concerned, any layout regression is a showstopper until decided otherwise, so they should be nominated. It’s that simple.
Andrew, I don’t know enough about the cairo situation to say much useful.
Steve, I think your assumption that the 1.8 branch should be getting more testing is wrong, as I’ve said above. I agree that the cairo thing on trunk does make testing hard, but cairo is not the only thing that’s landed. Perhaps we should indeed ship a no-cairo alpha1. As for what testers can see, if we’re talking about web designers there’s plenty of stuff that’s changed that they can see (starting with much better painting; just compare Acid 2 on trunk and 1.8 branch). Given that, again, the goal is to have minimal to no changes in layout on branch, it really doesn’t make that much sense for web developers in particular to be testing branch.
As for the planned release date of FF 3.0, I believe it’s pretty fluid right now…. ;) But the 2.0 schedule is slipping a good bit, from what I can tell, and it no longer makes sense for trunk to wait on 2.0 before shipping alphas.
mooky, I think I’ve already addressed your suggestion above.
April 17th, 2006 at 5:50 pm
IMO, having the automatic update system in a highly fragile state for 2 whole months really hit the testers. It wasn’t until new-guy rhelmer stepped in to fix it did I finally receive a partial update to a nightly build. That should have been fixed earlier IMO.
April 17th, 2006 at 8:36 pm
Fantasai’s summary acutally only covers one of the proposals, which would be the one she endorses ;) In the original thread, I’d presented three options:
1. Firefox 2 releases as planned. Simultaneously, stabilize trunk such
that a set of solid builds can be made at some regular interval (let’s
say monthly?) and then call those builds “latest-stable-trunk” or
something and socialize the need for testing and how these builds are
good enough for day to day use.
2. Firefox 2 releases as planned. Quickly build out a set of milestone
dates for Firefox 3 (with several alphas) and start releasing those as
well, perhaps initially synchronized to the Firefox 2 release
schedule. Or maybe it would be better to stagger the releases such
that our testers could use one for a month, then another for a month,
etc.
3. Make Firefox 3 releases the “flagship” alphas which we ask people
to use, since any front-end bugs will be discovered from here, too.
When the front-end stuff reaches beta level, then continue publishing
alphas from trunk, but also release Firefox 2 betas based on the 1.8
branch.
April 17th, 2006 at 11:02 pm
The problem I have with testing, is I still need it to be basically usable. Otherwise I’m just not going to use it for anything. I can’t even import my bookmarks.
My point is, if I could test it while using it, fine; but if I have to sit down, tell myself “I am going to test Firefox now”, and try to find problems, I just don’t have time for that. Maybe this “need more testers” isn’t aimed at people like me but I would like to help; they just need to stop breaking the basics.
April 17th, 2006 at 11:51 pm
Release early release often.
Do the 1.9 backend change now (as in tonight or whatever time of day it is for you) for trunk nightlies. Break it hard if it is going to break.
Yes I use current trunk – building from cvs almost everyday.
If you are using a trunk nightly have a backup browser (I have a known good nightly and Konq) installed and a backup of your profile.
I am sure the devs will hear about it if it is a disaster. Have a backup plan – to undo the major change if it is unworkable.
From a planning perspective any major issues that require refactoring/rewriting the code will take a long time. Better to find out early.
Ok maybe not tonight – give two weeks notice to nightly trunk testers – major changes coming – be prepared for major data loss and failure.
April 18th, 2006 at 12:12 am
I just use seperate profiles. “c:\program files\mozilla firefox\firefox.exe -p default” for default (working) Firefox and “c:\program files\mozilla firefox trunk\firefox.exe -p trunk” for testing Trunk builds. Set the Profile Manager not to start automatically to prevent applications from launching with the wrong profile.
April 18th, 2006 at 5:29 am
As Hemebond points and someone else commented in the main thread, a big selling point to download any nightly without fear would be if it was built and preconfigured like Portable Firefox, so I can download the zip, extract that to any directory and use it without any fear that it would eat my babies.
For webdevelopers would be great to have something that they can use at any moment to check the current state of the trunk, verify that it works at least as good as the current official release and that they won’t have any problem in the future. It should be strongly remarked that if something goes wrong using those builds they should file a bug in bugzilla, not change their code to workaround the problem.
Also a list of all the known regressions (and if possible also a summary or bug list of the improvements) would make easier to evaluate the quality of the builds, (in order to avoid as easy as possible to make people go mad hunting for bugs that are already known)
April 18th, 2006 at 8:17 am
I have to agree that the portable Firefox idea is an EXCELLENT one. It would greatly alleviate the fears of dataloss and give the user the option to easily switch back to their “good” build if need be. I hope the powers that be will give the idea some serious consideration.
April 18th, 2006 at 10:59 am
Boris wrote:
As for priorities… The whole point of the current setup is that no changes to Gecko happen on branch without getting tested on trunk first. That’s the theory. So testing on trunk is much more useful than testing on branch for Gecko changes. Once we get closer to a beta on branch it may be worth testing some there to make sure nothing weird has happened, of course.
Boris, you know that landing on trunk first to iron out regressions is a joke, there’s often far too little time between the trunk and branch landings.
For serious testing we need 3-4 month (minimal, which offcourse isn’t a realistic option) and even than we’ll still find the odd regression.
The current flags, ?1.8.1 and +1.8.1 are also meaningless for trunk testers, they often mean little more than “we’d like to have this on branch but if there are too many regressions we won’t take it”
A keyword like “will_land_on_1.8.1_branch ” (no matter what) would mean a lot more.
It would mean that we should blindly nominate almost any regression from a bug with this keyword.
Not that we shouldn’t nominate any that don’t have the keyword, but they would get have a lesser priority (e.g. like the current cairo bugs, if it doesn’t crash or render the build useless it’s called a NORMAL bug, no need to panic).
April 18th, 2006 at 8:34 pm
Peter, if patches that have regression risk are not baking on trunk long enough to get testing before landing on branch, we have a problem. I haven’t been paying that much attention to how the “module owner approval” setup is working, but if it’s not, please post to m.d.planning (and cc drivers@mozilla.org, with followups and replies to .planning). We can’t afford to screw this up.
In any case, core patches that have known unfixed regressions should NOT be landing on branch; if they are, we have a really serious problem. And yes, any regression from any bug that has a blocking? flag for branch gets a blocking? flag itself. Any regression from any bug seeking approval for the branch should be in the dep list and explicitly mentioned in the bug as a regression.
The “doesn’t crash or render build useless” thing is not great on trunk, but it’s completely unacceptable on branch.
April 29th, 2006 at 9:00 pm
Great website! Bookmarked! I am impressed at your work!