Comments on: Continuous integration at Mozilla

By: links for 2009-02-22 | The Build Doctor

links for 2009-02-22 | The Build Doctor — Mon, 23 Feb 2009 00:33:52 +0000

[...] Jesse Ruderman » Continuous integration at Mozilla CI at Mozilla – fascinating. [...]

By: Chad

Chad — Fri, 20 Feb 2009 10:40:20 +0000

In fact we are! chad@imvu.com http://www.imvu.com/jobs/

By: Raoul Duke

Raoul Duke — Fri, 20 Feb 2009 01:23:55 +0000

@Chad: right on! you hiring?

everybody owes it to themselves to read up on the whole Lean Kanban For Software thing, perhaps.

By: Chad Austin

Chad Austin — Thu, 19 Feb 2009 23:53:23 +0000

BTW, you may need hundreds of build machines to get a 10-minute turnaround on testing your entire suite, but that’s doable too. Three years ago, we had two machines and hardly any tests. Now we have 40-50 and piles of tests.

By: Chad Austin

Chad Austin — Thu, 19 Feb 2009 23:51:35 +0000

Hi Jesse,

Nice post. I’ll elaborate on a few key differences between development at IMVU and development at Mozilla.

When the build goes red, commits are automatically blocked. Every engineer who is blocked helps contribute to the fix. This provides a natural feedback loop that causes the builds to get better over time. Eventually, we noticed we were wasting a lot of time on the same types of failures, so we required that each person who broke the build send out a five whys ( http://startuplessonslearned.blogspot.com/2008/11/five-whys.html )

BTW, we only have two types of builds: red and green. If there’s something you care about, make regressions break the build. Build with -Werror too. Heck, it’s crazy to me that reviewers spend their time nitting on style, when you could automate that and just make style regressions break the build.

In short, keep finding and removing waste in your development processes.

In these comments I hear an undercurrent of “wah wah testing mozilla is hard”, but if you keep pushing on it, it gets easier and easier. IMVU’s got PHP tests, perl tests, Flash tests, client acceptance tests, unit tests, integration with third-party services, integration with our own services, Python tests, C++ tests, assembly tests, selenium tests against internet explorer, selenium tests against firefox, various performance tests (augmented by reporting from the field) and some that I’m probably forgetting. You obviously can’t do all of that overnight, but if you keep improving the situation, it will get better.

One brick at a time.

By: Robert Kaiser

Robert Kaiser — Thu, 19 Feb 2009 22:46:38 +0000

“Then it is always ok to pull or push, because mozilla-central is always green. ” Well, that’s only half the truth, as tryservers aren’t testing all uses of the Mozilla tree. There’s platforms and applications using mozilla-central as a base that might actually have failures Fennec, Thunderbird, SeaMonkey, and whatever else will probably never all be covered by try setups. And then, I fail to see how the try-to-push setup is much different than the current setup other than the tree you pull is always green for Firefox tier-1 platforms. If one tried push fails and another slightly later pushed one doesn, I don’t think it’ll be trivial to automatically sort that out without breaking the flow of patches into the repo, if automatically resolving it is possible at all.