I've been reading Firefox security bug reports over the years, and I've collected a list of things that can go wrong in security dialogs. New security dialogs should be tested against these attacks, or preferably designed to not be dialogs.

My slides are in the S5 web-based presentation format. You can click the Ø button to view the presentation in "handout mode" and see what I planned to say while each slide was up.

I shared a presentation slot with Mozilla contractor Paul Nickerson, who has a separate slide deck. He wisely saved the best part of his talk for the end: a demo of his font fuzzer causing Windows 7 to blue-screen.

One cost of running so many tests is that a few tests that each fail 1% of the time can quickly add up to 3-5 intermittent failures per checkin. Historically, this has been a major source of pain for Mozilla developers, who are required to identify all oranges before and after checking in.

Ehsan and I have pretty much eliminated the difficulty of starring intermittent failures on Tinderbox. Ehsan's assisted starring feature for TinderboxPushlog was a breakthrough and keeps getting better. The orange almost stars itself now. The public data fairy lives.

I'm only aware of two frequent oranges that are difficult to star, and we have fixes in hand for both.

But we should not forget the need to reduce the number of intermittent failures now that they are easy to ignore. They're still an annoyance, and many of them are real bugs in Firefox.

What makes it hard to diagnose and fix intermittent failures in Firefox's automated tests? Let's fix these remaining unnecessary difficulties.

When a debug build of Firefox hits a non-fatal assertion, an in-process stack walker prints out libraries and offsets. A new Python script post-processes the stack trace, replacing the library+offset with function names and line numbers that it gets from Breakpad symbol files. (Tinderbox strips native symbols from binaries, so the old scripts using atos/addr2line don't work on Tinderbox.)

The new script was added in bug 570287 and now runs on Linux64, Linux32, and Mac32 Tinderboxen. It will work on Mac64 soon. It could work on Windows if someone brave were to dive into nsStackWalk.cpp and improve its baseline output on Windows.

As we add more tests and platforms to Tinderbox, it's increasingly important for developers to be able to identify each test failure quickly and accurately. Good error summaries make assisted starring of random oranges possible, which greatly reduces the pain induced by intermittent failures. Good error summaries also make it possible to track which failures are most frequent, and therefore concentrate on fixing the most important ones.

Error summaries for crashes could be further improved by showing what kind of process crashed and a signature based on the stack trace.

I'd also like to see better error summaries for memory leaks, compiler errors, python errors, and failures in smaller build steps.

If you see other error summaries on Tinderbox that could be improved, please file bugs. It's an easy way to help Mozilla scale across branches, and it's cheaper than cloning philor.

If you want to stay safe from phishing and other forms of online fraud you need at least a basic understanding of a bewildering array of technologies—URLs, paths, domains, subdomains, ports, DNS, SSL as well as fundamental concepts like browsers, web sites and web servers. Misunderstand any of those concepts and you’ll be an easy target for even the most basic phishing attempts. It almost makes me uncomfortable encouraging regular people to use the web because I know they’ll be at massive risk to online fraud.

- Simon Willison