Comments on: Introducing Lithium, a testcase reduction tool

By: James Napolitano

James Napolitano — Fri, 28 Sep 2007 12:11:53 +0000

OK. I've done some clean up, some testing, and fixed a bug, and the result is here: http://www.geocities.com/james_napolitano/TestcaseBookmarklets.html. This was my javascript-learning exercise from 3 years ago. I used many of your bookmarklets as a model. Let me know what you think.

By: James Napolitano

James Napolitano — Sun, 23 Sep 2007 01:36:40 +0000

Let me check if they still work first…

By: Jesse Ruderman

Jesse Ruderman — Sat, 22 Sep 2007 08:46:28 +0000

I assume though that this only works for bugs where you can programmatically determine if the bug is still present (i.e. crashes, assertions, warnings), as opposed to visual bugs like “this widget is 1px too wide”.

Pretty much. Note that if you can detect the bug from JavaScript, you can have the JavaScript output “FAIL” and have Lithium reduce based on that. For example, you can detect incremental layout bugs from JavaScript if they cause an element’s height to change when it shouldn’t change. I used this trick to reduce bug 368621 with Lithium.

A while ago, I came out with some javascript bookmarklets that were able to help quickly make a reduced testcase by stripping out parts of a webpage quickly that were unrelated to the bug in question. Would these be of any use now, for bugs where you couldn’t programmatically determine if the bug were still present?

Yes, they would probably still be useful. I bet they work better than using Lithium with an “interestingness test” that waits for you to press ‘y’ or ‘n’ each time :) Where can I find them?

By: James Napolitano

James Napolitano — Sat, 22 Sep 2007 06:57:37 +0000

Sounds awesome. I assume though that this only works for bugs where you can programmatically determine if the bug is still present (i.e. crashes, assertions, warnings), as opposed to visual bugs like “this widget is 1px too wide”. Maybe you could hook it up to a fuzzer so that your PC automatically generates crashing testcases and reduces them in an endless cycle while its idle. It could even submit a bug report to bugzilla via email, complete with reduced testcase and breakpad crash id!

I know you came out with jsfunfuzz to fuzz test javascript, but does Mozilla have fuzzers for html, css, xul, etc? I’ve seen and used iexploder, but it doesn’t look like it was mature or actively worked on. It certainly could be expanded.

A while ago, I came out with some javascript bookmarklets that were able to help quickly make a reduced testcase by stripping out parts of a webpage quickly that were unrelated to the bug in question. Would these be of any use now, for bugs where you couldn’t programmatically determine if the bug were still present?

By: Fredrik

Fredrik — Sat, 15 Sep 2007 09:37:27 +0000

Check out the optparse module for parsing options (successor to getopt). It has a lot of flexibility and you could either use an append action for program/args options (meaning each occurrence is appended to a list) or define callbacks to set up your own objects, if necessary.

With the append action these arguments:

./lithium.py -p ../fx2/firefox -a “-P fx2″ – p ../trunk/firefox -a “-P fxtrunk”

(as much of a PITA they are to write) would result in a dict like this

{‘p’ : ["../fx2/firefox", "../trunk/firefox"], ‘a’ : ["-P fx2", "-P fxtrunk"]}

Of course, using a YAML file or whatever for config, and having each test check for a .cfg file with the same name as the test file is a neat solution too. Avoiding typing is a good thing.

Having each test importing Lithium is probably easier (may require refactoring a bit, wrapping things up in a Lithium object). Although if you want some sort of automation later on, the reverse may be beneficial. Or adapting things in some way to fit into PyUnit or twisted.trial or whatever.