Jeff Walden tried to ask:
How do you find security holes?
This isn't a very coherent answer, and I don't know which parts are the most correct or most useful.
I'm good at ad-hoc testing, where I try to find and report as many bugs as I can. From this perspective, a security hole is just a bug that can be abused by an attacker in a way that hurts the user.
I frequently ask myself "What might the author of this code not have considered?". Sometimes I ask myself "How can I prove that this code is correct and secure?" instead.
I spent a lot of time triaging and testcasing Mozilla bugs, so I know a little about many Gecko and Firefox features. I know of many JavaScript and DOM features that tend to hurt security: getters and setters, eval and the version of setTimeout that takes a string, the with statement, prototype chains, and dynamic typing with implicit coercions. I know of several other Mozilla features that tend to lead to security holes: javascript: and data: protocols, synthetic events, security dialogs. I know of some security checks that are scattered and frequently missing: same-origin checks, CheckLoadURI calls.
I take note of places in Gecko where user data interacts with pages: visited link coloring, file upload controls, pasting from the clipboard. These exceptions often interact with other features to create security holes.
Even though I usually have access to source code, I rarely use it. Using Mozilla source is slow for me because lxr is slow, Google Desktop Search refuses to index my Mozilla source tree, and I'm not good at reading Mozilla source. Testing Mozilla like a black box is fast for me because I have tools like the JavaScript Shell.