Indistinguishable from Jesse

In the past, I've complained about banks not using https for login pages and software providers not using https for downloads. Both of these practices put large numbers of users at risk of financial harm through man-in-the-middle attacks, including attacks against unsecured wireless networks.

Starting today, I'm practicing what I preach: sections of my site that offer software, such as Firefox extensions and bookmarklets, are now served using https. I'm using the following .htaccess magic in each of those directories to redirect http requests to the correct https URL:

RewriteEngine On
RewriteCond %{HTTPS} !=on
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}

Supporting https will cost me about $65 per year: $17.99/year for a domain validation certificate from GoDaddy and $47.40/year for a unique IP from my web host.

Yesterday, at around 4pm, I noticed that the content on squarefree.com was missing, and the main page was an empty directory listing. I ssh'ed to my web server and noticed that the "squarefree.com" directory had been renamed to "squarefree.com_DISABLED_BY_DREAMHOST". Then I checked my email and saw a message from DreamHost support:

Hello,

I just had to disable your site squarefree.com as it's coming under some load and spawning countless php processes that are crashing the webserver. I wasn't able to figure out exactly what's going on, as leaving it up for more than a minute pretty much toasts the server. Please don't re-enable it until you've figured out what's going on, or disabled any possibly problematic php.

Thanks,

James

I jumped into #dreamhost on irc.freenode.net and started looking through my web server logs for suspicious requests. I was expecting to find that my blog had been DDoSed, perhaps by someone trying to leave comment spam. Instead, I found a large number of requests for non-existant files, falling into two categories:

• Requests for favicon.ico, a file that does not exist on my site. Some of these requests are expected: most browsers with tabs request favicon.ico to display it in the tab bar. But there were also hundreds of IP addresses that requested nothing but favicon.ico for the entire day, and some requested it many times. About 100 of these IPs were Internet Explorer users with the Google Toolbar, so apparently I was getting DDoS'ed by a bug in the Google Toolbar. Another 100 were Firefox users; I haven't figured out why Firefox would request nothing but favicon.ico over and over.
• Requests due to people using my Real-time HTML Editor to edit pages that used relative URLs for images, iframes, etc. One user made dozens of requests for a file named "border=0". Another user made a request for 14 gif files every time the editor refreshed. I also saw from referrers that the Real-time HTML Editor had been featured on Digg, greatly increasing its traffic.

But why would 404 requests create PHP processes? Due to a recent change in WordPress, Apache was directing each 404 request to WordPress. WordPress used to put detailed rules in .htaccess -- for example, it would ask Apache to direct requests for http://www.squarefree.com/2005/ to WordPress using RewriteRule ^([0-9]{4})/?$. But newer versions of WordPress instead ask Apache to send it all requests for nonexistent files. I imagine this puts less strain on Apache when a site uses lots of WordPress Pages, but it hurts when a site gets lots of 404 requests. Several months ago, I had instructed WordPress to serve my custom 404 page for these requests, but WordPress still had to do a lot of work to determine that the requests should be treated as 404s.

Once I realized what had happened, and determined that reconfiguring WordPress would be difficult, I did what I could to reduce the number of 404 requests WordPress would have to handle. I created a tiny favicon.ico file so those requests wouldn't be 404s, and I moved the Real-time HTML Editor onto its own subdomain so WordPress wouldn't handle the 404s it causes. My site was only down for 40 minutes, with the Real-time HTML Editor down a little longer while I waited for the new subdomain's DNS to propagate.

Some things DreamHost could have done better:

• It would have been nice if James had disabled PHP for my domain instead of disabling my site entirely. Pornzilla did not need to be down due to PHP problems.
• A per-user process limit might have allowed my site to send "503 Service Unavailable" in response to some requests instead of being down entirely. It would have also prevented my site from causing problems for other sites on the shared server.
• Better performance diagnostics would have helped both James and me isolate the problem. For example, it would have been great to have a list of PHP processes showing the request URL that caused each PHP instance to be triggered, the lifetime of each process, and perhaps some performance information (CPU used, RAM used, number of database requests).

Some things DreamHost did right:

• DreamHost allowed me to restore my site myself once I fixed the problems. All I had to do was rename "squarefree.com_DISABLED_BY_DREAMHOST" back to "squarefree.com".
• Knowing about DreamHost's .snapshot feature kept me from panicking about data loss when my site appeared to have disappeared.
• The employees in #dreamhost were helpful.

If anyone is wondering: yes, I still love DreamHost.

DreamHost sent me automated notices that I was using over 100 CPU minutes a day on slaw.dreamhost.com, a web server with over 300 accounts. In other words, I was using at least 1/60 the capacity of the quad-core server. I guessed that a lot of my CPU usage was from the 10,000 hits a day for The Burning Edge's feed, so I installed wp-cache for The Burning Edge. The plugin doesn't seem to break anything; I think it invalidates its entire cache when anything changes (except for templates).

Installing the plugin for The Burning Edge reduced my CPU usage to about 55 minutes a day, low enough to stop the automated notices but still not within the desired range of 30-40 minutes a day. I just installed it for this blog too.

DreamHost's "777" promotion ended in July 2005, but DreamHost is now allowing me to share my referral bonus with you. If you enter the promotion code squarefree when signing up for any one-year or two-year plan, you'll get $77 off (and I'll get $20 for referring you). To use the promotion code, enter squarefree into the promotion code box on step 5 while signing up. (This is the step after you enter your personal information but before you enter your credit card number.)

If you use this code to sign up for a monthly plan, you'll only get $50 off, which will cover the setup fee that monthly plans have. Since DreamHost has a generous 97-day money back guarantee period, I recommend using a yearly plan instead.

Related: Why I love DreamHost and Snapshots (cool backup feature) on DreamHost.

Update Jan 2010: DreamHost has discontinued this feature.

I just discovered a cool feature at DreamHost called snapshots. This feature lets you access automatic backups of all of your files without contacting DreamHost tech support. I don't know why DreamHost doesn't advertise this feature more heavily; I bet it would cut down on tech support costs and help them attract customers.

To access the snapshots for files in any directory, type cd .snapshot in that direcory. Alternatively, you can cd .snapshot from any ancestor directory and then cd to the desired directory within the snapshot. Snapshot direcoties are hidden from ls -a, gzip, rm -rf, find, and even tab completion, so they don't get in your way when you're not using them. Inside this directory, you will find directories named hourly.0, hourly.1, daily.0, daily.1, weekly.0, and weekly.1. The hourly snapshots are taken every eight hours, the daily snapshots are taken every two days, and the weekly snapshots are taken every week. The .0 snapshots are newer than the .1 snapshots.

Some things you can do with snapshots:

See the dates of all snapshot versions of a file

ls -lt .snapshot/*/sidebar.php

By default, ls shows the date the file was modified (before the snapshot was taken). If you add -u, you can see the date the snapshot was taken instead.

See differences since a snapshot

diff -u .snapshot/nightly.1/sidebar.php sidebar.php | less

Recover an entire directory, including subdirectories

cp --recursive .snapshot/hourly.0/themes themes-recovered
mv themes themes-die
mv themes-recovered themes
rm -rf themes-die

Snapshots are read-only, so you can't accidentally destroy backups. The permissions displayed when you use ls are the permissions of the original files and do not reflect your ability to modify the backups themselves.

The user interface for DreamHost's snapshots is the same as the user interface described in this paper (HTML version), but I don't know what kind of implementation DreamHost is using. In the system described by the paper, snapshots are implemented using copy-on-write at the block level, so they don't take up much server time and disk space when files change slowly.

I found out about DreamHost's .snapshot feature yesterday by following a link from http://del.icio.us/tag/dreamhost to a DreamHost knowledge base article titled "Backups". I then searched Google for "cd .snapshot" to find out how common this feature is, get ideas for how to use it, and learn how it might be implemented.

If reading about this feature makes you want to use DreamHost as your web host, see Why I love DreamHost for other information about DreamHost and a discount.

Sixteen squarefree.com readers and #bs members and have signed up for DreamHost using DreamHost's 777 promotion since I started talking about it four days ago. What hosts did you switch from and how do you like DreamHost so far?

Update Aug 9, 2005: DreamHost's "777" promotion ended in July. At the same time, DreamHost is now allowing me to share my referral bonus with you. If you enter the promotion code squarefree when signing up for any one-year or two-year plan, you'll get $77 off (and I'll only get $20 for referring you). To use the promotion code, enter squarefree into the promotion code box on step 5. (This is the step after you enter your personal information but before you enter your credit card number.) End of update

Last night, Grey was looking to switch to a new web host because he was close to exceeding his bandwidth limit on TotalChoice Hosting. He knew about DreamHost because MightyMu, Mike Connor, and I use it. Grey was turned off by DreamHost's requirement that he either pay a $50 setup fee or pay for an entire year of hosting in advance. I remembered hearing about a "777" promotion code several months ago, so I suggested that he try it. The promotion code worked and Grey used it to switch to DreamHost today.

This promotion code allows new DreamHost customers to sign up for Crazy Domain Insane at $0.77/mo for the first year. After the first year, you will pay the normal price of $9.95/mo with one-year prepayment or $7.95/mo with two-year prepayment, unless you change to another plan or payment schedule.

Crazy Domain Insane includes 120 GB/mo bandwidth and 2.4 GB disk space (at least if you sign up before August 29, 2005), one free domain registration (.com, .net, .org, or .info), and hosting for up to 3 domains. All DreamHost plans include MySQL, PHP/Perl/Python, SSH login, mail, webmail, mailing lists, the ability to control your DNS records, and access to raw Apache logs.

For more information about DreamHost, see my earlier post Why I love DreamHost.

I wrote DreamHost Rewards Reorganizer, which puts the block containing "You currently have $X.XX of Rewards payments built up!" at the top of the page. Now I don't have to scroll down to see if I've earned another $97 for referring someone to DreamHost. My script does not conflict with JenMad's script, which acts on the rewards report page rather than the rewards overview page.