Indistinguishable from Jesse

Back in February, Sprint invited me to become a Sprint Ambassador: they gave me a SPH-A920 phone and six months of free service, hoping that I'd give them useful feedback and/or blog about it. Here's some of the feedback I sent to Sprint:

The good

I loved being able to give my laptop access the Internet anywhere I had phone service. For example, the day after my son was born, I was able to research potential middle names while we were still at the hospital. I also used this feature to get some work done during boring parts of a family vacation. (This feature was free for me because I was part of the Ambassador program; I don't think it's free for other Sprint customers.)

Having an extra phone was surprisingly useful. I was able to lend my old phone to my girlfriend and talk with her for more than two minutes a day without worrying about her running out of minutes. I later found out that I could have added her to my family's plan for $10/month, but I wouldn't have thought of doing that otherwise.

The phone was better in many ways than my old phone, and it was certainly better than the phone for 4-year-olds Sprint sent to Joel Spolsky. But I still found a lot to be frustrated about with both the phone and the service.

The bad

There were some small problems with the "connect your laptop to the Internet" feature. The Windows software is hard to set up; my dad gave up on trying to get it work with his laptop. The fact that the feature works at all with Mac laptops kept secret from users. The phone gives up on the connection too easily when reception is poor, forcing me to click on "Connect" every few minutes. Once this feature works better, I hope Sprint promotes it heavily and stops trying to charge extra for it.

The "VoiceSMS" feature of the SPH-A920 phone works exactly as voicemail should: the messages are stored on my handset so I can listen to them even where reception is poor, I can see a list of messages without listening to a voice menu, I can rewind or fast-forward by 5 seconds by pressing arrow keys, and I can forward a message to my email address. But when someone tries to call me and leave a message, it doesn't become a VoiceSMS. Instead, it goes into the old-school voicemail system that everyone hates because it uses voice menus and makes you memorize shortcuts and surreptitiously deletes your messages after three weeks.

I'd like it if I could keep my phone on when I sleep, so I could respond to family emergencies. But I don't want every illegal telemarketing call, "wrong number" call, or poorly timed "how are you doing" call to wake me up. I want to be able to specify who is allowed to wake me up, and I want them to be greeted with "Jesse is asleep. Press 1 to leave a message or press 2 to wake him up." when they call.

When reception is poor on either end of a conversation, half of the conversation ends up being "I only heard every other word in that sentence" or "Are you still there?", and it takes forever to actually communicate something. This could be made much better by waiting until the entire sentence gets to my phone -- I'd gladly take some lag if it allowed me to actually hear the other person's sentences. I imagine the tricky parts would be figuring out where the breaks are if the other person is in a noisy area, and informing the other person about the lag so they don't confuse my slow responses for conversation pauses that need to be filled. Take some inspiration from the magic of TCP over IP and you could make using phones a lot less aggravating.

I couldn't figure out how to back up my contact list onto my computer. I get the feeling you're intentionally making this difficult in order to prevent customers from switching to other phone service providers. Don't make us sick the FCC on you again; you don't even have the "but that would be hard!" excuse you had with number portability.

The phone takes as long to start up as my computer. Palm handhelds start up quickly; why can't my phone?

Charging extra for each feature (e.g. automated 411, maps and directions, Web access using a browser in the phone, Internet using a nearby laptop, text messaging) is lame. The only charge should be for data transfer, which should be treated in the same way as "minutes". Customers who feel like they can't live without your phone are loyal customers; customers who feel that you tried to rip them off at every turn and barely used the phone's features are not. And I'd think loyal customers who stick with the service and tell their friends about it are worth much more than the revenue from infrequent use of the features.

I don't understand why the SPH-A920 is promoted as a "music phone". I can't plug my ordinary comfy headphones into it, because cell phones use a different type of headphone jack than laptops. I don't want to purchase music a second time through a menu on my phone; I want to sync with my computer and listen to music I own.

Yesterday, at around 4pm, I noticed that the content on squarefree.com was missing, and the main page was an empty directory listing. I ssh'ed to my web server and noticed that the "squarefree.com" directory had been renamed to "squarefree.com_DISABLED_BY_DREAMHOST". Then I checked my email and saw a message from DreamHost support:

Hello,

I just had to disable your site squarefree.com as it's coming under some load and spawning countless php processes that are crashing the webserver. I wasn't able to figure out exactly what's going on, as leaving it up for more than a minute pretty much toasts the server. Please don't re-enable it until you've figured out what's going on, or disabled any possibly problematic php.

Thanks,

James

I jumped into #dreamhost on irc.freenode.net and started looking through my web server logs for suspicious requests. I was expecting to find that my blog had been DDoSed, perhaps by someone trying to leave comment spam. Instead, I found a large number of requests for non-existant files, falling into two categories:

• Requests for favicon.ico, a file that does not exist on my site. Some of these requests are expected: most browsers with tabs request favicon.ico to display it in the tab bar. But there were also hundreds of IP addresses that requested nothing but favicon.ico for the entire day, and some requested it many times. About 100 of these IPs were Internet Explorer users with the Google Toolbar, so apparently I was getting DDoS'ed by a bug in the Google Toolbar. Another 100 were Firefox users; I haven't figured out why Firefox would request nothing but favicon.ico over and over.
• Requests due to people using my Real-time HTML Editor to edit pages that used relative URLs for images, iframes, etc. One user made dozens of requests for a file named "border=0". Another user made a request for 14 gif files every time the editor refreshed. I also saw from referrers that the Real-time HTML Editor had been featured on Digg, greatly increasing its traffic.

But why would 404 requests create PHP processes? Due to a recent change in WordPress, Apache was directing each 404 request to WordPress. WordPress used to put detailed rules in .htaccess -- for example, it would ask Apache to direct requests for http://www.squarefree.com/2005/ to WordPress using RewriteRule ^([0-9]{4})/?$. But newer versions of WordPress instead ask Apache to send it all requests for nonexistent files. I imagine this puts less strain on Apache when a site uses lots of WordPress Pages, but it hurts when a site gets lots of 404 requests. Several months ago, I had instructed WordPress to serve my custom 404 page for these requests, but WordPress still had to do a lot of work to determine that the requests should be treated as 404s.

Once I realized what had happened, and determined that reconfiguring WordPress would be difficult, I did what I could to reduce the number of 404 requests WordPress would have to handle. I created a tiny favicon.ico file so those requests wouldn't be 404s, and I moved the Real-time HTML Editor onto its own subdomain so WordPress wouldn't handle the 404s it causes. My site was only down for 40 minutes, with the Real-time HTML Editor down a little longer while I waited for the new subdomain's DNS to propagate.

Some things DreamHost could have done better:

• It would have been nice if James had disabled PHP for my domain instead of disabling my site entirely. Pornzilla did not need to be down due to PHP problems.
• A per-user process limit might have allowed my site to send "503 Service Unavailable" in response to some requests instead of being down entirely. It would have also prevented my site from causing problems for other sites on the shared server.
• Better performance diagnostics would have helped both James and me isolate the problem. For example, it would have been great to have a list of PHP processes showing the request URL that caused each PHP instance to be triggered, the lifetime of each process, and perhaps some performance information (CPU used, RAM used, number of database requests).

Some things DreamHost did right:

• DreamHost allowed me to restore my site myself once I fixed the problems. All I had to do was rename "squarefree.com_DISABLED_BY_DREAMHOST" back to "squarefree.com".
• Knowing about DreamHost's .snapshot feature kept me from panicking about data loss when my site appeared to have disappeared.
• The employees in #dreamhost were helpful.

If anyone is wondering: yes, I still love DreamHost.