The Googleplex Blog: Harold Davis's Blog

Spam Bites Man

I am the first to admit that I do not really understand email spam.

What kind of blithering idiot really thinks that an illicit fortune in Nigeria will actually be transferred to their bank account, or that some compound will make their sex organs or mammary glands larger, or that something offered in an unsolicited email might actually be a good idea?

Yet these blithering idiots must exist, otherwise spammers wouldn’t bother.

On the other side of the fence, what kind of jerk spends their time, creativity, and energy coming up with novel ways to bombard people with junk solicitations for stuff they don’t want (or are outright scams)?

Yet some jerks spend their time this way, or our inboxes wouldn’t be full of the stuff.

And my final rhetorical query: why all the indignation about spam? It is arguably easier to deal with email spam than junk snail mail (you just hit the delete key instead of having to cart it to the recycle bin) and less intrusive than telemarketing (the call inevitably comes just as dinner is on the table, or you are putting the kids to bed, and then you feel bad for being rude to a real human being).

Since email spam has recently bitten me, I now at least understand the pain that spam can cause (almost as bad as psoriasis) . Here’s my sad but true story of spam bites man.

First, we’ve set our business and personal email up for a long time to be forwarded by our web host (IX Webhosting) to our local ISP (Comcast). We then use Comcast’s POP and SMTP servers to pick up and send email.

Comcast’s spam filters have historically done a pretty good job of shielding us from the worst of spam. It’s true than my email address is published all over the place on websites, and that I’m listed as registry contact for various sites, so my email address was spammed frequently.

But Comcast shielded us from the worst of it, and my email client, Thunderbird, had junk mail screening tools that did the rest. Whatever got through I hit delete, and didn’t worry about it too much. (Although I did wonder: if I can tell from a glance at the sender and subject that an email is spam, why can’t an automated system?)

Those were the days!

The first sign of trouble was the report from a couple of friends and business associates that their email to me was bouncing with some kind of message from Comcast about spam. I pretty much assumed that the problem was with the senders, and that somehow they (or their ISPs) were associated with notorious spammers. Oh foolish pride! I should have realized that the problem was me (well, me and the whole spam email situation).

It was a dark and stormy night when our email stopped altogether. I don’t mean that the quantity slowed down or that we lost connectivity. I mean that we stopped getting our email. Our inboxes were empty.

Comcast technical support said sure your email works, and sent me an email to prove it. IX Webhosting said sure your email forwarding works, and had me forward an email to a new address to prove it.

I may not have been the brightest bulb on this issue, but it didn’t take me too long to figure out that Comcast had blocked all email from our domain as spam.

It turned out that they weren’t going to unblock it, either.

What to do? It seemed like the simplest solution was to use the email services for our domain provided by IX Webhosting, rather than forwarding the email to Comcast. I changed our server-side settings, and our email clients, and the email starting flowing again. And flowing. And flowing. Simply incredible quantities of spam, hundreds of spam emails in a single hour.

Perhaps Comcast’s blocking my email made some sense, after all.

What to do? It was time consuming and stressful to go through all the spam, and something had to give. Taking a new email address really didn’t seem desirable.

I did a Google search and decided to try a third-party email sanitizing service offered by an outfit called SpamStopHere.com. StamStopHere auto-generated a new set of MX records, which I was supposed to get IX Webhosting to implement. (I think this meant that I was asking IX to give over our email to this other outfit.)

IX Webhosting refused to comply, and SpamStopHere said, “We are adding IX Webhosting to the list of web hosts we cannot work with.”

Great.

IX Webhosting said that they had their own server-side filtering, and showed me how to write a custom filter in Thunderbird that recognized the spam header their filtering system added. (This, however, didn’t do Phyllis’s Outlook Express client any good, as it doesn’t have a mechanism for recognizing custom headers.)

IX Webhosting also had me implement a client-side open source program, SpamPal.

These tweaks were no doubt well intentioned, but they didn’t really touch my spam problem. One morning I spent an hour cleaning up my spam. (Well, it was complicated by a virus payload and Norton Antivirus. The AV software made me click OK to deleting each and every infected file, and OK again after the file had been deleted).

After a bit more research, I signed up for another service, Spam Arrest. You give Spam Arrest access to your email in-box, and it automatically downloads your email. Your email client connects to the Spam Arrest servers via a secure connection.

Spam Arrest has quite a few features, but mostly it works by maintaining blocked and unblocked (authorized) sender lists. A good starting place for the authorized sender list is your address book, which you can upload from any email client.

If you haven’t authorized the sender, Spam Arrest sends them a challenge-response email. The sender must click a link to authenticate the email, something most spammers can’t do.

The service costs $6.00 per month, with a free trial and various discounts. So far, I am really pleased with it. It’s great to open my email in-box and to see only a handful of emails, all of which are real. I also enjoy opening my webmail page on the Spam Arrest site and seeing all those hundreds of spam emails sitting forever lost in cyberspace (they are erased once a week by default). The goal of an intense week of my life was to elude spam, and I think I’ve succeeded!

Upgrading MovableType

I wrote a little while ago about customizing the "skin"---or look and feel---of my Photoblog 2.0. Recently I was confronted with a different task with my Googleplex Blog: upgrading from version 2.6 of MovableType to the latest and greatest version 3.31.

MovableType is the software that powers this blog. I upgraded not because I wanted some new bells and whistles (I tend to be pretty conservative about that kind of thing) but rather because I had to: security flaws in the old version had allowed some spammers to hijack the email notification system. Yuck! Why do people spend their time trying to hack systems? What a waste of time.

MovableType has moved in the enterprise direction, leaving your average solo webmaster to the wondrous delights of open source WordPress. So MovableType is now positioned as software to manage enterprise blogs---although they'll still let you download it for free for "personal" use.

As enterprise upgrades go, this one went pretty smoothly. I was careful to make backups of my server-side database, and to follow directions carefully. But, of course, the whole thing didn't work when I restarted it. And you get what you pay for: the free download personal version comes without support.

A little research showed that the problem was that I'd put my MovableType "static" directory within my web server's CGI bin directory, used to execute scripts, which is where my MovableType itself lived. Moving the MT static stuff out of there, and correctly pointing to it in the configuration files solved my problem.

I'm reasonably happy with the upgraded version, and I'm glad to have plugged my security problem. Personally, I still prefer WordPress, but I can see whay an enterprise running multiple blogs with multiple authors and a variety of editorial roles would want to manage them with MovableType.

More generally, I wonder about software getting more complex as it matures. Is it because there are more bells and whistles ("features") in successive versions? This process seems sort like the opposite of entropy, and yet related to entropy at the same time because of the chaos it throws into the lives of IT people who must cope with it.

Building Traffic and Making Money

SEO-the common abbreviation for Search Engine Optimization-is, as I've said before, "a wild-west frontier of the Internet: a boisterous new field with burgeoning revenues that employs tens of thousands of people with job titles and descriptions that did not exist five years ago."

That said, SEO has come of age, along with its slightly more respectable relative, CPC-or Cost Per Click-advertising via Google AdWords (or elsewhere).

Any business, no matter its size, needs to include an Internet marketing and advertising component as part of its business plan. For some businesses, this Internet marketing effort is the only advertising that is necessary. A key component of this marketing plan is crafting an effective SEO strategy.

In crafting your effective SEO strategy, it is necessary to walk a narrow line. Over aggressive tricks-sometimes called Blackhat SEO-do not work in the long run, except for scamsters. On the other hand, it is right and just to put one's best foot forward, and get as a high a natural search ranking as possible.

Contextually, SEO needs to be regarded as a component of online marketing and advertising. It's perfectly reasonable to allocate a percentage of resources to SEO at the same time as a CPC budget is formulated.

While there are some responsible and reputable SEO consultants, the field has not entirely emerged from an era of gaudy patent-medicine *get rich quick* hucksterism.

In response, my electronic book Search Engine Optimization: Building Traffic and Making Money is available as downloadable PDF from O'Reilly, the publisher. It's not about getting rich quick, and is focused on concise nuts and bolts issues. You can take this PDF to the boardroom and understand the technology issues underlying SEO (it won't replace your marketing consultant). If you are a do-it-yourself webmaster, my PDF should tell you everything you need to become a SEO whiz from a technical perspective.