The Great WU Spam Debacle—Some Further Clarification

Thanks to Paul for giving a heads-up about the unfortunate loss of some archived comments. I thought I'd add some words as the resident WU tech-support guy to try to paint a slightly fuller picture of what happened, and hopefully allay some concerns.

First of all, the recent site redesign had nothing to do with the loss of some of the archived comments. The two are totally separate issues.

The story really begins during my two-year hiatus from active participation in the site, during which I failed to pay the close attention I should have to the functioning of the site, and so I didn't notice a massive, tumor-like mass of spam growing on the site. The problem wasn't the spam that appeared in the comments. That was only the tip of the iceberg. The real problem was the member list and the member profile pages. Many of you might not even be aware these existed... but the spammers were very aware of them. And thousands of them were registering as members every day so they could get a member profile page on which they could post spammy links in their "bios". By the time I noticed the problem, a few weeks ago, we had a quarter-million spammers registered as members.

Having that many spam members was having serious, negative consequences for the site. First, having up to 10,000 spammers registering a day was putting a big strain on the server and was slowing down the entire site. Second, the google bot was busily trying to index all these spam member pages, but was getting increasingly confused about what was the real content on the site, and was frantically indexing and re-indexing, provoking our webhost, at some point, to block the google bot entirely from the site.

So we were kind of hosed. Blocked from google and slowly being drowned in spam.

To fix the problem, the first thing I did was to block new member registration. Then I made the member profile pages non-public. Finally, I decided that all the spam members had to be deleted. The problem was how to delete the spammers while preserving the legitimate members and their comments.

I figured that the one unifying feature of all the spam members was that they hadn't left comments on the site. Instead, they had been posting the spam on the member profile pages. So I instructed the server to delete all members who had left no comments. The server churned and churned for a looooonnngg time. Finally, when it was all done, there were a couple hundred legitimate members left. And that, I thought, was the end of the matter. We were free to move forward with a clean slate.

Until Dumbfounded pointed out that a lot of old comments left by unregistered members had disappeared.

It was never my intention to delete any comments. In fact, I thought I had told the server specifically to preserve all the comments. But I'm guessing I wasn't specific enough in my instructions. I told the server to delete members who had left no comments, but I didn't say anything about NON-MEMBERS. So maybe the server interpreted my command to mean that it should delete everything EXCEPT members with comments. So all the non-registered guests got purged. That's my best guess as to what happened. Though I'm really not certain of this. The lesson is that whenever you give the server a broad, system-wide command to delete stuff, strange things can happen!

So there you have it. The Great WU Spam Debacle of 2012. At least the spam problem is far more under control than it was. So we've been left a little bruised and battered by the whole experience, but we're still standing. My apologies for the screw-up. Hopefully we can accumulate lots of great new comments to replace the old ones that are gone.