I know what you're thinking off the bat: "what does he mean, scraped?" Simply, it means that some web service is going through your web pages, looking for vulnerable entries. This could be email addresses, open database points, or unprotected comment forms. A more confusing explanation is on Wikipedia.

But it's simple to know if your site is being scraped, by a cursory glance at your system's server logs. Last night I had such occasion, as I noticed some unusual activity in my forum. That's right, my forum.

Now you're asking yourself, "but, waitaminute, I didn't know there was a forum here?" Guess what? Finding out that MacStansbury.com has a forum was news to me as well. Here's what happened, and how you can recognize, it as well.

I got a hit from my good friends in Spain (oddly enough, using the RIPE Network). And, 20 seconds later, on the same page, I got a hit from my good friends in China (who, completely unrelatedly were also using RIPE). Then, in a completely un-related event, my good German friends showed up (with the completely coincidental RIPE Network IP address). Then ze German started to go and look at my forum pages.

The ones that don't really exist.

I know this, because I would've had to install some.

As we all know, that being work...well, you know the rest.

But I digress. And I digress well...Here's a look at the pages Mr. Deutschland looked up:

http://forum.macstansbury.com/
http://forum.macstansbury.com/forum/index.php
http://forum.macstansbury.com/phpbb2/index.php
http://forum.macstansbury.com/phpbb/index.php
http://forum.macstansbury.com/forums/index.php
http://forum.macstansbury.com/board/index.php

Man, I didn't know I had so many forums! And with phpBB2, to boot! Why, that's just like what I had in 2004! But, not anymore. And not for 3 years.

This is the exact reason why I suggest the wholesale blocking of RIPE for most small blogs. If it weren't for some people who view my site from Asian countries (real people, not fake spammer-types), I'd have that block in place. As it stands, I'm back to the occasional scrape, but Bad Behavior helps cut down on that, as well.

Look for this kind of behavior whenever you're going through your server logs. This will help you recognize bandwidth theft, will help you cut off spammers, and will give you a healthy obsessive-compulsive obsession with how many people visit your site every day.

status: published
lastmod: '2006-10-01T01:14:02.000Z'
author: John Stansbury
cms: wordpress
custom:
  podPressPostSpecific: 'a:6:{s:15:"itunes:subtitle";s:15:"##PostExcerpt##";s:14:"itunes:summary";s:15:"##PostExcerpt##";s:15:"itunes:keywords";s:17:"##WordPressCats##";s:13:"itunes:author";s:10:"##Global##";s:15:"itunes:explicit";s:7:"Default";s:12:"itunes:block";s:7:"Default";}'
  _utw_tags_0: 'a:2:{i:0;O:8:"stdClass":1:{s:3:"tag";s:4:"rant";}i:1;O:8:"stdClass":1:{s:3:"tag";s:18:"social-engineering";}}'
  similarterms: forum macstansbury http index ripe site completely people phpbb recognize friends good pages scraped network logs forums digress german server