I’ve recently installed some tracking software on my site so I can see who is online, who’s been online, and where they were from. Nothing sinister, but I just like to know what people are searching for in the context of where they live.
The software also lets you see which bots are crawling on your site. A bot (or spider) is sent by search engines to index your site so people can find it.
Now, you’d think that the likes of Google or Microsoft would be the ones doing all the crawling, wouldn’t you? In actual fact, a Googlebot makes a visit at such frequency that you will most likely not notice it has been there. At this moment in time, I have had a visit in the last 10 minutes or so from a Googlebot in Cabot, AR.
But get this: at the same time, I have been visited 22 times by Baiduspider (based in Beijing, China). And I bet the use of the words ‘Beijing’ and ‘China’ are going to get me a few dozen more visits.
Baiduspider appears to aggressively hunt for new posts. It has indexed my post about the first snow on all tags in a fraction of the time it takes Google to pick up on the same. Over the last few days, I have noticed that it is more or less continually present on my site.
I say this with my tongue firmly in my cheek, but I wonder why Beijing is so anxious to find all new posts so quickly?
EDIT: Wow. Both Google (Cabot, again) and Baiduspider got this within 1 minute of publishing! However, whereas the Googlebot is a single entity, Baiduspider is (so far) TWELVE separate entities. That’s 12 separate bots.