{"id":1621,"date":"2009-12-18T12:56:32","date_gmt":"2009-12-18T12:56:32","guid":{"rendered":"https:\/\/www.diaryofanadi.co.uk\/?p=1621"},"modified":"2009-12-18T12:56:32","modified_gmt":"2009-12-18T12:56:32","slug":"spiders-and-bots","status":"publish","type":"post","link":"https:\/\/diaryofanadi.co.uk\/?p=1621","title":{"rendered":"Spiders And Bots"},"content":{"rendered":"

I’ve recently installed some tracking software on my site so I can see who is<\/strong> online, who’s been<\/strong> online, and where they were from. Nothing sinister, but I just like to know what people are searching for in the context of where they live.<\/p>\n

The software also\u00a0lets you see which bots are crawling on your site. A bot (or spider) is sent by search engines to index your site so people can find it.<\/p>\n

Now, you’d think that the likes of Google or Microsoft would be the ones doing all the crawling, wouldn’t you? In actual fact, a Googlebot makes a visit at such frequency that you will most likely not notice it has been there. At this moment in time, I have had a visit in the last 10 minutes or so from a Googlebot in Cabot, AR.<\/p>\n

But get this: at the same time, I have been visited 22 times by Baiduspider (based in Beijing, China). And I bet the use of the words ‘Beijing’ and ‘China’ are going to get me a few dozen more visits.<\/p>\n

Baiduspider appears to aggressively hunt for new posts. It has indexed my post about the first snow on all tags<\/strong> in a fraction of the time it takes Google to pick up on the same. Over the last few days, I have noticed that it is more or less continually present on my site.<\/p>\n

I say this with my tongue firmly in my cheek, but I wonder why Beijing is so anxious to find all new posts so quickly?<\/p>\n

EDIT:<\/strong> Wow. Both Google (Cabot, again) and Baiduspider got this within 1 minute of publishing! However, whereas the Googlebot is a single entity, Baiduspider is (so far) TWELVE<\/strong> separate entities. That’s 12 separate bots. <\/p>\n","protected":false},"excerpt":{"rendered":"

I’ve recently installed some tracking software on my site so I can see who is online, who’s been online, and where they were from. Nothing sinister, but I just like to know what people are searching for in the context of where they live. The software also\u00a0lets you see which bots are crawling on your […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[121],"tags":[],"class_list":["post-1621","post","type-post","status-publish","format-standard","hentry","category-latestposts"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t