{"id":1621,"date":"2009-12-18T12:56:32","date_gmt":"2009-12-18T12:56:32","guid":{"rendered":"https:\/\/www.diaryofanadi.co.uk\/?p=1621"},"modified":"2009-12-18T12:56:32","modified_gmt":"2009-12-18T12:56:32","slug":"spiders-and-bots","status":"publish","type":"post","link":"https:\/\/diaryofanadi.co.uk\/?p=1621","title":{"rendered":"Spiders And Bots"},"content":{"rendered":"

I’ve recently installed some tracking software on my site so I can see who is<\/strong> online, who’s been<\/strong> online, and where they were from. Nothing sinister, but I just like to know what people are searching for in the context of where they live.<\/p>\n

The software also\u00a0lets you see which bots are crawling on your site. A bot (or spider) is sent by search engines to index your site so people can find it.<\/p>\n

Now, you’d think that the likes of Google or Microsoft would be the ones doing all the crawling, wouldn’t you? In actual fact, a Googlebot makes a visit at such frequency that you will most likely not notice it has been there. At this moment in time, I have had a visit in the last 10 minutes or so from a Googlebot in Cabot, AR.<\/p>\n

But get this: at the same time, I have been visited 22 times by Baiduspider (based in Beijing, China). And I bet the use of the words ‘Beijing’ and ‘China’ are going to get me a few dozen more visits.<\/p>\n

Baiduspider appears to aggressively hunt for new posts. It has indexed my post about the first snow on all tags<\/strong> in a fraction of the time it takes Google to pick up on the same. Over the last few days, I have noticed that it is more or less continually present on my site.<\/p>\n

I say this with my tongue firmly in my cheek, but I wonder why Beijing is so anxious to find all new posts so quickly?<\/p>\n

EDIT:<\/strong> Wow. Both Google (Cabot, again) and Baiduspider got this within 1 minute of publishing! However, whereas the Googlebot is a single entity, Baiduspider is (so far) TWELVE<\/strong> separate entities. That’s 12 separate bots. <\/p>\n","protected":false},"excerpt":{"rendered":"

I’ve recently installed some tracking software on my site so I can see who is online, who’s been online, and where they were from. Nothing sinister, but I just like to know what people are searching for in the context of where they live. The software also\u00a0lets you see which bots are crawling on your […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[121],"tags":[],"class_list":["post-1621","post","type-post","status-publish","format-standard","hentry","category-latestposts"],"aioseo_notices":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/diaryofanadi.co.uk\/index.php?rest_route=\/wp\/v2\/posts\/1621","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/diaryofanadi.co.uk\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/diaryofanadi.co.uk\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/diaryofanadi.co.uk\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/diaryofanadi.co.uk\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1621"}],"version-history":[{"count":0,"href":"https:\/\/diaryofanadi.co.uk\/index.php?rest_route=\/wp\/v2\/posts\/1621\/revisions"}],"wp:attachment":[{"href":"https:\/\/diaryofanadi.co.uk\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1621"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/diaryofanadi.co.uk\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1621"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/diaryofanadi.co.uk\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1621"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}