Google Spiders Ignoring Meta Tags & Robots.txt Files?
January 15th, 2007 by pamela
Digital Inspiration has posted on Googlebot ignoring the robots META tag or robots.txt instructions, the two methods used to prevent Google crawling webpages. Taking two case studies – one from del.icio.us and another from Google Finance, it seems that Googlebot has even cached pages form one of it’s own sites which were disallowed.
The responses generated by the posting are interesting, have you noticed this happening?
