Google obeying external REP requests?
Does Google Crawler actually check the http status codes of the robots.txt and obey them or does it just behave strangely.
moreHi, my name is Tim Nash I'm a "stuff" consultant, I look at web technologies and Information Architecture here on my blog. I live in Leeds, Yorkshire UK and am the Technical Director at Coding Futures Ltd. more
Does Google Crawler actually check the http status codes of the robots.txt and obey them or does it just behave strangely.
moreHacknorth is cancelled, Tim explains why and the plans for the future of the first Hackday in the north.
moreA quick round up of Tims latest movements including an overview of his talks at Think Visibility and then a trip to Paypal HQ in London for charityhack09
moreTim is looking for feedback on who he should pitch his behaviour modelling ideas to in form of seminars. Could it be you? Give him your feedback and shape the future of timnash.co.uk and his seminars.
moreTim is running a seminar pre Think Visibility the first public one he has done in a couple of years do you want to be at it?
moreThink Visibility is coming are you ready for it? Also would you be interested in joining me the day before for an indepth seminar in some of my latest research areas?
moreTim investigates a blackhat SEO cell in Leeds and discovers their secret plans at great danger to himself.
more