Friday, May 30, 2008

Google Sample robots.txt

I went to this little SEO talk for developers at Google Fremont last night puttogether by Ignition EIR Vanessa Fox. Absolutely the highlight was hearing an anecdote about Google posting a sample robots.txt file for webmasters to use that excluded the MSN crawler - and how, as you can imagine, that same robots.txt file is still in use by a large number of websites to this day.

No comments: