Today I had“ someone ask me how to keep bots off of their forum. They were seeing a noticed slowdown in the site when it was being indexed.“ That and they don’t want their “crazy cat lady talk” indexed on Google.
So here goes: if you want to keep Google off of your site – the easiest way is to add a file called robots.txt to the root directory of your website:
An example to disallow all robots would be:
User-agent: *
Disallow: /
With those 2 lines, all robots (including Google) will leave your site entirely alone.
If you wanted to disallow Google, but allow all others (like Yahoo, etc)
User-agent: Google
Disallow: /
User-agent: *
Disallow:
Everything you never wanted to know about the use of the robots.txt file is available at: http://www.robotstxt.org.
The best thing to keep in mind is that the “bad” bots will ignore robots.txt with wild abandon – doing their best to leave spammy comments for shoddy websites.“ For those bad boys you’ll need to block them by IP address, something we’ll address another day.