Today I had“ someone ask me how to keep bots off of their forum. They were seeing a noticed slowdown in the site when it was being indexed.“ That and they don’t want their “crazy cat lady talk” indexed on Google.
So here goes: if you want to keep Google off of your site – the easiest way is to add a file called robots.txt to the root directory of your website:
An example to disallow all robots would be:
With those 2 lines, all robots (including Google) will leave your site entirely alone.
If you wanted to disallow Google, but allow all others (like Yahoo, etc)
Everything you never wanted to know about the use of the robots.txt file is available at: http://www.robotstxt.org.
The best thing to keep in mind is that the “bad” bots will ignore robots.txt with wild abandon – doing their best to leave spammy comments for shoddy websites.“ For those bad boys you’ll need to block them by IP address, something we’ll address another day.