Skip to content

Attack of the killer robots

Today I had“  someone ask me how to keep bots off of their forum. They were seeing a noticed slowdown in the site when it was being indexed.“  That and they don’t want their “crazy cat lady talk” indexed on Google.

So here goes: if you want to keep Google off of your site – the easiest way is to add a file called robots.txt to the root directory of your website:

An example to disallow all robots would be:
User-agent: *
Disallow: /

With those 2 lines, all robots (including Google) will leave your site entirely alone.

If you wanted to disallow Google, but allow all others (like Yahoo, etc)
User-agent: Google
Disallow: /

User-agent: *
Disallow:

Everything you never wanted to know about the use of the robots.txt file is available at: http://www.robotstxt.org.

The best thing to keep in mind is that the “bad” bots will ignore robots.txt with wild abandon – doing their best to leave spammy comments for shoddy websites.“  For those bad boys you’ll need to block them by IP address, something we’ll address another day.

Leave a Reply

Your email address will not be published. Required fields are marked *