User-agent: * # comments begin with the hash. user-agent refers to the robot, in this case, all robots Disallow: /cgi-bin/ # disallow all robots from looking in the cgi-bin folder, you could do this with * if you want to disallow all # if you like search for robots to disallow that rip pages and email addresses etc # bad robots do not read robots.txt at all Disallow: /editor/system/ Disallow: /editor/articles/ Disallow: /editor/pages/