A robots text file, or robots.txt file (often mistakenly referred to as a robot.txt file) is a must have for every website. Adding a robots.txt file to the root folder of your site is a very simple process, and having this file is actually a ‘sign of quality’ to the search engines. Let’s look at the robots.txt options available to your site. A robots.txt is simply an ASCII or plain text file that tells the search engines where they are not allowed to go on a site – also known as the Standard for Robot Exclusion. Any files or folders listed in this document will not be crawled and indexed by the search engine spiders. Having a robots.txt, even a blank one, shows you acknowledge that search engines are allowed on your site and that they may have free access to it. We recommend adding a robots text file to your main domain and all sub-domains on your site.