Robots.txt Generator


Default - All Robots are:  
    
Crawl-Delay:
    
Sitemap: (leave blank if you don't have) 
     
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo
  Yahoo MM
  Yahoo Blogs
  Ask/Teoma
  GigaBlast
  DMOZ Checker
  Nutch
  Alexa/Wayback
  Baidu
  Naver
  MSN PicSearch
   
Restricted Directories: The path is relative to root and must contain a trailing slash "/"
 
 
 
 
 
 
   



Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.


About Robots.txt Generator

Search engines send out small programmes called Robots to your website to search your website or web pages and fetch information back so that your website or web page can be indexed properly and the web surfer can found your website. Robot.txt Generator tool does this for your website. Robot.txt Generator generates robot.txt files for your web pages. So, it will allow major web engines to crawl through each and every part of your web page. Robot.txt Generator is a very beneficial tool for your website. One more job of this tool is to tell the search engine to keep away from which part of your website. So, you also have to be careful while generating the Robot.txt file as it can also prevent search engines to crawl into your website. Several times a day, It cleans up your logs. One thing should also have in mind that malware robots can also ignore your robot.txt files if they want to scan the web for some security vulnerabilities. Robot.txt Generator can directly generate Robot.txt file by some custom settings which you have to do in the Robot.txt Generator tool. There are some dropdown boxes you have to choose according to your need and then press “Create Robot.txt” button.