Robots.txt Generator

Robots.txt Generator tool helps you to generate effective 'robots.txt' file, that helps to ensure Google and other search engines are able to crawl and index your website properly.

Your (robots.txt) File

Copy the above code and save it in as 'robots.txt' file and upload to your website's top-level directory.

If you want this rule to affect all files and directories, type '/'. To specify individual files or directories, type them one per line. Example:


You can also use your 'robots.txt' files to prevent dynamic URLs from being indexed by the search engine spiders. You can also instruct the spiders to avoid an entire folder on your website while still allowing it to access specific pages in that folder.

Also if you want the crawlers and spiders to avoid indexing certain types of files, you will need to use the dollar ($) sign symbol. Make sure you upload your XML sitemap in root folder.

Joydeep Deb

Marketing . Innovation . Technology


About the Author
Joydeep Deb

Joydeep Deb is a Senior Digital Marketer and Project Manager with strong experience in Digital Marketing, Lead Generation, Online Brand Management, Marketing Campaigns, Project Management, Search Engine Optimization (SEO), Search Engine Marketing (SEM), PPC, eMail Marketing, Web Analytics, Web Technologies, Web Design and Development.

With an MBA in Marketing. IIM Calcutta Alumini. Lives in Bangalore, Karnataka - India.

Get social with Joydeep at Twitter | Facebook | LinkedIn

Robots.txt Generator

Subscribe to Newsletter

Receive my latest posts right in your inbox? Enter your email address below to subscribe.

We'll never share your email with anyone else.