Robots.txt Generator

Robots.txt Generator tool helps you to generate effective 'robots.txt' file, that helps to ensure Google and other search engines are able to crawl and index your website properly.



Your robots.txt File

Copy the above code and save it in as 'robots.txt' file and upload to your website's top-level directory.
Example: http://www.example.com/robots.txt

If you want this rule to affect all files and directories, type '/'. To specify individual files or directories, type them one per line. Example:

/directory/file-name.html
			/directory/subdirectory/

You can also use your 'robots.txt' files to prevent dynamic URLs from being indexed by the search engine spiders. You can also instruct the spiders to avoid an entire folder on your website while still allowing it to access specific pages in that folder.

Also if you want the crawlers and spiders to avoid indexing certain types of files, you will need to use the dollar ($) sign symbol. Make sure you upload your XML sitemap in root folder.


           

Add a comment



About the Author
Joydeep Deb

is a Digital Marketing Specialist with 12+ years of experience in Digital Marketing, Lead Generation, Online Brand Management, Web Marketing Campaigns, Search Engine Optimization (SEO), Search Engine Marketing (SEM), PPC, eMail Marketing, Web Analytics, Web Technologies, Web Design and Development.

With an MBA in Marketing. Lives in Bangalore, Karnataka - India and working as an Program Manager SEO.

Get social with Joydeep at Twitter | Facebook | Google+ | LinkedIn

Robots.txt Generator


Subscribe to Newsletter

Receive my latest posts right in your inbox? Enter your email address below to subscribe.