Robots.txt Generator
Robots.txt Generator tool helps you to generate effective 'robots.txt' file, that helps to ensure Google and other search engines are able to crawl and index your website properly.
Copy the above code and save it in as 'robots.txt' file and upload to your website's top-level directory.
Example: http://www.example.com/robots.txt
If you want this rule to affect all files and directories, type '/'. To specify individual files or directories, type them one per line. Example:
/directory/file-name.html /directory/subdirectory/
You can also use your 'robots.txt' files to prevent dynamic URLs from being indexed by the search engine spiders. You can also instruct the spiders to avoid an entire folder on your website while still allowing it to access specific pages in that folder.
Also if you want the crawlers and spiders to avoid indexing certain types of files, you will need to use the dollar ($) sign symbol. Make sure you upload your XML sitemap in root folder.
Joydeep Deb is a Senior Digital Marketer and Project Manager with strong experience in Digital Marketing, Lead Generation, Online Brand Management, Marketing Campaigns, Project Management, Search Engine Optimization (SEO), Search Engine Marketing (SEM), PPC, eMail Marketing, Web Analytics, Web Technologies, Web Design and Development.
With an MBA in Marketing. IIM Calcutta Alumini. Lives in Bangalore, Karnataka - India.
