The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website. Although it’s not an official standard, using robots.txt is a common approach used for Internet-facing websites to exclude pieces of your website from crawling. Find out how you can easily create and manage the robots.txt file with Mavention Robots.txt for Sitecore.
What is Mavention Robots.txt for Sitecore?
Mavention Robots.txt for Sitecore is a module that allows you to easily manage the contents of the robots.txt file on your Sitecore web site.
Creating a robots.txt file using Mavention Robots.txt for Sitecore
After you have installed the package onto you Sitecore instance a new template for creating the robots.txt file is available. Open the Content Editor and create a new item in the tree under Sitecore > Content by navigating to File > Home > Insert From Template. After the new window has opened, select the Robots.txt settings template from Templates > Mavention > Robots Txt. Name the item robots.txt and click the Insert button.
It’s recommended to create the new robot.txt item outside of your navigation tree.
Once the new robots.txt item is created you can start adding you entries. The template already contains some entries to make you life a little easier.
Once you have your entries in place the last thing to do is to generate the robots.txt file. For this go to File > Publish and click the Publish button.
Mavention Robots.txt will automatically generate a robots.txt file for you using the entries you defined in the robots.txt item.
Mavention Robots.txt for Sitecore is a simple yet convenient solution for creating and managing robots.txt files on Internet-facing websites built on the Sitecore platform. Although it’s not that difficult to create a robots.txt file by hand, Mavention Robots.txt for Sitecore becomes extra useful in scenario’s when you don’t allow access to the filesytem of your website or want to provide your webmasters with an easy interface.
Download: Mavention Robots.txt for Sitecore module from the Sitecore Marketplace.