Squiz Matrix allows you to create a robots.txt file to restrict access to sections of your site by robot (or spider) programs.

The permissions that you set for your site also restrict access by robot programs. For example, if you have an intranet site  or a member's area for which public read access has not been granted, Squiz Matrix will not allow access to these areas by robot programs.

The steps to create a robots.txt file are as follows:

1. Create a Text File asset in the root directory of the site to which you wish to restrict access.
2. You can then either upload a pre-existing robots.txt file or use the Edit Text screen of the Text File asset and enter the appropriate configuration. The syntax of a robots.txt file can be found at http://www.robotstxt.org/, and an example entry to restrict access by all robots to all areas of your site is:

   User-agent: *
   Disallow: /

3. Uncheck the Allow Unrestricted Access on the Details screen of the Text File asset.
4. Set the Status of the Text File asset to Live and ensure that you have granted Public Read access.

Once you've created your robots.txt file, you can test it to see if it's configured correctly using various online tools such as:

• https://support.google.com/webmasters/answer/6062598
• https://www.websiteplanet.com/webtools/robots-txt
• https://en.ryte.com/free-tools/robots-txt