Controlling search engines and web crawlers using the robots.txt file
Controlling Search Engines and Web Crawlers Using robots.txt
Section titled “Controlling Search Engines and Web Crawlers Using robots.txt”The robots.txt file allows you to control which parts of your site search engines and web crawlers can access. You place the file in your document root (public_html) and use directives to specify crawler behavior.
Important Note
Section titled “Important Note”Directives in a robots.txt file are requests, not enforceable rules. Most search engines respect them, but some crawlers may ignore them. Do not rely on robots.txt to hide sensitive content.
Common Directives
Section titled “Common Directives”1. Allow All Crawlers to Access All Files
Section titled “1. Allow All Crawlers to Access All Files”User-agent: *Disallow:User-agent: *applies to all crawlers.Disallow:with no value allows access to all files.
2. Block All Crawlers from Accessing the Site
Section titled “2. Block All Crawlers from Accessing the Site”User-agent: *Disallow: /- The
/path blocks crawlers from all files on the site.
3. Block Crawlers from a Specific Directory
Section titled “3. Block Crawlers from a Specific Directory”User-agent: *Disallow: /scripts/- Crawlers are prevented from accessing the
/scripts/directory.
4. Block Crawlers from a Specific File
Section titled “4. Block Crawlers from a Specific File”User-agent: *Disallow: /documents/index.html- Crawlers cannot access the file
/documents/index.html.
5. Control Crawl Interval
Section titled “5. Control Crawl Interval”User-agent: *Crawl-delay: 30- Crawlers are instructed to wait at least 30 seconds between requests.
For more information about the robots.txt file, visit http://www.robotstxt.org.