# Controlling search engines and web crawlers using the robots.txt file

> Use a robots.txt file to tell search engines and web crawlers which parts of your site they may visit.

Source: https://helpcenter.ruachost.com/developers/guides/controlling-search-engines-and-web-crawlers-using-the-robots-txt-file/

## Controlling Search Engines and Web Crawlers Using robots.txt

The `robots.txt` file allows you to control which parts of your site search engines and web crawlers can access. You place the file in your document root (`public_html`) and use directives to specify crawler behavior.

### Important Note

Directives in a `robots.txt` file are requests, not enforceable rules. Most search engines respect them, but some crawlers may ignore them. Do **not** rely on `robots.txt` to hide sensitive content.

### Common Directives

#### 1. Allow All Crawlers to Access All Files

```
User-agent: *
Disallow:
```

- `User-agent: *` applies to all crawlers.
- `Disallow:` with no value allows access to all files.

#### 2. Block All Crawlers from Accessing the Site

```
User-agent: *
Disallow: /
```

- The `/` path blocks crawlers from all files on the site.

#### 3. Block Crawlers from a Specific Directory

```
User-agent: *
Disallow: /scripts/
```

- Crawlers are prevented from accessing the `/scripts/` directory.

#### 4. Block Crawlers from a Specific File

```
User-agent: *
Disallow: /documents/index.html
```

- Crawlers cannot access the file `/documents/index.html`.

#### 5. Control Crawl Interval

```
User-agent: *
Crawl-delay: 30
```

- Crawlers are instructed to wait at least 30 seconds between requests.

For more information about the `robots.txt` file, visit [http://www.robotstxt.org](http://www.robotstxt.org/).