Bot rules control how search engines and other automated visitors (called bots or crawlers) interact with your website. The Bot Rules page is where you tell search engines like Google which parts of your site they're allowed to look at, and whether they should include your site in search results at all.
This setting is powered by a standard file called robots.txt that lives on your website. The content you save here becomes the content of that file.
What you can do here
From the SEO > Bot Rules page, you can:
- Turn site indexing on or off for your whole site.
- Write or update the rules that tell search engines and other bots what they can and can't access.
Before you make changes
Bot rules have a big impact on how your site shows up in search engines. A bad rule can hide your entire site from Google. If you're not sure what you're changing, leave the defaults in place and reach out to support for help.
About Site Indexable
The Site Indexable toggle controls whether search engines are allowed to include your site in search results.
- On (recommended). Search engines can crawl and index your site. Your pages can show up in Google search results.
- Off. Search engines are asked not to index your site. Your pages will not show up in search results.
Turn this off only when you don't want your site to appear in search engines. Common reasons include a site that's still being built, a private staging site, or an internal-only website.
About the bot rules content
The large editor on the left side of the page holds the rules themselves. Each rule tells bots which parts of your site to allow or block.
A simple example looks like this:
User-agent: * Allow: / Disallow: /private/
That example says: "Any bot is welcome to look at any page, except for anything under /private/."
Writing bot rules uses a specific format called the robots.txt standard. If you're not familiar with it, we recommend leaving the defaults in place or reaching out to support for help.
Update your bot rules
Step 1: Open the Bot Rules page
- From the left sidebar, click SEO.
- Click the Bot Rules tab.
Step 2: Set Site Indexable
On the right side of the page, use the Site Indexable toggle to turn search engine indexing on or off.
Step 3: Update the rules (optional)
In the editor on the left, add or edit the bot rules as needed. Only make changes here if you know the robots.txt format.
Step 4: Save
Click Save & Publish in the top right corner.
Changes take effect on your live site after saving.