Manage Your Robots.txt File

How to Manage Your Robots.txt File

The Robots.txt tab lets you edit and manage the robots.txt instructions used for website crawlers.

Because robots.txt can affect crawler access, review changes carefully before saving them.

In this guide

• Open Robots.txt Editor
• Find the robots.txt URL
• Understand the editor
• Save robots.txt
• Validate robots.txt
• Reset to Default
• Apply Enhanced
• Delete Custom Robots.txt

---

Open the Robots.txt Editor

Go to:

SEO Repair Kit → Bot Manager → Robots.txt

At the top, SEO Repair Kit displays the URL where your robots.txt file is available.

For example:

https://example.com/robots.txt

Always use the URL displayed for your own website.

---

Understand the Robots.txt Editor

The large editor displays the current robots.txt content.

Your screenshot contains an example configuration similar to:

User-agent: *

Disallow: /wp-admin/

Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

This controls crawler instructions and provides the sitemap location shown by the current configuration.

Important: Do not copy robots.txt rules from another website without understanding what they do. Incorrect directives can prevent crawlers from accessing important website areas.

---

Edit Robots.txt

Make the required changes directly in the Robots.txt Editor.

Review every rule before saving.

Pay particular attention to:

• User-agent
• Allow
• Disallow
• Sitemap

---

Save Robots.txt

After making changes, click:

Save Robots.txt

This saves the custom robots.txt configuration.

---

Validate Robots.txt

Click:

Validate

to use the validation workflow provided by Bot Manager.

Use validation after editing the file and before considering the configuration complete.

---

Reset to Default

Click:

Reset to Default

when you intentionally want to return the robots.txt configuration to the default provided by the feature.

Important: Review any custom crawler rules before resetting them because they may need to be recreated afterward.

---

Apply Enhanced

The Robots.txt Editor also provides:

Apply Enhanced

Use this action when you intentionally want Bot Manager to apply its enhanced robots.txt configuration.

Because your screenshot does not show the resulting rules after pressing Apply Enhanced, the documentation should not list or promise specific rules that this button will add until that resulting screen is documented.

This keeps the guide accurate to the current UI.

---

Delete Custom Robots.txt

Click:

Delete Custom Robots.txt

when you intentionally want to remove the custom robots.txt configuration managed through this screen.

Important: Deleting the custom file is different from resetting its content. Use this action only when the custom robots.txt should actually be removed.

---

Review the Public Robots.txt File

After saving, use the robots.txt URL displayed at the top of Bot Manager to review the accessible file.

Confirm that the saved rules match the intended crawler configuration.

Bot Manager FAQs →