Step by Step Guide

Step 1: Access Bot Manager

From your WordPress admin panel, go to SEO Repair Kit → Bot Manager. This is your central hub for controlling all bot and crawler access.

Step 2: Review the Overview Cards

At the top of the dashboard, you’ll see a quick summary:

• Robots.txt Status – Whether the file exists and is writable
• LLMs.txt Status – Whether the AI discovery file is generated
• Blocked AI Bots – Number of AI crawlers currently blocked
• Allowed Bots – Number of bots explicitly allowed

These cards give you an instant snapshot of your bot control health.

Step 3: Manage Robots.txt

Click on the Robots.txt Editor tab.

• Use the visual interface to add new rules (e.g., Disallow: /private/)
• Choose user agents from a dropdown (Googlebot, GPTBot, etc.)
• Set allow/disallow paths with auto-completion
• See a live preview of the generated file

You can also switch to Code View if you prefer manual editing.

Step 4: Control AI Bots

Go to the AI Bot Control section.

• See a list of known AI crawlers (ChatGPT, Claude, Bard, etc.)
• For each bot, choose:
  • Allow (default)
  • Block via robots.txt
  • Block at server level (403)
• Changes are applied immediately

Bot Manager automatically updates both robots.txt and server config files when needed.

Step 5: Generate LLMs.txt

Navigate to the LLMs.txt Generator.

• Enable the generator with a single toggle
• Select which post types to include (posts, pages, custom post types)
• Select which taxonomies to include (categories, tags, etc.)
• Preview the generated llms.txt file
• Click Save & Generate

The file will be created at yoursite.com/llms.txt and automatically updated when you publish new content.

Step 6: Validate and Test

Click the Validate button to check for:

• Syntax errors in robots.txt
• Conflicts between rules
• Missing directives

Use the Test Crawler tool to simulate how a specific bot (e.g., GPTBot) would see your site.

Step 7: Apply Server-Level Blocking (Optional)

For enhanced security, enable Server-Level Enforcement:

• Go to Advanced Settings
• Toggle on “Block unwanted bots at server level”
• Choose which blocked bots should receive a 403 error
• Bot Manager writes the necessary rules to .htaccess or nginx.conf

Step 8: Monitor Bot Activity

Visit the Bot Log section (if enabled) to see:

• Which bots have accessed your site recently
• How often they were blocked or allowed
• Bandwidth usage by crawler

Use this data to refine your bot management strategy.

Step 9: Export Configuration (Optional)

Click Export Settings to download a JSON backup of your:

• Robots.txt rules
• AI bot blocklist
• LLMs.txt settings

This is useful for migrating configurations to other sites or keeping a backup.

Step 10: Maintain Ongoing Bot Control

Regularly review:

• New AI crawlers added to the known list (plugin updates bring them in)
• Changes to your site structure that may need new robots.txt rules
• LLMs.txt content after adding new post types or taxonomies

Set up automatic weekly scans to ensure your bot policies remain optimal.