Getting Started

Bot Manager helps manage crawler access.

Use it to:

• Edit robots.txt
• Validate robots.txt
• Generate llms.txt
• Manage AI crawler access
• Allow or block selected bots

Robots.txt

Robots.txt tells crawlers which parts of your site they can access.

llms.txt

llms.txt helps AI systems understand important public information about your website.