robots.txt Generator

Control search engine bots.

Bot Directives

User-Agent

Disallow Paths (comma separated)

Allow Paths (comma separated)

Crawl-delay (seconds)

Clean-param (Yandex only)

XML Sitemap URL (Optional)

AI Crawler Simulation AuditChecks for catastrophic rendering blocks (e.g. blocking Next.js chunks or WordPress CSS).

Enterprise Crawl Budget OptimizationAI prunes parameter-heavy URLs, faceted navigation, pagination, and sorting parameters that destroy enterprise crawl budget.

AI Output

Your generated writing will appear here magically.

Tip: Ctrl+Enter runs the tool