Disallow Rule Scraper
Parses all Disallow directives to identify blocked site directories.
Sitemap Locator
Extracts sitemap location declarations declared for search bots.
Raw Content Viewer
Inspect raw text directives formatting directly.
Fetching and auditing robots.txt file...
Robots.txt Status
Disallow Directives Count
0
Sitemap Declarations
0
Raw Robots.txt Content
What is a Robots.txt File?
A robots.txt file is a plain text file placed at the root directory of a website (e.g. https://example.com/robots.txt). It provides crawl instructions to search engine web crawlers (like Googlebot, Bingbot) detailing which pages or directories they are permitted or forbidden to crawl.
Auditing your site's robots.txt file ensures that search engines are not accidentally blocked from indexing key revenue-generating landing pages or CSS/JS design assets.
How to Use Robots.txt Checker
1 Enter Website URL
Input any domain or URL address into the search bar.
2 Click Check Robots.txt
Our engine fetches `/robots.txt` and parses all User-agent rules.
3 Inspect Directives
Review Disallow rules, Allow directives, and XML sitemap locations.
Robots.txt Best Practices for SEO
Recommended Practices
- Always declare your primary XML sitemap URL inside robots.txt.
- Disallow administrative folders (`/admin/`, `/wp-admin/`, `/cart/`).
- Ensure CSS and JavaScript assets are accessible to Googlebot.
- Keep file size under 500 KB to guarantee parser readability.
Pitfalls to Avoid
- Never declare `Disallow: /` on live production websites.
- Do not use robots.txt to hide sensitive user security data.
- Avoid conflicting Disallow and Allow wildcard rules.
