Robots Meta Tag Checker
Inspect `` directives (`index`, `noindex`, `follow`, `nofollow`, `noarchive`, `nosnippet`) and `X-Robots-Tag` HTTP response headers to ensure proper search engine indexing.
Indexability Check
Verify whether Googlebot, Bingbot, and search engines are permitted to index your URL or blocked via noindex.
Link Following Check
Inspect follow vs nofollow crawl rules for internal and external hyperlink equity.
X-Robots-Tag Inspection
Detect server-level X-Robots-Tag HTTP response headers that override HTML meta directives.
Scanning robots meta directives...
Search Engine Indexability Status
Indexable
Link Followability Status
Followable
| Bot Target Name | Directive Content |
|---|
Understanding Meta Robots & Crawling Directives
The meta robots tag provides page-level instructions to search engine web crawlers. By default, crawlers assume index, follow if no tag is present. Using directives like noindex, nofollow, noarchive, or nosnippet lets webmasters precisely control how search engines index pages and handle snippet previews.
How to Use the Robots Meta Tag Checker
1 Enter URL
Input any public website address or staging page URL.
2 Click Check Directives
Our tool fetches page HTML and HTTP headers to extract all crawler rules.
3 Verify Indexability
Review status badges for indexable and followable status before publishing.
Robots Directives Best Practices
Recommended
- Allow indexing (
index, follow) on all primary landing pages and posts. - Use
noindexon internal search results, staging environments, and thank-you pages. - Use server-level
X-Robots-TagHTTP response headers to control PDF/binary downloads. - Keep HTML meta directives consistent with HTTP headers.
Avoid
- Do not leave accidental
noindextags on production pages after launching. - Avoid blocking pages in
robots.txtif you want Google to see thenoindextag. - Avoid conflicting
indexvsnoindexrules across multiple tags. - Don't use
nofollowon internal navigation links unless necessary.
Frequently Asked Questions (FAQ)
robots.txt blocks crawlers from accessing a URL, but the page can still be indexed if linked externally. The noindex meta tag explicitly tells crawlers not to include the page in search engine index.
X-Robots-Tag is an HTTP header sent by the web server (e.g. Apache or Nginx). It allows webmasters to apply robots directives to non-HTML files like PDFs, images, or JSON API endpoints.
noindex, follow instruction tells search engine bots not to show the current page in search results, but allows bots to follow hyperlinks on the page to discover and rank target linked pages.
noindex tag, which usually takes from a few hours to a few days.
