How does the tester select a crawler group?

It combines every group whose User-agent value exactly matches the entered product token, ignoring case. If no exact group exists, it combines wildcard groups. It never adds wildcard rules to a specific match.

How is the winning robots.txt rule chosen?

Matching begins at the first path octet and includes the query. The longest matching pattern wins, and an Allow rule wins an equal-specificity tie. The tool supports * and a terminal $.

Are encoded URLs and Unicode paths supported?

Yes. The tester applies RFC percent-encoding comparison rules, decodes percent-encoded unreserved ASCII, encodes Unicode as UTF-8 octets, keeps the query, and removes the fragment.

Does this tool fetch robots.txt from my website?

No. Paste the content you want to inspect. The static browser tool makes no fetch or XMLHttpRequest call, avoiding cross-origin and remote-content risks.

Can robots.txt protect private pages?

No. robots.txt asks compliant crawlers not to visit certain URLs, but it does not authenticate requests or prevent direct access. Protect sensitive content with real authorization.

robots.txt Tester - Test Crawler Allow and Disallow Rules

robots.txt Tester

Paste a robots.txt file, enter a crawler product token and URL, then see the applicable group and winning Allow or Disallow rule. Testing runs locally without fetching a website.

Test a robots.txt decision

robots.txt content99 / 512000 UTF-8 bytes

Content is parsed only in this browser. This tool does not fetch, upload, or send robots.txt data.

Crawler product tokenUse one product token containing letters, underscores, or hyphens—not a full User-Agent header.

URL or path to testThe path and query are tested. A fragment such as #section is removed.

Crawl guidance, not access control

robots.txt is voluntary guidance for compliant crawlers. It does not authenticate users, protect private content, or block direct browser access.

Test result

Paste the file, enter a valid product token and URL or path, then run the test.

Test robots.txt Allow and Disallow rules

This robots.txt tester follows the supported RFC 9309 decision path: it selects exact product-token groups, falls back to wildcard groups only when needed, normalizes URL octets, and chooses the longest matching rule. The pasted file never leaves your browser.

What this tester does not do

The tool does not fetch a remote /robots.txt file, imitate a live crawler, or verify that a search engine has refreshed its cached policy. robots.txt is crawl guidance and is not a security boundary.

When to test robots.txt

Check a proposed robots.txt change before deploying it to a website.
Compare the exact group selected for Googlebot, Bingbot, or another product token.
Explain why one Allow or Disallow rule wins by line number and matching-octet specificity.
Find ignored pre-group rules, malformed records, and unsupported records without changing the source text.

Related SEO tool

robots.txt Generator

robots.txt Tester FAQ

How does the tester select a crawler group?: It combines every group whose User-agent value exactly matches the entered product token, ignoring case. If no exact group exists, it combines wildcard groups. It never adds wildcard rules to a specific match.
How is the winning robots.txt rule chosen?: Matching begins at the first path octet and includes the query. The longest matching pattern wins, and an Allow rule wins an equal-specificity tie. The tool supports * and a terminal $.
Are encoded URLs and Unicode paths supported?: Yes. The tester applies RFC percent-encoding comparison rules, decodes percent-encoded unreserved ASCII, encodes Unicode as UTF-8 octets, keeps the query, and removes the fragment.
Does this tool fetch robots.txt from my website?: No. Paste the content you want to inspect. The static browser tool makes no fetch or XMLHttpRequest call, avoiding cross-origin and remote-content risks.
Can robots.txt protect private pages?: No. robots.txt asks compliant crawlers not to visit certain URLs, but it does not authenticate requests or prevent direct access. Protect sensitive content with real authorization.