What does 'Allowed' or 'Disallowed' mean in the result?

It shows whether the specific path you tested would be permitted or blocked for the specified (or default '*') user-agent, based on parsing the robots.txt content and applying standard precedence rules.

How does the tool decide which rule wins when rules conflict?

It follows the widely adopted 'most specific rule wins' precedence: the Allow or Disallow rule with the longest matching path pattern takes priority, regardless of order in the file. If tied in length, Allow wins.

What's the difference between Paste mode and Live mode?

Paste mode lets you test a robots.txt file you're drafting or editing, before it's ever deployed. Live mode fetches the real robots.txt currently live on a public website's server.

Does Live mode work on any website?

It works on any public website with a reachable robots.txt file. For security, the tool blocks requests to localhost, private IP ranges, and other internal addresses.

Does the tool support wildcard (*) and end-anchor ($) patterns?

Yes. Both the '*' wildcard and the '$' end-of-string anchor are supported, matching the pattern syntax used by Google and other major crawlers.

What if I don't specify a user-agent?

The tool defaults to testing against the 'User-agent: *' group, which is the group that applies to any crawler without its own specific, named group in the file.

What happens if the path doesn't match any rule at all?

The result shows 'Allowed' with a note that no rule matched — this is correct behavior, since robots.txt operates on an allow-by-default basis.

Is this the same as Google Search Console's robots.txt tester?

It follows the same precedence logic Google's own tester and crawler use, though Search Console's version is tied to a verified property. This tool works on any site without requiring ownership verification.

Does fetching a site's robots.txt in Live mode notify that site?

The fetch is a standard HTTP GET request with a descriptive User-Agent header, which will appear in the target server's access logs like any other visit — no stealth or hidden request is made.

Robots.txt Tester — Test URLs Against Any Robots.txt File

Our Robots.txt Tester checks whether a given path is allowed or blocked according to a robots.txt file's rules, correctly applying the same longest-match-wins precedence real search engine crawlers use. Paste robots.txt content directly, or switch to Live mode to fetch any public site's actual robots.txt server-side. Enter a path and optional user-agent, and see exactly which rule matched and why.

robots.txt content

Path to test

User-agent (optional)

Paste robots.txt content and enter a path to test.

How to Use the Robots.txt Tester — Check If a URL Is Allowed or Blocked

1
Choose 'Paste' mode to paste robots.txt content directly, or 'Live' mode to fetch it from a real website.
2
In Live mode, enter the site's URL and click 'Fetch' to load its actual robots.txt content.
3
Enter the path you want to test and optionally a specific user-agent like 'Googlebot'.
4
Review the Allowed/Disallowed result and see exactly which rule in the file matched, based on the same specificity rules real crawlers use.

Why Use Robots.txt Tester — Check If a URL Is Allowed or Blocked?

Robots.txt rules can silently conflict — a broad 'Disallow: /' combined with a more specific 'Allow: /public/' further down the file is valid and common, but it's easy to misjudge which rule actually wins for a given URL without knowing the spec's precedence logic. Manually reading a robots.txt file and guessing the outcome is error-prone, especially across multiple User-agent groups and wildcard patterns. This tool applies the real precedence algorithm crawlers use and shows exactly which rule matched, removing the guesswork.

Frequently Asked Questions

Related Tools

Robots.txt Generator — Create a Valid Robots.txt File

Generate a valid robots.txt file with Allow/Disallow rules, sitemap directive, and crawl-delay. Free, browser-based robots.txt builder — download or copy instantly.

Sitemap Checker — Validate XML Sitemaps & Find Broken Links

Check any website's XML sitemap for errors, broken links, and missing metadata. Free online sitemap validator with live URL status checks — no signup required.

Website Intelligence Extractor — SEO & Metadata Snapshot Tool

Extract title, meta description, Open Graph tags, headings, word count, and internal/external link counts from any URL in one request. Free SEO snapshot tool.

Website to PNG, PDF, WebP & JPG — Free Screenshot Tool

Convert any website URL to PNG, PDF, WebP or JPG for free. Screenshot any webpage in desktop, tablet or mobile view. Instant — no signup required.