Free Robots.txt Tester: See Exactly What Gets Blocked

Paste a robots.txt and a URL, pick from Googlebot to GPTBot to a custom crawler, and see the exact Allow or Disallow line that wins, and why. No signup, no daily cap, unlimited checks: the interactive tester Google's own Search Console dropped in 2023.

Google retired its own tester in December 2023

Search Console used to ship a genuinely interactive robots.txt Tester: paste a ruleset, paste a URL, pick a user-agent, get a verdict. Google pulled it on 12 December 2023 and replaced it with a read-only robots.txt report that only shows the file Google last fetched for a property you already own and have verified. You can no longer test a hypothetical rule against an arbitrary URL, or check a site you don't control, without leaving Search Console entirely. Google's own support forum still has threads from site owners asking why the tool disappeared.

What filled the gap were SEO-suite free tools that cap real usage at a handful of checks a day before pushing a trial signup. This tool has no cap, because the entire ruleset evaluation runs in your browser: nothing you paste is sent anywhere.

How precedence actually works, with a worked example

Two rules apply to the same path, so which one wins is not "first match" or "most recently written". Google's algorithm, now standardised as RFC 9309, is: within the group of rules that applies to a crawler, the rule whose pattern is the longest match by character count wins. If an Allow and a Disallow tie on length, Allow wins the tie.

Take this file:

User-agent: *
Disallow: /photos
Allow: /photos/public

A request for /photos/public/holiday.jpg matches both rules: Disallow: /photos (7 characters) and Allow: /photos/public (15 characters). The Allow rule is longer, so it wins and the crawler is allowed in, even though a Disallow rule for the parent folder exists. Reverse the specificity and the outcome flips: a request for /photos/private/holiday.jpg only matches the shorter Disallow rule, so it stays blocked. This is exactly the kind of interaction that is easy to get backwards by eye and is the whole reason the tool above shows the winning line and the reason it won, not just a yes or no.

Two more mechanics worth knowing: * inside a rule matches any run of characters, and $ at the very end of a rule anchors the end of the path, so Disallow: /*.pdf$ blocks any URL ending in .pdf but leaves /report.pdf?ref=email untouched, because the query string means the path doesn't actually end at .pdf.

The AI crawler question: GPTBot, ClaudeBot, CCBot, PerplexityBot

The most common robots.txt question in 2026 isn't about Googlebot at all. It's whether a site should block the crawlers that feed AI systems: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl, whose dataset many models train on), PerplexityBot (Perplexity's answer engine), and a growing list of others. These are separate user-agent tokens from Googlebot and Bingbot, so a rule written for User-agent: * covers them, but a rule written specifically for Googlebot does not, and neither does the reverse. Blocking Googlebot does nothing to GPTBot, and blocking GPTBot does nothing to your search rankings. That is the entire reason this tool lets you test every named crawler at once against the same file: the answer is genuinely different per crawler, and eyeballing one ruleset for twelve different bots is where mistakes happen.

There is no universally correct answer to whether to block them. Sites that don't want their content used for model training commonly block GPTBot and CCBot while leaving search crawlers untouched. Sites that want to appear in AI-generated answers leave the answer-engine crawlers (like PerplexityBot) open on purpose. Either choice is legitimate; the point of this section is that it's a per-crawler decision, not a single on/off switch.

robots.txt is a request, not a lock

Every rule in a robots.txt file is voluntary. Well-behaved crawlers, including all the ones named above from their operators' own documentation, read the file and respect it. Nothing on the server actually enforces it: a script that ignores the file entirely can still fetch every page, because robots.txt is not authentication and not a firewall. If content genuinely needs to stay off-limits, that requires a real access control (a login wall, an IP allowlist, a signed URL), not a text file that only asks nicely. robots.txt is the right tool for shaping which URLs a well-behaved crawler spends its budget on; it is the wrong tool for keeping anything actually private.

robots.txt versus a noindex tag, and why blocking a page can leave it indexed anyway

These solve different problems and mixing them up produces a specific, confusing failure. Disallow in robots.txt tells a crawler not to fetch a URL. noindex, set as a meta tag in the page's <head> or an X-Robots-Tag HTTP header, tells a crawler that has already fetched the page not to include it in search results.

The trap: if a URL is disallowed in robots.txt but other pages link to it, Google can still list that URL in search results, because a URL doesn't need to be crawled to be known about. What it can't do is read the page to see a noindex tag it was never allowed to fetch, and it can't show a real title or snippet, since it never saw the content. The result is a bare URL sitting in the index with no description, which usually looks worse than either fully indexed or fully absent. If the actual goal is "keep this out of search results", the fix is a noindex tag with the page left crawlable, not a Disallow rule. Disallow and noindex should almost never target the same URL at the same time.

Common mistakes that quietly break crawling

Blocking CSS and JavaScript. Google renders pages to judge layout and mobile-friendliness, and a folder-wide Disallow: /assets/ or Disallow: /_next/ that catches stylesheets and scripts can make a page render as a broken, unstyled shell in Google's eyes even though it looks fine to a visitor.
A relative Sitemap URL. Sitemap: /sitemap.xml is invalid; the directive requires a full absolute URL, Sitemap: https://example.com/sitemap.xml, and most crawlers silently ignore a relative one rather than guessing the domain.
Trailing-slash assumptions. /blog and /blog/ are different paths as far as prefix matching is concerned. A rule aimed at a whole section needs the trailing slash, or it can match unrelated URLs that merely start with the same characters, like /blog-archive.
Forgetting that a more specific group silently overrides the general one. A User-agent: * block and a User-agent: Googlebot block are never merged. If Googlebot has its own group, its rules are the only ones that apply to Googlebot, and anything only written under * is invisible to it.

Where the fetch-by-domain button fits in

Pasting a robots.txt is all this tool needs to work; that's the primary path and it never leaves your browser. The "fetch a live domain" option is a convenience on top: since a browser can't read an arbitrary third-party site's robots.txt directly (the site would need to opt in with CORS headers, and almost none do), that one step routes through a small server-side fetch. If that fetch ever fails for any reason, paste the file's contents instead, which is exactly as capable and doesn't depend on anything beyond this page. If you're also cleaning tracking parameters off outbound links while you're here, the link decoder does the same paste-and-see approach for URLs.

Questions

Does blocking a page in robots.txt remove it from Google's search results?

Not by itself. Disallow stops a page from being crawled, but Google can still list a disallowed URL in results (usually with no title or snippet) if other pages link to it. To actually keep a page out of results, use a noindex meta tag or X-Robots-Tag header on a page that is allowed to be crawled, not a Disallow rule.

Do I need separate rules for GPTBot, ClaudeBot and Googlebot, or does one rule cover all of them?

Separate rules. A User-agent: Googlebot group only applies to Googlebot; a User-agent: GPTBot group only applies to GPTBot. Only a User-agent: * group applies to every crawler that doesn't have its own named group. If you want one policy for search crawlers and a different one for AI crawlers, they need their own groups.

Should I block AI crawlers like GPTBot and CCBot?

There's no single right answer. It depends on whether you want your content used for model training (block GPTBot, CCBot) versus appearing in AI-generated answers (leaving crawlers like PerplexityBot open). Either is a legitimate choice; what matters is making it deliberately per crawler rather than assuming one rule covers all of them.

Why does the trailing slash on a path matter?

Robots.txt matching is a prefix match, so /blog also matches /blog-archive and /blog2026 unless the rule ends the pattern more precisely. Writing /blog/ (with the trailing slash) or /blog$ (anchoring the end) targets only the intended section instead of anything that happens to start with the same letters.

Is Crawl-delay respected by Google?

No. Crawl-delay is a non-standard directive that Bing and some other crawlers honour, but Google has never supported it and ignores it entirely. Google's crawl rate is instead controlled through Search Console's crawl-rate settings, not through robots.txt.

Does this tool send my robots.txt anywhere?

No. Parsing and rule matching both run entirely in your browser. The only thing that ever leaves your device is an optional domain lookup if you use the fetch-by-domain button, which retrieves a public robots.txt file server-side because browsers can't read another site's files directly; the file's contents are never logged or stored.

Test what your robots.txt actually blocks.