Free tool · No signup
Googlebot checker
Test whether Googlebot — and Bingbot, GPTBot, ClaudeBot and PerplexityBot — can crawl a given path. We apply the same group-selection and longest-match rules Google's robots.txt parser uses.
Quick answer
What is a Googlebot checker?
A Googlebot checker tests whether a given URL path is crawlable by Googlebot according to a site's robots.txt. This tool goes further: it evaluates the same path for Bingbot, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, CCBot, and Google-Extended at once, applying the real matching rules - the most specific User-agent group wins, then the longest matching path pattern wins, with Allow breaking ties.
How robots.txt matching actually works
Most people read robots.txt top to bottom and assume the first matching rule applies. It does not, and that assumption is behind a large share of accidental blocks. Google's parser resolves two questions in order:
- Group selection: a crawler obeys exactly one group - the one whose User-agent value most specifically matches its token. A group for 'googlebot' applies to Googlebot-Image; a group for 'googlebot-image' does not apply to Googlebot. If any named group matches, the wildcard '*' group is ignored entirely for that crawler.
- Rule selection: within that group, the rule with the longest path pattern wins, regardless of order. 'Allow: /blog/public/' beats 'Disallow: /blog/' because it is longer and more specific.
- Ties go to Allow. Equal-length Allow and Disallow patterns matching the same path resolve as allowed.
- '*' matches any sequence of characters and '$' anchors the end of the URL, so 'Disallow: /*.pdf$' blocks PDFs anywhere on the site.
- An empty 'Disallow:' blocks nothing at all, while 'Disallow: /' blocks everything.
Crawling is not indexing
A robots.txt Disallow stops a crawler fetching a URL. It does not remove that URL from search results - Google can still index a blocked URL it has learned about from links, showing it with no description. If your goal is to keep a page out of the index, the tool is a noindex meta tag or an X-Robots-Tag header, and the page must be crawlable for that directive to be read at all.
Blocking a URL you have also marked noindex is the classic self-defeating combination: the crawler never fetches the page, never sees the noindex, and the URL lingers in the index.
AI crawlers deserve a deliberate decision
GPTBot, ClaudeBot, PerplexityBot, and CCBot fetch pages for AI systems. Google-Extended and Applebot-Extended are different animals - they are not crawlers at all, but opt-out tokens that control whether already-crawled content is used for AI training and generative features.
Blocking them keeps your content out of AI answers as well as AI training. If you sell something and want to be found where people now ask questions, allowing these crawlers while structuring your content for citation is usually the visibility-maximising choice. Either way it should be a decision you made, not one a CMS default made for you - which is what this checker is for.
- Crawl-delay is not supported by Google. Bing and several others honour it; Google's crawl rate is managed in Search Console.
- A robots.txt returning 5xx is treated by Google as a full disallow for the whole site until it recovers - a genuinely dangerous failure mode.
- A 404 on robots.txt means no restrictions, which is safe.
- Sitemap directives are global: they apply regardless of which User-agent group they sit next to.
Where this data comes from
Fetch a domain and we request its /robots.txt server-side over HTTPS, or paste the file to evaluate it without a fetch. Parsing follows the robots exclusion protocol: consecutive User-agent lines share the rules that follow them, comments are stripped, and unknown directives are reported but ignored. Matching implements Google's documented behaviour - most specific group, longest matching pattern, Allow wins ties, with '*' and '$' wildcard support. This is a faithful implementation of the public specification, not a call to Google's own tester.
Frequently asked questions
How do I test whether Googlebot can crawl a URL?
Enter the domain and the path here and the checker applies the same group-selection and longest-match rules Google's parser uses, showing which rule decided the outcome. For a definitive answer on your own verified site, Search Console's URL Inspection tool reports what Google actually did on its last crawl, including issues robots.txt cannot express.
Why does my page show as blocked when the Disallow appears above an Allow?
Order does not matter - specificity does. The longest matching path pattern wins, and Allow only breaks ties at equal length. If 'Disallow: /products/hidden/' is longer than 'Allow: /products/', the disallow wins wherever both match, no matter which line came first in the file.
Does blocking a page in robots.txt remove it from Google?
No. It stops Google fetching the page, but a URL discovered through links can still appear in results with no snippet. To remove a page from the index, allow crawling and serve a noindex meta tag or X-Robots-Tag header - Google has to fetch the page to see the directive. Blocking and noindexing at once guarantees the URL stays.
Should I block GPTBot and ClaudeBot?
It is a business decision, not a technical one. Blocking keeps your content out of AI training and out of AI answers and citations, which are a growing referral surface. If discovery matters more to you than exclusivity, allow them. What you should avoid is inheriting a block from a CMS default or a copied robots.txt without realising it.
What is Google-Extended?
A control token, not a crawler. Disallowing Google-Extended tells Google not to use your already-crawled content for Gemini training and generative features, while leaving normal Search crawling by Googlebot untouched. Applebot-Extended works the same way for Apple. Blocking them has no effect on your ordinary search rankings.
Does Google honour Crawl-delay?
No. Google ignores the Crawl-delay directive entirely; crawl rate is adjusted automatically and can be limited in Search Console. Bingbot and several smaller crawlers do honour it. This checker flags any Crawl-delay it finds so you know which crawlers it will and will not affect.
Related free tools
Crawlable pages are only half the job
Auto SEO publishes articles built to be fetched, understood, and cited - correct crawler access, clean structure, and schema on every page. Start free.
Start your $1 trial