What Does a Robots.txt Checker Do?
Robots.txt is the plain-text file at the root of a website (https://example.com/robots.txt) that tells crawlers which URLs they may fetch. One wrong line can hide an entire site from Google, or let AI crawlers read pages you meant to keep out.
This checker downloads the file the way a crawler does, reads every rule with Google's matching logic, and tells you for any URL and any bot whether it's allowed or blocked, and which line decided it. Google retired its own robots.txt Tester from Search Console in December 2023, so this is a quick way to test rules before and after you change them.
How to Test Your Robots.txt
Type a domain such as example.com, or a full page URL to test that page right away. The checker fetches /robots.txt from the site's root and follows redirects.
A 200 response means the rules apply. A 404 means crawlers may fetch everything. A server error makes Google stop crawling the site until the file loads again.
Enter a path like /blog/post?id=1 and pick a crawler. You'll see allowed or blocked, plus the exact rule and line number that matched.
Switch to Paste robots.txt to edit a draft and see the result instantly, then publish it and check the live file again.
Advertisement
How Crawlers Read Robots.txt
Google, Bing and most well-behaved crawlers follow RFC 9309, the robots exclusion standard. These four rules decide almost every result:
A crawler obeys only the group with the most specific User-agent that matches its name. If Googlebot has its own group, it ignores the User-agent: * group completely.
When several rules match a URL, the one with the longest path wins. Allow: /shop/sale/ beats Disallow: /shop/ for pages under /shop/sale/.
If an Allow and a Disallow rule are equally long, Google picks the less restrictive one, Allow.
* matches any characters and $ marks the end of the URL. Disallow: /*.pdf$ blocks every PDF but not /file.pdf?download=1.
What the HTTP Status of Robots.txt Means
Crawlers react to the response code before they read a single rule. This is how Google handles each case:
The file is read and its rules are followed. Google reads the first 500 KiB and ignores anything after that.
Google follows at least five redirects. If it still hasn't reached a robots.txt, it treats the file as missing.
Treated as if there were no robots.txt: every URL may be crawled. 429 Too Many Requests is the exception.
Google treats the whole site as blocked while the error lasts. After 30 days it falls back to the last cached copy, or assumes no restrictions if it has none.
Advertisement
Blocking AI Crawlers With Robots.txt
AI companies use separate crawler names for model training, AI search and on-demand fetches, so you can block one without the others. Blocking Google-Extended stops your content being used for Gemini but doesn't affect Google Search; blocking Googlebot would.
The checker's AI crawlers table shows at a glance which of 13 AI bots your robots.txt lets in. To block the main training crawlers, add:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
User-agent: Applebot-Extended
Disallow: /Common Robots.txt Mistakes
A leftover `Disallow: /` from a staging site blocks everything. It's the most common reason a new site never appears in Google.
Using robots.txt to hide pages. A blocked URL can still be indexed from links, just without a description. To keep a page out of search, let it be crawled and add a
noindexmeta tag orX-Robots-Tagheader.`Noindex:` lines in robots.txt. Google stopped supporting them in September 2019; they're ignored.
Blocking CSS and JavaScript. Google renders pages like a browser; blocked files can make your pages look broken to it.
Wrong letter case. Directive names aren't case-sensitive, but paths are:
Disallow: /Admin/doesn't block/admin/.One file for every subdomain. Robots.txt applies only to its own host and protocol.
blog.example.comneeds its own file.Crawl-delay for Google. Googlebot ignores it. Bing honours it.
Advertisement
Robots.txt Examples
Copy the pattern you need, then paste your full file into the checker to test it:
User-agent: *
Disallow:User-agent: *
Disallow: /admin/
Allow: /admin/public/User-agent: *
Disallow: /*?
Disallow: /*.pdf$Sitemap: https://example.com/sitemap.xmlMore SEO and Crawling Tools
Build and check the rest of your crawl setup: