Popular tools

Theme

Robots.txt Generator

Build a robots.txt you can trust: every rule explained as you type it, the file read back by our robots.txt checker against RFC 9309 and Google's own reading, and a live verdict for any crawler and any path before you publish.

Presets built from the WordPress, Shopify, Next.js and Google documentation · the crawlers each operator documents, by purpose · the deciding rule with its line · written in your browser, nothing uploaded
Written and checked in your browser

One file per host and protocol: www.example.com and example.com each need their own.

Start from
What it does
  • Every crawler may fetch every URL on the site.
  • The Sitemap: line points crawlers at your sitemap.
What it does not do
  • It keeps nothing private: robots.txt is a public file, and a request rather than a lock.
  • It does not ask a search engine to index a page. Crawling and indexing are separate.

Source: RFC 9309, read on October 26, 2026.

Rules

Every other crawler

User-agent: *

Read by every crawler that has no group of its own.

  • Allows the whole site.
Google ignores this; Bing reads it.

Crawlers by purpose

Choose what each known crawler gets. A crawler with a group of its own reads only that group.

Search engines8

They crawl to build a search index. Blocked pages can still be listed from links elsewhere, but with nothing read from them.

  • GooglebotGoogleYes, says the operator
  • Storebot-GoogleGoogleYes, says the operator
  • BingbotMicrosoftYes, says the operator
  • ApplebotAppleYes, says the operator
  • Amzn-SearchBotAmazonYes, says the operator
  • DuckDuckBotDuckDuckGoYes, says the operator
  • YandexBotYandexYes, says the operator
  • BaiduspiderBaiduThe operator does not say
AI training6

They collect pages to train models. A separate token lets you refuse training and still appear in search results.

  • Google-ExtendedGoogleA token only, never sent in a request
  • GPTBotOpenAIYes, says the operator
  • ClaudeBotAnthropicYes, says the operator
  • Applebot-ExtendedAppleA token only, never sent in a request
  • meta-externalagentMetaYes, says the operator
  • AmazonbotAmazonYes, says the operator
AI answers4

They fetch pages to answer questions with a link back to you.

  • OAI-SearchBotOpenAIYes, says the operator
  • Claude-SearchBotAnthropicYes, says the operator
  • PerplexityBotPerplexityYes, says the operator
  • meta-webindexerMetaYes, says the operator
Fetched for one person5

They fetch one page because a person asked for it, so some operators say they may ignore robots.txt.

Refusing these refuses a person, not a crawl: when someone asks an AI assistant to open one of your pages, an assistant that obeys the file will not read it for them. Refuse the training crawlers above if training is what you mean.

  • ChatGPT-UserOpenAINot always, says the operator
  • Claude-UserAnthropicYes, says the operator
  • Perplexity-UserPerplexityNot always, says the operator
  • meta-externalfetcherMetaNot always, says the operator
  • Amzn-UserAmazonNot always, says the operator
Previews, ads and others8

Link previews, site-owner tools, ad checks, archives, and crawlers whose operators state another job or none.

  • Google-InspectionToolGoogleYes, says the operator
  • GoogleOtherGoogleYes, says the operator
  • AdsBot-GoogleGoogleYes, says the operator
  • OAI-AdsBotOpenAIThe operator does not say
  • facebookexternalhitMetaNot always, says the operator
  • meta-externaladsMetaYes, says the operator
  • BytespiderByteDanceThe operator does not say
  • CCBotCommon CrawlYes, says the operator

Sitemaps

  • Type your site above and this fills in.

The file

1 warning

Your robots.txt/robots.txt

2 lines
  1. User-agent: *
  2. Allow: /

ErrorWarningNote23 B of 500 KiB

What the checker says

1 warning

  • No Sitemap lineWarning

    What to do It is the cheapest line in the file and every crawler reads it. Add Sitemap: with the full address of your sitemap.

Test it before you publish it

Pick a crawler and a path: the verdict, the rule that decides it and the group the crawler reads come from the Robots.txt Checker's own matcher.

Try a path
Crawler

  • This crawler may request the URL.

    Group chosen

    User-agent: *starts on line 1

    No group names this crawler, so the * group applies.

    Decided by

    Allow: /line 2

    The same rule also catches

    /any-page

    Compared

    /

    Only the path and query are compared, byte for byte after percent-encoding. The host, the scheme and the fragment are never part of a rule.

How to create a robots.txt file

Write the rules, save them as a plain text file named robots.txt, encoded as UTF-8, and upload it to the top level of your site, so that it answers at https://example.com/robots.txt. There is nothing to submit: crawlers ask for that address on their own.

Google puts it in three sentences: “The file must be named robots.txt. Your site can have only one robots.txt file. The robots.txt file must be located at the root of the site host to which it applies.” It must be “a UTF-8 encoded text file”, and “Google's crawlers will automatically find and start using your robots.txt file.”

The generator above starts from a preset, explains each rule as you type it, and reads the result back with the parser and matcher of our robots.txt checker before you copy it. The download is named robots.txt: UTF-8 with no byte-order mark, line feeds only, and no date or comment you did not ask for.

Four steps from rules to a working robots.txt
  1. 1Write the rulesStart from a preset, then add the groups and paths your site needs.
  2. 2Save it as robots.txtPlain text, UTF-8, no byte-order mark.
  3. 3Upload it to the rootIt must answer at the top of the host:https://example.com/robots.txt
  4. 4Test itHere before you publish, then the live file with the Robots.txt Checker.

There is no submit step: Google's crawlers find the file on their own.

Robots.txt generator: the WordPress preset, each rule explained, the file with its line numbers and the findingsRobots.txt generator: the WordPress preset, each rule explained, the file with its line numbers and the findingsWhat the preset doesAnd what it does not: none of these presets keeps a page out of Google, and each one says so.Each rule explainedThe paths a rule blocks and the ones it misses, from the same matcher that decides every verdict, and the longer rule that carves an exception.The file and its findingsThe file with its line numbers, read back by the Robots.txt Checker's own audit before you copy it.The verdictPick a crawler and a path: allowed or blocked, the rule that decided it with its line, and the group the crawler reads.
The WordPress preset in the generator: what the preset does and does not do, each rule explained from the matcher, the file as it will be published, and the verdict for a crawler and a path. Select a number to see what each part shows.

What goes in a robots.txt file, line by line

Groups and Sitemap lines. A group starts with one or more User-agent lines naming crawlers, followed by the Allow and Disallow rules those crawlers obey; a Sitemap line stands on its own, outside every group.

RFC 9309 defines a group as “one or more user-agent lines that are followed by one or more rules”, and a crawler obeys exactly one group: the one naming its own product token, or the * group when none does. Google reads four fields and no others: user-agent, allow, disallow and sitemap.

Field names are case-insensitive, paths are not: in Google's own example, disallow: /file.asp applies to /file.asp “but not” to /FILE.asp. A Sitemap: line must be a full address with the protocol and the host; this generator refuses any other kind and says why.

A robots.txt, line by line
robots.txt
  1. 1User-agent: *Starts a group for every crawler with no group of its own
  2. 2Disallow: /cart/A path these crawlers must not fetch
  3. 3Allow: /cart/helpAn exception: longer than the Disallow, so it wins
  4. 4
  5. 5User-agent: GPTBotStarts a group naming two crawlers
  6. 6User-agent: ClaudeBot
  7. 7Disallow: /Both are refused the whole site
  8. 8
  9. 9Sitemap: https://www.example.org/sitemap.xmlOutside every group: a full address

Allow vs Disallow, * and $: how a rule matches

When several rules match a URL, the one with the longest path wins, whichever kind it is, and an Allow wins a tie. * stands for any run of characters and $ marks the end of the URL; nothing else in a path is special.

RFC 9309: “The most specific match found MUST be used. The most specific match is the match that has the most octets”, and “if an allow rule and a disallow rule are equivalent, then the allow rule SHOULD be used.” WordPress's own file works this way: Disallow: /wp-admin/ closes the folder, and the longer Allow: /wp-admin/admin-ajax.php opens one file inside it.

A rule without $ is a prefix, so Disallow: /admin also blocks /administrator; add the slash (/admin/) when you mean the folder. Under every rule, the editor above shows the paths it catches and some it does not, from the matcher that decides every verdict, and it flags a rule that another one always beats.

* and $, and what each pattern really catches
PatternCatchesDoes not catch
/admin/admin/ · /administrator · /admin-tools/wp-admin
/admin//admin/users/administrator
/*.pdf$/files/report.pdf/report.pdf?v=2
/*?/search?q=x/search

?, ., [ and + are plain characters. There are no regular expressions in robots.txt.

Two rules match one URL: the longer path wins
/admin/report.pdf
  • Disallow: /admin/7 characters
  • Allow: /admin/report.pdf17 characters

The Allow wins: it matches more of the path.

An exact tie
Disallow: /files/7 charactersAllow: /files/7 characters

Equal length, so the Allow wins. “The least restrictive rule”, as Google puts it.

The example file belongs to www.example.org, a made-up site: no real robots.txt is shown as broken.

robots.txt for WordPress, Shopify and Next.js

All three already serve one, and the presets above write what each platform's own documentation describes: WordPress refuses its admin area but allows admin-ajax.php, Shopify refuses carts, checkouts and filtered or sorted collections, and Next.js serves the file you put in app/robots.txt.

WordPress writes its file in code, in do_robots(): User-agent: *, a Disallow for the admin path and an Allow for admin-ajax.php, plus, on a public site since WordPress 5.5, a Sitemap: line for /wp-sitemap.xml. With “Discourage search engines” on, WordPress 5.3 and later no longer write Disallow: /; they use a robots meta tag instead.

Shopify says its default “works for most stores”. There is no file to upload: changing it means editing the theme's robots.txt.liquid template, which Shopify calls “an unsupported customization”. Next.js serves app/robots.txt as written, or builds the file from app/robots.ts.

What WordPress, Shopify and Next.js serve, and where you change it
WordPress
  1. 1User-agent: *
  2. 2Allow: /wp-admin/admin-ajax.php
  3. 3Disallow: /wp-admin/
  4. 4
  5. 5Sitemap: https://example.com/wp-sitemap.xml

Written by WordPress itself. Change it with a plugin or the robots_txt filter.

Shopify
  1. 1User-agent: *
  2. 2Allow: /
  3. 3Disallow: /admin
  4. 4Disallow: /cart/
  5. 5Disallow: /checkout
  6. 6Disallow: /checkouts/
  7. 7Disallow: /account
  8. 8Disallow: /orders
  9. 9Disallow: /collections/*+*
  10. 10Disallow: /collections/*sort_by*
  11. 11
  12. 12Sitemap: https://example.com/sitemap.xml

Shopify's default. Change it in the theme's robots.txt.liquid template.

Next.js
  1. 1User-agent: *
  2. 2Allow: /
  3. 3
  4. 4Sitemap: https://example.com/sitemap.xml

Your own file in app/robots.txt, or app/robots.ts, served at /robots.txt.

Each file is what the preset of the same name writes for example.com.

Blocking AI crawlers: what actually works

A group naming the crawler, with Disallow: / under it, works for every crawler whose operator says it obeys robots.txt, and only for those. It is a request the crawler honours, not a lock: a crawler that ignores the file can still fetch every page.

Most AI companies run more than one crawler, so decide per job. Refusing a training crawler such as GPTBot or ClaudeBot refuses the one that collects pages for models; refusing an answer crawler such as OAI-SearchBot or PerplexityBot also takes you out of the links those assistants show. Google's training token is not Googlebot: “Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.”

The third kind is different. ChatGPT-User, Claude-User, Perplexity-User and Amzn-User fetch one page because a person asked an assistant to open it, so refusing them refuses that person, and some of their operators say these fetchers may ignore robots.txt anyway. The crawler list above keeps them in a section of their own for that reason.

The AI crawlers' three jobs, and what each operator says about robots.txt
OperatorTrainingAI answersOne person's request
OpenAIGPTBotObeysOAI-SearchBotObeysChatGPT-UserMay ignore
AnthropicClaudeBotObeysClaude-SearchBotObeysClaude-UserObeys
PerplexityNo such crawlerPerplexityBotObeysPerplexity-UserMay ignore
GoogleGoogle-ExtendedToken onlyNo such crawlerNo such crawler
AppleApplebot-ExtendedToken onlyNo such crawlerNo such crawler
Metameta-externalagentObeysmeta-webindexerObeysmeta-externalfetcherMay ignore
AmazonAmazonbotObeysNo such crawlerAmzn-UserMay ignore
  • TrainingCollects pages to train a model.
  • AI answersFetches pages to answer a question, usually with a link back to you.
  • One person's requestFetches one page because someone asked for it. Some operators say this one may ignore robots.txt.

robots.txt does not keep a page out of Google: use noindex

It asks crawlers not to fetch a page; it does not ask a search engine not to list it. Google “may still index the URL and show it in search results without a snippet” when other pages link to it. To keep a page out, use noindex, and leave the page crawlable so the tag can be read.

Search Console calls it “Indexed, though blocked by robots.txt”. And Google is explicit that “for the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file”: a disallowed page's own noindex is never seen, because the crawler never fetches the page.

That is why every preset above says what it does not do. The staging preset's Disallow: / keeps polite crawlers out, but a staging URL linked from anywhere can still be listed: put staging behind a password, and never ship that line to the live site.

Why a disallowed page can still appear in Google, and why its noindex is never read
  1. Crawler
  2. Reads robots.txt
  3. The URL is disallowed
  4. Does not fetch the page
The page itself<meta name="robots" content="noindex">Never read
Another site links to it
The search indexThe URL, with no description
Search Console calls thisIndexed, though blocked by robots.txt

To keep a page out: let the crawler in and use noindex, or require a login.

Where the robots.txt file must live

At the root of the host it applies to, as /robots.txt, one file per host and protocol. https://example.com/robots.txt says nothing about https://www.example.com or https://shop.example.com: each needs a file of its own, and a robots.txt inside a folder is never read.

Google: “You must place the robots.txt file in the top-level directory of a site, on a supported protocol.” Search Console's robots.txt report lists the files of the top 20 hosts of a property separately, with HTTP and HTTPS counted apart.

Type your site at the top of the generator and it gives the exact address the file must answer at, fills in the Sitemap: line for that host, and links to the Robots.txt Checker, which reads the live file once it is up.

One robots.txt per host and protocol
This fileApplies to
https://example.com/robots.txtEvery URL on https://example.com
https://www.example.com/robots.txtOnly www.example.com: a separate host
http://example.com/robots.txtOnly the http:// site: a separate protocol
https://shop.example.com/robots.txtOnly shop.example.com
https://example.com/blog/robots.txtNothing: a robots.txt in a folder is never read

Crawl-delay, Host, Clean-param and Content-Signal

Google reads none of them. Crawl-delay is read by Bing, Clean-param by Yandex, Host is in neither the standard nor Google's list, and Cloudflare's Content-Signal states what your content may be used for rather than what may be crawled.

Google lists its four fields and adds that “other fields such as crawl-delay aren't supported”. Bing's own blog says “the crawl-delay directive accepts only positive, whole numbers as values”, so the editor refuses a fraction rather than write one. Bing's current help page could not be read to confirm a range or a unit, so this page states neither.

None of them breaks a file: RFC 9309 requires that parsing other records “MUST NOT interfere with the parsing of explicitly defined records”. Of the four, the generator writes only Crawl-delay, labelled with who reads it.

Which search engine reads which directive
DirectiveGoogleBingYandex
User-agentReads itReads itReads it
AllowReads itReads itReads it
DisallowReads itReads itReads it
SitemapReads itReads itReads it
Crawl-delayIgnores itReads itNot confirmed
Clean-paramIgnores itIgnores itReads it
HostIgnores itIgnores itNot confirmed
noindexIgnores itNot confirmedNot confirmed
Content-SignalIgnores itNot confirmedNot confirmed
  • Google supports four fields and says so: user-agent, allow, disallow and sitemap.
  • Bing's own blog confirms Crawl-delay. We do not publish a range or a unit for it: Bing's current help page could not be read to check them.
  • Retired by Google in 2019, and never part of the standard.
  • Cloudflare's Content Signals Policy: a statement about how content may be used, not a crawl rule.

How to test the robots.txt file you just wrote

Twice. Before you publish, with Test it before you publish it above: pick a crawler and a path and see the rule that decides it. After you upload it, run the Robots.txt Checker on the live file, because what the server answers matters as much as what the file says.

A file that answers 404 is read as no file at all, so every crawler may take everything; a file that answers 500 or 503 means a crawler that follows the standard must take nothing. RFC 9309 §2.3.1.4: the crawler “MUST assume complete disallow”. Google softens that with a ladder of its own, shown in the figure.

Google's own robots.txt Tester is gone: Search Central announced its retirement on 15 November 2023, and the robots.txt report that replaced it shows the file Google fetched, without testing a URL against a rule. The generator's Test it before you publish it section above and the robots.txt checker do that test.

What each answer to GET /robots.txt means for the whole site
The server answersA crawler then
200Follows the rules in the file.
301, 302Follows at least five hops, then stops. Google then reads the file as a 404.
403, 404Crawls everything, as if there were no file.
429Crawls nothing. Google counts this with the 5xx answers, not the 4xx ones.
500, 503Crawls nothing at all, by the standard.
No such nameThere is no site to read, so there is no verdict.
What Google does with a 5xx, day by day
  1. 1First 12 hoursStops crawling the site and keeps retrying the file
  2. 2Up to 30 daysUses its last good copy of robots.txt
  3. 3After 30 daysIf the site is up, behaves as if there were no robots.txt

robots.txt mistakes that cost sites their traffic

The costly mistakes are each one short line: a Disallow: / left over from staging, a prefix that reaches further than you meant, a Sitemap: line written as a path, a path without its leading slash, and a noindex line Google stopped reading in 2019.

Each is caught here before you copy the file. The checker's own audit runs as you type and its findings sit under the file; a line a crawler would reject is never written at all, and the editor says on its row why it was left out.

One mistake is a pair, not a line: a URL disallowed here that also carries noindex on the page. The crawler never fetches the page, so it never sees the noindex, and the URL can stay in search results. Pick one: block the crawl, or let it in and tell it not to index.

Sources

Six robots.txt mistakes, and what each one really does
The line as writtenWhat actually happens
Disallow: /Left over from staging: the whole live site is closed to every crawler.
Disallow: /adminAlso blocks /administrator and /admin-tools. Add the trailing slash.
Allow: / under User-agent: *Never read by a crawler that has its own group anywhere in the file.
Sitemap: /sitemap.xmlA Sitemap line must be a full address with the host.
Disallow: adminA path starts at /. This rule matches nothing.
noindex: /private/Dead since 2019. The page stays crawlable and indexable.

Robots.txt generator: common questions

How do I create a robots.txt file?

Choose a preset above or write the rules yourself, then copy or download the file. Save it as plain UTF-8 text named robots.txt and upload it to the root of your site, so that it answers at https://your-site/robots.txt. There is no submit step: Google's crawlers find the file on their own. Then run the Robots.txt Checker on the live address to confirm the server answers 200.

Where do I put the robots.txt file?

At the root of the host it applies to, as /robots.txt, never inside a folder, where no crawler looks for it. Each host and protocol needs its own file, so example.com, www.example.com and shop.example.com each serve a separate robots.txt. On Shopify there is no file to upload: you edit the theme's robots.txt.liquid template. In Next.js the file goes in the app folder.

Do I need a robots.txt file at all?

Not strictly. When a site has no robots.txt and the address answers 404, a crawler may fetch everything, and RFC 9309 allows that. A file is worth having for the Sitemap line alone, and for keeping crawlers out of carts, internal search results or endless filter URLs. What matters most is that the address never answers a server error: a 5xx tells a compliant crawler to fetch nothing at all.

Can robots.txt remove a page from Google?

No. It stops a crawler fetching the page, not Google listing it. A disallowed URL linked from elsewhere can still be indexed and shown without a description, under the Search Console status “Indexed, though blocked by robots.txt”. To keep a page out, let Google crawl it and serve a noindex meta tag or X-Robots-Tag header, or put it behind a login. Never disallow and noindex the same URL.

Does blocking AI crawlers affect my Google rankings?

Not when you refuse AI training. Google-Extended, the token Google reads for model training, is separate from Googlebot, and Google says it does not affect a site's inclusion in Google Search and is not a ranking signal. Refusing GPTBot, ClaudeBot or the other training crawlers changes nothing for search engines either, because each crawler reads only the group that names it.

Should I block ChatGPT-User and Claude-User?

Only if you mean to refuse people. These fetch a single page because someone asked ChatGPT or Claude to open it, so blocking them stops that person's assistant reading your page; it does not stop a crawl. Some operators say these fetchers may ignore robots.txt anyway. If training is what you object to, refuse GPTBot and ClaudeBot instead: the No AI training preset does exactly that.

Do I need a separate robots.txt for each subdomain?

Yes. A robots.txt applies only to the host and protocol it is served from, so blog.example.com, shop.example.com and www.example.com each need a file of their own, and the rules in example.com/robots.txt never reach them. Search Console's robots.txt report lists files per host for the same reason. Type each host into the generator in turn: it fills in the right Sitemap line for each one.

Does Google support Crawl-delay?

No. Google lists the four fields it supports, user-agent, allow, disallow and sitemap, and says fields such as crawl-delay are not supported. Bing does read it, and Bing's own blog says the value must be a positive whole number, which is why this generator refuses fractions. A Crawl-delay line does no harm, but it will not slow Googlebot down, and Search Console no longer has a crawl-rate setting.

What is in WordPress's default robots.txt?

Three lines and, on a public site, a Sitemap line: User-agent: *, a Disallow for /wp-admin/, an Allow for /wp-admin/admin-ajax.php, and since WordPress 5.5 a Sitemap line for /wp-sitemap.xml. That is what WordPress's own do_robots() function writes. The longer Allow wins over the Disallow, so admin-ajax.php stays crawlable. Pick the WordPress preset to start from exactly that file.

How do I test my robots.txt file?

Before you publish, use Test it before you publish it on this page: pick a crawler and a path to see allowed or blocked, the rule that decided it with its line, and the group the crawler read. After you upload the file, run the Robots.txt Checker on your live site. It fetches the real file, so it also tells you what the server's status code means for every crawler.

Why does my Allow rule not work?

Usually because a longer Disallow matches the same URL: the longest path wins, whichever kind it is, and an Allow wins only a tie. Or the crawler has a group of its own, so it never reads the * group your Allow sits in. The editor above flags a rule that another one always beats, and Test it before you publish it names the deciding rule and the group the crawler read.

Is this generator free, and does it upload my file?

It is free, with no sign-up and no daily limit. Everything happens in your browser: the file is written, parsed and checked on this page, and nothing you type is sent anywhere. The only links out are the presets' sources and the Robots.txt Checker, which fetches your live file from our server only when you open it and ask for a check.

Missing something?Suggest a feature or a new tool

More crawling and indexing tools