Free SEO Tool
Robots.txt Generator
Generate a properly formatted robots.txt file for your website. Choose a preset or customize.
Auto SEO manages your technical SEO — automatically
Start $1 TrialQuick answer
What is a robots txt generator?
A robots txt generator builds the plain-text file crawlers read at yourdomain.com/robots.txt to learn which parts of your site they may request. This one starts you from four working presets - standard, ecommerce, blog, and restrictive - which you can edit inline and copy straight to your server. Robots.txt is the oldest crawler convention on the web and one of the easiest files to get quietly, expensively wrong.
The syntax, line by line
A robots.txt file is a series of groups, each starting with one or more User-agent lines followed by the rules for those crawlers. A crawler obeys the single most specific group naming it and ignores all others - so once you write a Googlebot group, Googlebot stops reading the wildcard group entirely and you must repeat any rules it still needs.
- User-agent: names the crawler. An asterisk matches any crawler with no group of its own.
- Disallow: a path prefix the crawler should not request. Disallow: / blocks the whole site; an empty Disallow: blocks nothing.
- Allow: carves an exception out of a broader Disallow. The more specific rule wins, regardless of order.
- Sitemap: an absolute URL to your XML sitemap. It is independent of user-agent groups and can appear anywhere in the file.
- Wildcards: * matches any run of characters and $ anchors the end of a URL, so Disallow: /*?sort= blocks any sorted URL and Disallow: /*.pdf$ blocks PDFs specifically.
- Case sensitivity: paths are case-sensitive, directive names are not. /Admin/ and /admin/ are two different rules.
- Comments start with # and the file must live at the domain root - a robots.txt in a subfolder is never read.
Robots.txt controls crawling, not indexing
This is the distinction that costs people traffic. Disallow tells a crawler not to fetch a URL; it does not tell a search engine to leave that URL out of its index. If other pages link to a blocked URL, Google can index it from those links alone and show it with no description - the familiar 'no information is available for this page' result.
To keep a page out of the index it must be crawlable and carry a noindex directive in its meta robots tag or X-Robots-Tag header. Blocking it in robots.txt as well guarantees the opposite, because the crawler can never fetch the page to see the noindex. For genuinely private content, neither directive is the answer: put it behind authentication.
Mistakes that cost real traffic
- Blocking /assets/, /static/ or /wp-includes/ and taking your CSS and JavaScript with them - Google renders pages, and one it cannot style or script may be judged broken.
- Using Disallow as a privacy control. robots.txt is public at a fixed URL, so a Disallow: /internal-pricing/ line is a signed map to what you wanted hidden.
- Combining Disallow with noindex on the same URL, which prevents the noindex from ever being read.
- Shipping a staging file to production. Disallow: / on a live site is the most damaging one-line deploy in SEO.
- Forgetting the trailing slash: Disallow: /news blocks /news, /news/ and /newsletter-signup alike.
- Omitting the Sitemap line, which is free discovery for every crawler that reads the file.
The AI crawler decision
Robots.txt is now where you decide your relationship with AI systems as well as search engines. The main groups worth naming explicitly are GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), CCBot (Common Crawl, which feeds many training sets), and Google-Extended - which controls use of your content in Google's generative products without affecting how Googlebot crawls and ranks you in classic search.
There is no universally right answer. Publishers whose product is the content itself often block them; businesses that sell something else usually allow them, because a blocked site cannot be cited in an AI answer. Make it a deliberate choice rather than one you inherit - the ecommerce preset here blocks GPTBot and CCBot by default, so remove those groups if you want AI visibility.
Where this data comes from
The generator is a client-side editor over four hand-written presets covering common site shapes: a standard site, an ecommerce store with faceted-URL and AI-crawler rules, a WordPress-style blog, and a restrictive allowlist that permits only Googlebot and Bingbot. Nothing is fetched or validated against your live site here - the output is a template you review, edit, and upload to your domain root. Verify the deployed file with a robots.txt tester before trusting it.
Frequently asked questions
Where does the robots.txt file go?
At the root of each domain and subdomain: https://example.com/robots.txt. Crawlers only ever request that exact path, so a file in a subfolder is ignored entirely. Subdomains need their own file - a robots.txt on example.com says nothing about shop.example.com. http and https on the same host are also treated as separate origins.
Does robots.txt stop a page from appearing in Google?
No. It stops the page being crawled, which is not the same as being indexed. A blocked URL that other sites link to can still appear in results, usually with no description because Google was never allowed to fetch it. To remove a page from the index, allow crawling and add a noindex meta tag or X-Robots-Tag header. For private data, use authentication.
Should I block AI crawlers like GPTBot and ClaudeBot?
It is a business decision, not a technical one. Blocking GPTBot, ClaudeBot, PerplexityBot and CCBot keeps your content out of AI training and answers. Allowing them keeps you eligible to be cited when an assistant answers a question in your niche, which is a growing referral channel. Google-Extended is separate again: it governs generative use without affecting classic Google Search.
What is the difference between Disallow and noindex?
Disallow is a robots.txt rule that prevents a crawler from requesting a URL. Noindex is a directive on the page itself - a meta robots tag or X-Robots-Tag header - that tells search engines not to keep it in the index. They are not interchangeable, and using both on one URL breaks the noindex, because the crawler must fetch the page to read it.
Do wildcards work in robots.txt?
Google and Bing both support * to match any sequence of characters and $ to anchor the end of a URL, which is how you block patterns like Disallow: /*?sort_by= or Disallow: /*.pdf$. Support is not universal across every crawler, since wildcards are an extension to the original specification rather than part of it. Keep rules as simple as the job allows.
How do I check my robots.txt is working?
Load yourdomain.com/robots.txt in a browser to confirm it is served as plain text at the root, then test specific URLs in Google Search Console's robots.txt report. Check the URLs you most want crawled, not just the ones you meant to block - the expensive errors are almost always over-blocking, and they are silent until traffic drops.
Related free tools
Crawlable is only step one
Once crawlers can reach your site, it needs pages worth crawling. Auto SEO writes and publishes keyword-targeted articles on autopilot, with clean metadata and structured data on every one. Start free.
Start your $1 trial