Free tool
Robots.txt Generator
Build a robots.txt with user-agent groups, allow and disallow rules, sitemaps and ready presets.
How it works
What is the Robots.txt Generator?
The Robots.txt Generator is a free tool that builds a robots.txt file from user-agent groups, each with its own Allow and Disallow rules, plus Sitemap lines and an optional Crawl-delay. Start from a preset, such as allow all, block all, WordPress or Shopify-style, edit the rows, and copy the finished file.
How to use the Robots.txt Generator
- Pick a preset or start with the default group for all crawlers.
- Edit each group: the user-agent it applies to and its Allow and Disallow paths.
- Add your sitemap URL, and a crawl-delay only if a specific crawler needs it.
- Copy the file and upload it as robots.txt at the root of your site.
What is a robots.txt file?
robots.txt is a plain text file at the root of a host that tells crawlers which paths they may request. It is made of groups. Each group starts with one or more User-agent lines and is followed by Allow and Disallow rules. Sitemap lines can sit anywhere and apply to the whole file.
The file covers one host and protocol only. https://example.com/robots.txt says nothing about https://shop.example.com.
How do Allow and Disallow rules work?
A rule matches any URL whose path starts with the rule's value. Disallow: /private/ blocks everything in that folder. * matches any run of characters and $ marks the end of the URL. When an Allow and a Disallow both match, Google applies the most specific rule, meaning the longest path, and Allow wins a tie.
A crawler obeys only one group: the one with the most specific user-agent that matches it. A group for Googlebot replaces the * group for Googlebot; the two are not combined.
Does robots.txt stop a page from being indexed?
No. It stops crawling, not indexing. A blocked URL can still appear in search results if other pages link to it, usually with no description. To keep a page out of the index, let it be crawled and add a noindex robots meta tag or header.
robots.txt is also not access control. The file is public and compliance is voluntary, so never use it to hide private content.
Does Google respect crawl-delay?
No. Googlebot ignores the Crawl-delay line. Some other crawlers, including Bingbot, honour it. The generator still lets you add one per group and shows a note when you do, so you know which crawlers it reaches.
What should a robots.txt file contain?
For most sites, very little: a group for * that blocks admin, cart, checkout and internal search paths, and a Sitemap line. The WordPress preset blocks /wp-admin/ and allows admin-ajax.php. The Shopify-style preset is a short example of the kind of rules Shopify ships; Shopify generates its own file, which you change through its robots.txt theme template.
After uploading, test the live file with the Robots.txt Checker. To set rules for AI bots, use the AI Crawler Robots.txt Generator, and compare versions before a deploy with the Robots.txt Diff Tracker.
Questions
- Where do I put robots.txt?
- At the root of the host, so it loads at
https://example.com/robots.txt. A file in a subfolder is ignored. - What does an empty Disallow mean?
Disallow:with no path blocks nothing, so the group allows everything. The generator writes it for any group that has no rules.- Can one group cover several crawlers?
- Yes. Type the user-agents separated by commas and the generator writes one
User-agentline for each above the shared rules. - Is there a size limit for robots.txt?
- Google reads the first 500 kibibytes of the file and ignores anything after that. Few sites come close.
- Is the Robots.txt Generator free?
- Yes, with a free spyc.ai account. It runs in your browser, so nothing you type or paste is sent to a server.
Technical SEO Tools
More in Technical SEO Tools
Opengraph Viewer
Paste a URL and see the card it produces when someone shares it — title, description, image.
View OpengraphOpen Graph Generator
Write the Open Graph tags, preview the card, copy the markup.
Generate Open Graph tagsRobots.txt Checker
Fetch any site's robots.txt and test one URL against it: allowed, or blocked by which rule.
View robots.txtReady when you are
Before your next buyer asks AI about you, ask AI about yourself
One check, four engines, about twenty seconds. We show you the answers before we ask for anything.