Robots.txt Generator — Free
Generate a robots.txt file for your website. Control which bots can crawl which pages.
Quick Presets
Rules
User-agent: * Allow: / Disallow: /admin/
Block AI crawlers
One-click preset to block GPTBot, ClaudeBot, CCBot and other AI scrapers.
Live preview
Output updates instantly as you change rules or options.
Download ready
Download the file and place it at the root of your website.
What is a robots.txt generator?
A robots.txt file tells search engines and other web crawlers which parts of your site they're allowed to access, using a simple text format placed at the root of your domain. Pixvert's robots.txt generator builds this file visually, letting you allow or disallow specific paths, set a crawl delay, add your sitemap URL, and block AI crawlers like GPTBot or ClaudeBot with a single toggle.
Every major search engine checks robots.txt before crawling a site, and getting the syntax wrong — a misplaced wildcard, a missing colon, an incorrect user-agent line — can accidentally block search engines from your entire site or fail to block what you intended. This generator produces a syntactically correct file every time.
With AI training crawlers becoming more common, many site owners now want the option to allow search engine indexing while blocking AI models from scraping content for training — this tool includes presets for the most common AI bots so you don't need to look up their exact user-agent strings.
How to generate a robots.txt file
- 1
Choose a preset or start blank — select a common configuration (allow all, block all, or block AI bots) as a starting point
- 2
Set allow/disallow rules — specify which paths crawlers can or can't access
- 3
Add your sitemap URL — include a link to your sitemap.xml so crawlers can discover it directly
- 4
Set a crawl delay if needed — optionally throttle how frequently bots request pages from your server
- 5
Copy or download — copy the text or download the ready-to-use robots.txt file
When to use a robots.txt generator
Blocking admin or staging areas
Prevent search engines from indexing login pages, admin panels, or staging environments.
Blocking AI training crawlers
Disallow bots like GPTBot or ClaudeBot from scraping your content for AI model training while still allowing regular search indexing.
Pointing crawlers to your sitemap
Add a sitemap directive so search engines discover all your pages efficiently.
Managing crawl load
Set a crawl delay to reduce server load from aggressive bots on a resource-constrained server.
Launching a new site
Temporarily block all crawlers while a site is still in development, then update the file once ready to launch.
Why use Pixvert's robots.txt generator?
- ✓Correct syntax every time: no risk of a typo accidentally blocking your entire site from search engines
- ✓AI bot presets included: block GPTBot, ClaudeBot, and other AI crawlers with a single toggle
- ✓100% local and private: your configuration is generated entirely in your browser
- ✓Free, instant, no signup: generate as many robots.txt files as you need
Frequently Asked Questions
Where do I put the robots.txt file?
It must be placed at the root of your domain, accessible at yoursite.com/robots.txt — it won't be recognized anywhere else.
Does robots.txt guarantee my pages won't appear in search results?
Not entirely — it prevents crawling, but a disallowed page can still be indexed (without content) if other pages link to it. Use a noindex meta tag for a stronger guarantee.
Can I block specific AI crawlers only?
Yes, the generator lets you selectively block bots like GPTBot, ClaudeBot, CCBot, and others individually, or all of them at once.
Is my data sent to a server?
No, the file is built entirely in your browser based on your selections. Nothing is transmitted or stored.
Does blocking a crawler in robots.txt actually stop it?
It works for crawlers that respect the robots exclusion standard, which includes all major search engines and most reputable AI companies, but malicious bots may ignore it entirely.
Should every website have a robots.txt file?
It's not strictly required, but having one — even a simple "allow all, here's my sitemap" file — gives you explicit control instead of relying on default crawler behavior.