Can I block ChatGPT and other AI crawlers?

Yes. Add `User-agent: GPTBot`, `User-agent: CCBot`, `User-agent: Google-Extended`, `User-agent: ClaudeBot`, and `User-agent: Bytespider` with `Disallow: /`. Note: robots.txt is a voluntary standard. Well-behaved bots (OpenAI, Anthropic, Google, Common Crawl) honor it; rogue scrapers ignore it and need other blocks (firewall, WAF).

Where does robots.txt live?

At the root of your domain — `https://example.com/robots.txt`. It must be served with a 200 status and `text/plain` content type. One robots.txt per host (subdomains need their own).

What's the difference between `Disallow` and `noindex`?

`Disallow` tells crawlers not to fetch a URL. `noindex` (in a meta tag or X-Robots-Tag header) tells them they can fetch but must not show it in search results. A disallowed URL can still appear in search results with no snippet if other sites link to it — which is why disallowed-but-sensitive pages should also use noindex or require authentication.

robots.txt Generator

SEO & Marketing· 10 tools

Free robots.txt generator. Build a valid robots.txt file with per-user-agent rules (Googlebot, Bingbot, GPTBot, CCBot), disallow and allow paths, optional crawl-delay, and your sitemap URL. Includes presets for common patterns — block all AI crawlers, allow everything, block admin/login pages, WordPress defaults. Output validates against the REP (Robots Exclusion Protocol) spec.

User-agent

Disallow paths (one per line)

Allow paths (one per line)

Crawl-delay (seconds, optional)

Sitemap URL (optional)

robots.txt

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Frequently Asked Questions

References

Google — Robots.txt specifications (REP) — Google Search Central

Related Tools

HTML Meta Tag Generator (SEO, OG, Twitter)
Generate a complete set of HTML meta tags: title, description, canonical, Open Graph, Twitter Card, and robots. Copy the full <head> snippet.
Google SERP Snippet Preview
Preview how your page title and meta description will appear in Google search results. Live pixel-width check + truncation warnings.
UTM Builder (Campaign URL Generator)
Build Google Analytics UTM-tagged URLs. Compose source, medium, campaign, term, and content parameters with proper URL encoding.

Frequently Asked Questions

References and sources

Related Tools