HyperTeamsHyperTeams
Get Started

Robots.txt Generator

Create robots.txt files for your website. Control how search engines crawl and index your content.

Rules

Rule 1

Examples: *, Googlebot, Bingbot, GPTBot

One path per line. Leave empty if none.

One path per line. Leave empty to allow all.

Time to wait between requests. Not all bots respect this.

Full URL to your XML sitemap. Helps search engines find your content.

Generated robots.txt

User-agent: *
Allow: /

How to Use

  1. 1. Configure your rules using the form on the left
  2. 2. Add multiple rules for different user agents if needed
  3. 3. Add your sitemap URL (recommended for SEO)
  4. 4. Copy or download the generated robots.txt file
  5. 5. Upload it to the root directory of your website

What is robots.txt?

A robots.txt file is a text file that tells web crawlers which pages or sections of your website should or should not be crawled and indexed. It is placed in the root directory of your website (e.g., example.com/robots.txt) and is one of the first files crawlers look for when visiting a site.

Why Do You Need robots.txt?

  • Control Crawling: Prevent search engines from indexing private or duplicate content
  • Save Server Resources: Reduce server load by limiting crawler access to unnecessary pages
  • SEO Optimization: Help search engines focus on your most important content
  • Privacy: Keep admin pages, staging areas, or user directories private
  • AI Bot Control: Block AI crawlers like GPTBot or CCBot from scraping your content

Common User Agents

Search Engine Bots

  • * (all bots)
  • Googlebot
  • Bingbot
  • Slurp (Yahoo)
  • DuckDuckBot

AI Crawlers

  • GPTBot (OpenAI)
  • CCBot (Common Crawl)
  • anthropic-ai
  • Google-Extended
  • ClaudeBot

robots.txt Best Practices

  1. Always include a Sitemap directive to help crawlers find your content
  2. Be careful with Disallow - blocking the wrong paths can hurt your SEO
  3. Use specific user agents for fine-grained control over different bots
  4. Test your robots.txt using Google Search Console or Bing Webmaster Tools
  5. Remember that robots.txt is publicly accessible - do not rely on it for security

Example robots.txt Files

Allow All Crawlers

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Block AI Crawlers

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Block Specific Directories

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/
Disallow: /tmp/

Sitemap: https://example.com/sitemap.xml