What is a robots.txt file?

robots.txt is a plain text file placed at the root of a domain that tells well-behaved web crawlers which parts of the site they may or may not crawl, using User-agent, Allow, and Disallow directives.

Where do I put the generated robots.txt file?

It must be placed at the root of your domain — accessible at https://yourdomain.com/robots.txt — not in a subdirectory. Most CMS platforms and static site generators have a specific location for this file.

What does 'User-agent: *' mean?

The asterisk is a wildcard meaning 'all crawlers' — any rules under a 'User-agent: *' group apply to every bot that doesn't have its own more specific group elsewhere in the file.

Can I block just one search engine and allow others?

Yes. Add a separate User-agent group with that specific bot's name and its own Disallow rules. Rules only apply to the crawlers matching that exact group's User-agent — they don't cascade to other bots.

Does robots.txt guarantee a page won't be indexed?

No. Disallowing a path in robots.txt only asks crawlers not to fetch it — a URL can still appear in search results if it's linked to from elsewhere. To reliably prevent indexing, use a 'noindex' meta tag or HTTP header on the page itself.

Should I add my sitemap to robots.txt?

Yes, it's a widely followed best practice. Adding a 'Sitemap:' line pointing to your sitemap.xml gives crawlers an easy, standard way to discover it.

What does Crawl-delay do, and do all search engines respect it?

Crawl-delay asks a crawler to wait a given number of seconds between requests to reduce server load. Google does not support Crawl-delay, but Bing and some other crawlers do honor it.

Can a wrong robots.txt file break my SEO?

Yes — a broad 'Disallow: /' under 'User-agent: *' blocks your entire site from being crawled, which is a common accidental mistake. Always double-check the generated file before deploying it to production.

Does this tool store or send my robots.txt content anywhere?

No. The entire file is built in your browser from the form fields you enter. Nothing is uploaded or transmitted — copying or downloading the result happens entirely client-side.

Robots.txt Generator — Build a Custom robots.txt File

Our Robots.txt Generator builds a syntactically valid robots.txt file from a simple form: add one or more User-agent groups, list Allow and Disallow paths for each, optionally set a Crawl-delay, and add a Sitemap directive. One-click presets cover common patterns like blocking /admin/ or /api/. The output updates live and can be copied or downloaded as robots.txt, ready to drop into your site's root directory.

User-agent

Disallow paths (one per line)

Allow paths (one per line)

Sitemap URL

Crawl-delay (seconds)

Generated robots.txt

User-agent: *

How to Use the Robots.txt Generator — Create a Valid Robots.txt File

1
Enter a User-agent (use '*' to target all crawlers, or a specific bot name like 'Googlebot').
2
List the paths to disallow and/or allow, one per line, or click a preset button for common rules.
3
Optionally add more User-agent groups, a Sitemap URL, and a Crawl-delay value.
4
Copy the generated robots.txt text or download it directly as a robots.txt file.

Why Use Robots.txt Generator — Create a Valid Robots.txt File?

A robots.txt file is the first thing well-behaved crawlers check before indexing your site, and a single syntax mistake can silently tank your SEO by hiding pages you wanted indexed or exposing pages you wanted hidden. Hand-writing robots.txt is deceptively easy to get subtly wrong, especially once multiple User-agent groups are involved. This generator enforces correct structure automatically: it groups directives under the right User-agent, orders Disallow/Allow lines correctly, and outputs a file that matches the format search engines expect, removing the guesswork.

Frequently Asked Questions

Related Tools

Robots.txt Tester — Check If a URL Is Allowed or Blocked

Test whether a specific path is allowed or disallowed by a robots.txt file, for any user-agent. Paste robots.txt or fetch it live from any site. Free, instant.

Sitemap Checker — Validate XML Sitemaps & Find Broken Links

Check any website's XML sitemap for errors, broken links, and missing metadata. Free online sitemap validator with live URL status checks — no signup required.

Website Intelligence Extractor — SEO & Metadata Snapshot Tool

Extract title, meta description, Open Graph tags, headings, word count, and internal/external link counts from any URL in one request. Free SEO snapshot tool.

Tech Stack Detector — Identify Any Website's Technology

Instantly detect the CMS, JavaScript framework, analytics tools, CDN, and server technology behind any website. Free online tech stack checker — no signup required.