Create a valid robots.txt to guide Googlebot and other crawlers. Set Allow, Disallow, Crawl-delay and Sitemap paths, then copy or download the file offline.
Part of our SEO & Marketing collection — Explore more SEO & Marketing tools · See also: Sitemap Generator, SERP & Open Graph Preview.
Quick answer: Robots.txt Generator — Generate robots.txt Free, no signup, runs in your browser. See below for the interactive tool, 4-step guide and FAQs.
Configure crawling rules for search bots. One path per line. Preview updates live.
e.g. / or /public/
e.g. /admin/ — leave blank to allow all
Numeric value, e.g. 10 or 2.5
Deep dives from the Tool4SaaS blog — tested workflows, print sizes and fixes.
Reviewed by the Tool4SaaS Editorial Team — In-house reviewers who build and test every tool locally for accuracy across current Chrome, Edge, Firefox, and Safari. Last updated: . See our authors, methodology or contact us.
It is a plain-text directive file placed at domain root telling polite crawlers which paths they may fetch. For example, Disallow /admin shields dashboards while Allow / opens public catalogs and Sitemap https://example.com/sitemap.xml invites discovery. Syntax uses User-agent, Allow, Disallow, Crawl-delay and absolute sitemap references.
No. Construction occurs entirely within your browser tab, so Allow, Disallow and Sitemap entries never leave the device. Generation works after initial load without accounts, produces UTF-8 text under 500KB instantly, and vanishes on tab close. Copy results directly for root deployment.
Place the UTF-8 file at https://example.com/robots.txt, keeping size below 500KB for reliable parsing. For example, combine User-agent star, Disallow /admin, Allow / and Sitemap https://example.com/sitemap.xml across four lines. Validate with Search Console tester after deployment for syntax confidence and confirm directive ordering carefully.
Introduce Crawl-delay 5 to 10 when shared hosting buckles under aggressive polite bots fetching thousands of images. For example, a vintage catalog with 8,000 thumbnails benefits from pacing, while Google ignores the directive and prefers Search Console rate controls. Bing and Yandex still honor delays, so retain modest values without expecting universal throttling.
Blocked URLs can remain indexed when external links point toward them, because Disallow prevents crawling but not index retention. For example, /admin/login hidden from fetching may still show title-only entries. Add noindex meta or authentication for genuine removal, then request Search Console clearing for sensitive endpoints.
Create a valid robots.txt guiding Googlebot and polite crawlers through Allow, Disallow, Crawl-delay and Sitemap directives without server uploads. Select User-agent star for universal coverage or Googlebot for search-specific tuning, then block /admin while opening public sections. Reference table clarifies intent: User-agent star Disallow /admin shields logins | Allow / permits catalog browsing | Crawl-delay 5 paces image fetching | Sitemap https://example.com/sitemap.xml advertises 1,200 URLs | Disallow /checkout?sort= avoids faceted duplication. Example one protects a boutique shop: star, Disallow /admin, Disallow /cart, Allow /, Sitemap product index, deployed under 500KB. Edge handling trims trailing spaces, normalizes capitalization of directives, and flags missing colons immediately before file generation completes smoothly. Administrators preview line endings and UTF-8 encoding consistency for reliable crawler parsing worldwide.
Rules match path prefixes from domain root, so Disallow /admin conceals /admin/login and /admin/reports while Allow / reopens public siblings for nuanced governance. Crawl-delay 5 instructs courteous visitors to pause seconds between hits, easing fragile shared hosts during image-heavy crawls, although Google disregards pacing and expects Search Console throttling instead. Keep files plain UTF-8 below 500KB because oversized directives truncate and squander crawl budgets. Example two calms a recipe blog: star, Disallow /wp-admin, Allow /recipes, Crawl-delay 8, Sitemap https://example.com/sitemap.xml with 2,400 posts, reducing timeouts noticeably. Edge cases include case-sensitive paths where /Admin differs from /admin, query strings requiring explicit Disallow patterns and conflicting Allow Disallow precedence resolved by longest match evaluation. Testers validate User-agent spelling, slash placement and absolute sitemap URLs before production release cycles.
Enter User-agent star, Disallow /admin, Allow /, Crawl-delay 5 and Sitemap https://example.com/sitemap.xml, then download the four-line file for https://example.com/robots.txt deployment in one action. Confirm syntax through Search Console tester, checking status 200 and correct text/plain UTF-8 headers for crawler acceptance. Limitations confine influence to cooperative bots only; malicious scanners ignore directives, sensitive URLs remain discoverable via external links, and ranking guarantees never follow. Authentication and noindex remains mandatory for genuine privacy. For XML discovery covering 1,200 product URLs with weekly changefreq and 0.8 priority, advance with our sitemap builder at /sitemap-generator to finalize crawl pathways cohesively. Re-audit quarterly after redesigns, migrations and staging exposures to prevent accidental Disallow inheritance. Document every Allow Disallow amendment with dates and owners for transparent SEO governance continuity.