Robots.txt Generator
Generate a valid robots.txt file for any website. Choose from WordPress, Shopify, and other presets, or build custom rules — then download or copy your file in one click. Control search engine crawler access to protect admin pages, private sections, and low-value URLs.
Generated robots.txt
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /wp-includes/ Disallow: /wp-content/plugins/ Disallow: /wp-content/themes/ Sitemap: https://example.com/sitemap.xml
From directives to crawler control.
Build your robots.txt file in three steps — no coding required.
Choose a Preset or Build Custom Rules
Select a platform preset (WordPress, Shopify) to start with sensible defaults for your CMS, or choose "Custom" to build rules from scratch. Each preset blocks sensitive admin areas and low-value URLs that search engines should not index.
Configure User-agents and Directives
Add Allow and Disallow directives for specific URL paths. Target all bots with User-agent: * or restrict rules to specific crawlers like Googlebot or GPTBot. Add your sitemap URL to help search engines discover all your indexable pages.
Download or Copy Your File
Copy the generated robots.txt to your clipboard or download it as a .txt file. Upload it to your website's root directory (domain.com/robots.txt) and validate it in Google Search Console to ensure crawlers are reading it correctly.
Frequently Asked Questions
What is a robots.txt file?
A robots.txt file is a plain text file in your website's root directory that tells search engine crawlers which pages they can and cannot access. It follows the Robots Exclusion Protocol, respected by Googlebot, Bingbot, and other well-behaved crawlers. It's used to manage crawl budget, block admin pages, and prevent indexing of low-value URLs.
Where should I put my robots.txt file?
The robots.txt file must be placed in the root directory of your domain at https://yourdomain.com/robots.txt. It cannot be in a subdirectory — a file at /blog/robots.txt will not be found by crawlers. Most CMS platforms (WordPress, Shopify) handle this automatically.
Does robots.txt prevent pages from being indexed?
No — robots.txt controls crawl access, not indexation. A blocked page cannot be crawled, but Google can still index it if other websites link to it. To prevent a page from appearing in search results, use a noindex meta tag or X-Robots-Tag HTTP header instead of (or in addition to) a Disallow rule.
What is Disallow: / in robots.txt?
"Disallow: /" blocks all crawlers from accessing any page on your website. This is the most extreme robots.txt configuration and is commonly used on staging environments to prevent Google from indexing test content. Never deploy "Disallow: /" on a live production site.
How do I block specific bots in robots.txt?
Use a specific User-agent name followed by Disallow directives. For example, to block GPTBot (OpenAI's crawler): "User-agent: GPTBot" on one line and "Disallow: /" on the next. You can find the official User-agent names for specific bots in their documentation.
Should I include my sitemap in robots.txt?
Yes — adding "Sitemap: https://yourdomain.com/sitemap.xml" at the end of your robots.txt helps search engines discover all your indexable pages. It's especially helpful for large sites where crawlers might otherwise miss recently published content.
Can robots.txt be used to protect sensitive content?
No — robots.txt is not a security mechanism. Many bots (including malicious scrapers) ignore robots.txt entirely. Sensitive content should be protected by server-level authentication, login requirements, or IP restrictions. Listing sensitive paths in robots.txt can actually advertise their existence to bad actors.
How do I test if my robots.txt is working?
Use Google Search Console's Robots.txt Tester (under Settings) to test whether specific URLs are allowed or blocked. You can also use the URL Inspection tool to check how Googlebot views a specific page. After making changes, monitor your Crawl Stats report to verify the impact on crawl activity.