Free Robots.txt Generator
Pick a preset or set the options yourself. Click Generate, then copy or download the file. Everything runs in your browser, so nothing you type is sent anywhere.
Always review the generated file before you upload it. A wrong rule can block your site from search engines. See the full disclaimer below.
Every website gets visits from more than just people. Automated programs called crawlers, spiders, or bots scan your pages around the clock. They decide what to list in search results and how to rank it. A small text file named robots.txt lets you tell these bots where they can go and where they should stay out.
In this guide, you will learn what robots.txt is, why it matters for SEO, how to write one, and how to install it on Blogger, WordPress, and other sites. You can also use the free generator above to build your file in under a minute.
What Is a Robots.txt File?
A robots.txt file is a plain text file that sits in the root folder of your website. It follows a rule set called the Robots Exclusion Standard, which gives instructions to visiting crawlers.
When Googlebot, Bingbot, or any other bot arrives at example.com, the first thing it requests is example.com/robots.txt. If the file exists, the bot reads your rules before it fetches anything else. If there is no file, the bot assumes every page is open and starts crawling every link it can find.
Why Robots.txt Matters for SEO
Most site owners think SEO is only about keywords, backlinks, and speed. But search engines must be able to crawl your site efficiently before any of that counts. Here is how robots.txt helps.
1. It protects your crawl budget
Search engines don’t spend unlimited time on every site. Google gives each site a rough limit on how many pages it will fetch in a given period. This is called the crawl budget. For small sites it rarely matters, but on large sites with thousands of low-value URLs (internal search results, filter pages, session IDs), Googlebot can waste its time there and reach your important pages late.
2. It reduces server load
Aggressive bots and scrapers can send hundreds of requests per second. Blocking the ones you don’t need, or slowing down bots that respect Crawl-delay, keeps your server fast for real visitors.
3. It points bots to your sitemap
Your XML sitemap lists the pages you want indexed. Adding its address to robots.txt means every bot that visits can find it right away.
4. It reduces duplicate content
Stores and blogs often create several URLs for the same content through tags, categories, and sorting options. Blocking those extra URLs helps search engines focus on the main version of each page.
Syntax and Directives
You don’t need coding skills to write robots.txt, but you do need to be precise. One misplaced slash can block your whole site. These are the building blocks.
User-agent
This line says which bot the rules below it apply to. Use an asterisk for all bots, or a name for one specific bot.
User-agent: *applies to all crawlers.User-agent: Googlebotapplies only to Google’s main crawler.User-agent: Bingbotapplies only to Bing.
User-agent: * group completely. Repeat any rules you still want applied to it. The generator above does this for you.
Disallow
This tells the bot which paths it should not crawl.
Disallow: /admin/blocks everything inside the/admin/folder.Disallow: /checkoutblocks any URL that starts with/checkout.Disallow:with nothing after it means nothing is blocked.
Allow
Allow makes an exception inside a blocked folder. For example, to block an uploads folder but still let bots fetch one image:
User-agent: *
Disallow: /wp-content/uploads/
Allow: /wp-content/uploads/public-banner.jpgSitemap
Add the full address of your sitemap. This line does not belong to any user-agent group, so you can put it at the top or bottom of the file.
Sitemap: https://www.yourdomain.com/sitemap.xmlQuick reference
| Directive | What it does | Example |
|---|---|---|
User-agent | Chooses which bot the rules apply to | User-agent: * |
Disallow | Blocks a path from crawling | Disallow: /private/ |
Allow | Opens a path inside a blocked folder | Allow: /private/info.html |
Sitemap | Shows bots where your sitemap is | Sitemap: https://site.com/sitemap.xml |
Crawl-delay | Asks a bot to wait between requests (Google ignores it) | Crawl-delay: 10 |
Ready-to-Use Examples
Example A: Allow everything (standard blog)
If all your content is public, this is all you need:
User-agent: *
Disallow:
Sitemap: https://www.yourdomain.com/sitemap.xmlExample B: Block a staging or test site
If you are building a site and don’t want unfinished pages in search results, block everything:
User-agent: *
Disallow: /Warning: never leave this on a live site unless you want it to disappear from search results.
Example C: Online store
Stores built with WooCommerce or custom software usually need to keep bots out of the cart, checkout, and account pages:
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /search/
Sitemap: https://www.yourdomain.com/sitemap.xmlExample D: Block AI training bots only
If you want to stay in search results but keep certain AI crawlers away, add a group for each one:
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: *
Disallow:Blocking Crawling vs. Blocking Indexing
This is one of the biggest SEO misunderstandings: robots.txt does not remove a page from Google.
It only stops Googlebot from reading the page. If another website links to your blocked URL, Google can still list that URL in search results, usually with no description and a note saying that no information is available for the page.
<head> section: <meta name="robots" content="noindex">. If the page is blocked, Google can’t see the tag.
How to Install Robots.txt
Blogger (Blogspot)
- Sign in to Blogger and open your blog.
- Click Settings in the left menu.
- Scroll down to Crawlers and indexing.
- Turn on Enable custom robots.txt.
- Click Custom robots.txt.
- Paste the code from the generator above.
- Click Save.
A safe starting file for most Blogger blogs looks like this. It keeps the AdSense crawler allowed and blocks only the label and search pages, which create duplicate content:
User-agent: Mediapartners-Google
Allow: /
User-agent: *
Disallow: /search
Allow: /
Sitemap: https://yourblogname.blogspot.com/sitemap.xmlIf you use a custom domain, replace the sitemap address with your own domain. You can pick the “Blogger” preset in the tool to fill this in for you.
WordPress
Most SEO plugins, such as Yoast SEO and Rank Math, have a built-in robots.txt editor. Open the plugin’s tools or file editor, paste your code, and save. If you don’t use a plugin, create a text file named robots.txt and upload it to your site’s root folder (the same folder that contains wp-config.php).
Other websites
Save the generated text as robots.txt (all lowercase) and upload it to the main folder of your hosting account, usually called public_html. Shopify, Wix, and Squarespace have their own settings for this, so check their help pages before editing.
Common Mistakes to Avoid
- Blocking CSS and JavaScript files. Google needs these files to display your pages the way visitors see them. Blocking them can hurt your rankings.
- Blocking the AdSense crawler. If you show Google ads, make sure
Mediapartners-Googleis not blocked. Otherwise, Google may not be able to match relevant ads to your pages. - Forgetting that paths are case-sensitive. Blocking
/Admin/does not block/admin/. Use the exact spelling of your folders. - Putting the file in a subfolder. Bots look for it only at
yourdomain.com/robots.txt. A file atyourdomain.com/blog/robots.txtis ignored. - Using it to hide private data. Anyone can open your robots.txt and read it. Protect private areas with a password, not with this file.
- Ignoring Search Console warnings. Check the “Indexed, though blocked by robots.txt” status in Google Search Console and fix any page you blocked by mistake.
How to Test Your Robots.txt File
After you publish the file, check that it works:
- Open it in a browser. Go to
https://yourdomain.com/robots.txtand make sure the text matches what you generated. - Check Google Search Console. Go to Settings and open the robots.txt report. It shows when Google last fetched the file and lists any errors.
- Use the URL Inspection tool. Test an important page and confirm it says crawling is allowed.
Frequently Asked Questions
Do I need a robots.txt file?
No. Without one, bots simply crawl everything they can reach. But a good file helps you manage crawling and add your sitemap, so it is worth having.
Will robots.txt improve my rankings?
Not directly. It helps search engines spend their time on your best pages, which supports good SEO, but it is not a ranking factor on its own.
Can I block just one page?
Yes. Add a line such as Disallow: /my-page.html. Keep in mind that this stops crawling, not indexing. Use a noindex tag if the page must stay out of search results.
How long do changes take to work?
Google usually re-reads robots.txt within a day, but it can take longer. Pages already in the index may stay there for a while after you block them.
What happens if I delete my robots.txt file?
Bots will treat your whole site as open to crawling. Nothing breaks, but you lose control over what gets crawled.
Final Thoughts
A clean robots.txt file is a small job with a real payoff. It keeps crawlers away from pages that waste their time, protects your server, and points them straight to your sitemap. Use the generator above, double-check the result, test it in Search Console, and revisit it whenever your site structure changes.
References and Further Reading
These are the official sources behind the facts in this guide. Read them if you want the full technical details.
- Introduction to robots.txt – Google Search Central
- How Google interprets the robots.txt specification – Google Search Central (covers supported rules, case-sensitivity, and group matching)
- Block Search indexing with noindex – Google Search Central (explains why a page must be crawlable for noindex to work)
- RFC 9309: Robots Exclusion Protocol – IETF (the official standard)
- About the AdSense crawler – Google AdSense Help
- Fix AdSense crawler access issues – Google AdSense Help
Disclaimer
For information only. This article and the generator are provided for general educational use. They are not professional SEO, legal, or technical advice.
No guarantee of results. Search engines can change how they read robots.txt at any time. A correct file does not guarantee better rankings, faster indexing, or that a page will stay out of search results.
Check before you upload. A wrong rule, even a single slash, can stop search engines from crawling your whole site. Always review the generated file, test it in Google Search Console, and keep a backup of your old file. You use the tool and apply its output at your own risk, and the author is not responsible for any loss of traffic, rankings, or revenue.
Not a security tool. Robots.txt is public, and not every bot obeys it. Never use it to protect private or sensitive data. Use passwords or server-level access controls instead.
Privacy. The generator runs entirely in your browser. The text you enter is not sent to or stored on any server by this tool.
Trademarks and links. Google, Googlebot, Bing, Blogger, WordPress, AdSense, and other product names belong to their respective owners. This site is not affiliated with or endorsed by them. External links are provided for reference, and we are not responsible for the content of other websites.
.jpg)
0 Comments