Every website has a conversation with search engines, and robots.txt is where that conversation starts. It is a small text file that tells crawlers which parts of your site they may visit and which they should leave alone.
Get it right and search engines index your important pages while ignoring admin areas and duplicate content. Get it wrong and you can accidentally block your entire site from Google — a mistake that has cost businesses real traffic.
This guide shows you how to create a robots.txt file from scratch, explains the syntax in plain English, walks through copy-paste examples, and shows how to test the result. You can also generate a valid file in seconds with the free Robots.txt Generator.
What is a robots.txt file?
A robots.txt file is a plain text file placed in the root of your website. It follows the Robots Exclusion Protocol, a widely supported standard that tells web crawlers which paths they may crawl.
It lives at:
https://yourdomain.com/robots.txt
For example, a file at https://example.com/robots.txt applies to the whole site. Crawlers check this file before crawling pages, so it is usually one of the first things they request.
What robots.txt can and cannot do
It is important to be clear about its limits:
- It can ask crawlers not to visit certain paths.
- It can point crawlers to your XML sitemap.
- It can set a crawl delay for bots that respect it.
- It cannot hide a page from search results if other sites link to it.
- It cannot protect private data. A disallowed page can still be indexed if it is linked elsewhere.
- It cannot enforce security. Use authentication and access controls for that.
Think of robots.txt as a polite request, not a lock. Most major crawlers honour it, but malicious bots may ignore it entirely.
Why robots.txt matters for SEO
A well-configured robots.txt helps search engines spend their crawl budget on pages that matter. Specifically, it can:
- Keep crawlers out of admin panels, checkout flows and internal search results.
- Prevent duplicate or thin pages from being crawled.
- Reduce crawling of staging or development sites.
- Point crawlers to your sitemap so new content is discovered faster.
Crawl budget is a real consideration for large sites. If a crawler wastes time on thousands of filter URLs or session parameters, it may reach your important pages more slowly. A sensible robots.txt reduces that waste.
The basic robots.txt syntax
A robots.txt file is made of simple field: value lines. The most common directives are:
| Directive | What it does |
|---|---|
User-agent | Which crawler the rules apply to (* means all) |
Disallow | Paths the crawler should not visit |
Allow | Paths the crawler may visit, even inside a disallowed folder |
Sitemap | The full URL of your XML sitemap |
Crawl-delay | Seconds to wait between requests (not honoured by Google) |
Rules are grouped into blocks. Each block starts with a User-agent line and is followed by one or more Allow or Disallow lines.
A minimal example
User-agent: *
Disallow:
This allows every crawler to visit everything. The empty Disallow line means "nothing is disallowed".
Blocking a single folder
User-agent: *
Disallow: /admin/
Disallow: /private/
This tells all crawlers to stay out of the admin and private folders.
Blocking everything
User-agent: *
Disallow: /
The single slash means the entire site. Use this only on staging or development sites, never on a live site you want indexed.
Allowing everything
User-agent: *
Allow: /
This is an explicit "crawl everything" instruction. It is functionally similar to an empty Disallow.
Adding a sitemap
User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml
The Sitemap directive is not tied to a user agent, but placing it at the end of the file is conventional and widely understood.
How to create a robots.txt file with a generator
Writing the file by hand is easy once you know the syntax, but a generator removes the risk of typos and ordering mistakes. Here is how to use the FreeTulz Robots.txt Generator:
- Open the Robots.txt Generator.
- Choose a default policy: allow all or block all.
- Select the user agent you want to target: all robots, Googlebot only, or Bingbot only.
- Enter any paths to disallow, one per line, such as
/admin/and/private/. - Optionally add a crawl delay in seconds.
- Paste your XML sitemap URL so crawlers can find it.
- Click Generate Robots.txt File.
- Copy the generated content and upload it to the root of your site.
The tool produces a correctly formatted file, so you can focus on deciding what to allow rather than remembering the exact syntax.
Where to put the file
The file must be at the root of your domain. The exact method depends on your platform:
- WordPress: Many SEO plugins create and manage robots.txt for you. If you prefer to edit it manually, you can upload a file via your host's file manager or FTP.
- Static sites: Add
robots.txtto the root folder that is served as your site's public directory. - Hosting panels: Use the file manager to upload the file to
public_htmlor the equivalent web root. - Frameworks: Place the file in the framework's public or static directory so it is served at
/robots.txt.
Always confirm the file is reachable by visiting https://yourdomain.com/robots.txt in a browser.
Copy-paste robots.txt examples
WordPress-style setup
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Sitemap: https://example.com/sitemap.xml
This blocks the admin area but allows the AJAX endpoint that some site features need, and it discourages crawling of internal search pages.
E-commerce site
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?orderby=
Disallow: /*?filter=
Sitemap: https://example.com/sitemap.xml
This keeps transactional pages and filter URLs from wasting crawl budget while leaving product and category pages open.
Staging site
User-agent: *
Disallow: /
Use this to discourage compliant crawlers from visiting a staging site. It does not guarantee exclusion from search results; restrict access with authentication. Remove the rule when the site goes live.
Block a single bot
User-agent: BadBot
Disallow: /
User-agent: *
Allow: /
This blocks one named crawler while allowing everyone else. Note that bots can ignore this, so do not rely on it for security.
Common robots.txt mistakes
Even experienced site owners make these errors:
- Accidentally blocking the whole site. A stray
Disallow: /on a live site can deindex pages. Check the file after every change. - Blocking CSS and JavaScript. Google needs these resources to render pages properly. Blocking them can hurt how your pages are evaluated.
- Using robots.txt for private data. It does not prevent access. Use authentication instead.
- Forgetting the sitemap. Adding a
Sitemapline helps discovery with no downside. - Editing the wrong file. Some platforms generate robots.txt dynamically. Make sure your changes are actually being served.
- Ignoring case sensitivity. Paths are case-sensitive on many servers, so
/Admin/and/admin/are different.
How to test your robots.txt file
After uploading, verify that it works:
- Visit
https://yourdomain.com/robots.txtand confirm the content appears. - Use Google Search Console's robots.txt report to check for syntax errors and see how Google interprets the file.
- Test specific URLs to confirm that important pages are allowed and private paths are blocked.
- Recheck after any site migration or platform change.
A file that looks correct but is served with the wrong content type or from the wrong path will be ignored.
Frequently asked questions
Is the Robots.txt Generator free?
Yes. The FreeTulz Robots.txt Generator is free and requires no account.
Does robots.txt improve rankings directly?
Not directly. It helps crawlers spend time on the right pages and can prevent duplicate content from being crawled, which supports SEO indirectly.
Can robots.txt hide a page from Google?
It can stop crawlers from visiting a page, but the URL may still appear in results if it is linked elsewhere. Use a noindex directive for that purpose and allow crawlers to access the page so they can see it.
Should I block admin pages?
Yes, blocking admin and account areas is a common and sensible use of robots.txt.
What happens if I use Disallow: / by mistake?
Crawlers may stop visiting your site and pages can drop out of search results over time. Remove the line and request a recrawl as soon as possible.
Do all search engines follow robots.txt?
Major engines such as Google and Bing do. Malicious bots may ignore it, so never use robots.txt as a security measure.
Final thoughts
A robots.txt file is small but powerful. It guides crawlers, protects crawl budget and connects search engines to your sitemap. The syntax is simple, but the consequences of a mistake can be serious, so generate the file carefully, test it, and review it whenever your site structure changes.
Use the Robots.txt Generator to produce a clean starting point, then confirm the result in Google Search Console. For the next step, pair it with a Sitemap Generator so crawlers can discover your pages efficiently.

