Robots.txt Generator
Builds a robots.txt from rules: user-agent, allow, disallow and sitemap. Ready to copy into your webroot.
Your inputs are processed in your browser and are not transmitted to our servers. Note: third-party resources (e.g. advertising and analytics from Google/Cloudflare) and an optional PayPal donation link may transfer data when loading or when clicked. Browser extensions or plugins may be able to read content that is visible in the input fields.
The result will appear here …
Robots.txt Generator: create a robots.txt easily
The robots.txt is a small text file in the root directory of your website that tells search engines which areas they may crawl and which they may not. With the robots.txt generator you put together the user-agent, allow and disallow rules, and the sitemap address without memorizing the syntax. The result is a finished, copy-ready robots.txt.
You specify for which search engines (user-agent) the rules apply, which paths are allowed (allow) and which are blocked (disallow), and optionally enter the path to your sitemap. The generator produces a valid robots.txt from this, which you can place directly in the root directory of your website.
How the construction works
A robots.txt consists of rules that begin with the user-agent line and specify for which search robot they apply. This is followed by allow and disallow statements that define which paths may be crawled. The generator translates your entries into this structure and at the end produces the reference to the sitemap.
- Set the user-agent, for example all search engines with an asterisk.
- Enter paths as allow or disallow.
- Optionally add the address of the sitemap.
- Copy the finished robots.txt and place it in the webroot.
What it can and cannot do
The robots.txt generator suits the typical access rules of a website and makes the syntax understandable. It helps avoid accidentally blocked areas because you can see and check every rule. It is important to know: the robots.txt is a recommendation to cooperative search engines and not protection against unwanted access. Confidential content instead belongs behind a real access restriction, such as a login. Moreover, the robots.txt must not replace the HTML of pages: if you want to remove a page from the index, use a robots tag or noindex instead.
Frequently asked questions
What is a robots.txt?
The robots.txt is a text file in the root directory of a website. It tells search engines which directories or files they may crawl. It is a recommendation, not a technical block.
What does user-agent mean in the robots.txt?
The user-agent defines for which search robot the following rules apply. An asterisk as the user-agent stands for all search engines. This lets you treat individual robots differently.
What is the difference between allow and disallow?
Disallow blocks a path from search, allow explicitly permits it. A disallow rule prevents search engines from crawling and indexing the relevant area. Allow is mainly used to re-enable individual paths within a blocked area.
Should the sitemap be in the robots.txt?
Yes, that is recommended. A reference to the sitemap helps search engines find your important pages. The line begins with "Sitemap:" and contains the full URL of your sitemap file.
Does the robots.txt block unwanted access?
No. It is solely a recommendation to cooperative search engines and is not followed by malicious access. Instead, protect confidential content with a real access restriction such as a login.