TTULAKITO
SEO

Robots.txt Generator

Generate a simple robots.txt file with sitemap support.

All tools

Details

Generated output

User-Agent: *
Allow: /
Disallow: /admin
Sitemap: https://example.com/sitemap.xml

Frequently asked questions

Does robots.txt prevent a page from being indexed?
No. Disallow only blocks crawling, but a URL can still appear in search results if it is linked externally. Use a noindex meta tag or X-Robots-Tag header to keep pages out of the index.
Where should the robots.txt file live?
It must sit at the root of the host, for example example.com/robots.txt, and is scoped per subdomain and protocol. A robots.txt at www does not apply to the apex domain.
Should I list my sitemap inside robots.txt?
Yes. Adding a Sitemap: directive with the absolute URL helps crawlers discover it without manual submission, and you can list multiple sitemaps if you have separate ones for posts, pages, and images.