Build a valid robots.txt file with custom bot rules, allow/disallow paths, crawl delay, and sitemap URL.
robots.txt is a plain text file placed at the root of your website (e.g. https://example.com/robots.txt) that tells web crawlers which pages they can or cannot access. Search engines read this file before crawling your site. It must be exactly at the root β not in a subfolder.
Disallowing a page in robots.txt prevents crawlers from visiting it, but it does not guarantee the page won't appear in search results. If other sites link to the disallowed page, search engines may still index the URL (without crawling the content). To guarantee removal, use a "noindex" meta tag or the HTTP X-Robots-Tag header instead.
The asterisk (*) is a wildcard that matches all bots. A rule under "User-agent: *" applies to every crawler that doesn't have its own specific block in the file. You can also target individual bots like Googlebot, Bingbot, or GPTBot by name.
Google does not honor the Crawl-delay directive in robots.txt. To control Googlebot's crawl rate, use Google Search Console's crawl rate settings instead. Bing and some other crawlers do respect Crawl-delay.
Yes. You can define separate blocks for different bots, each with their own allow/disallow rules. The most specific match wins. Bots process only the first matching group that applies to them.