06/11/2024
Hereโs how to set up a robots.txt.
Key Directives in robots.txt
โข ๐ค User-agent: Specifies which bot the rules apply to (e.g., User-agent: Googlebot or User-agent: * for all bots).
โข ๐ซ Disallow: Blocks bots from accessing certain files or directories (e.g., Disallow: /private/).
โข โ
Allow: Grants access to specific files or folders within a disallowed directory (only in Google) (e.g., Allow: /public/file.html).
โข ๐บ๏ธ Sitemap: Points bots to your XML sitemap, helping them navigate your siteโs structure (e.g., Sitemap: https://example.com/sitemap.xml).
Example of a Basic robots.txt
๐ค User-agent: *
๐ซ Disallow: /private/
โ
Allow: /public/
๐บ๏ธ Sitemap: https://example.com/sitemap.xml
This file helps search engines know what to crawl and index on your website! ๐
๐ค User-agent: *
๐ซ Disallow: /private/
โ
Allow: /public/
๐บ๏ธ Sitemap: https://example.com/sitemap.xml