Ship It Checklist

Checklists /Cloudflare /SEO and AI visibility

Medium

Publish a sitemap and allow AI crawlers on purpose

If robots.txt is missing, you have not decided what may be crawled. If it blocks GPTBot, ClaudeBot and similar agents by habit, chat tools cannot read the public pages you want them to cite. Public marketing pages should be allowed. Signed-in app URLs should not be listed in the sitemap.

How to check

  1. Open /robots.txt on the production host. Public pages should be allowed.
  2. The file should not disallow GPTBot, ClaudeBot, PerplexityBot or Google-Extended unless you have a reason.
  3. Open /sitemap.xml and check that only public URLs appear.
  4. Add a plain /llms.txt if you want a short map for chat tools.

Prompt

Fix prompt
Paste into Cursor or Claude Code
You are working in this repository. Do not add features.

Goal: /robots.txt allows general crawling and named AI crawlers, and /sitemap.xml lists public pages only.

Steps:
1. Add robots.txt with Allow rules and a Sitemap line using the production origin.
2. Explicitly allow GPTBot, ChatGPT-User, OAI-SearchBot, Google-Extended, PerplexityBot, ClaudeBot, anthropic-ai and Applebot-Extended.
3. Add a sitemap of public URLs. Leave out account pages.
4. Do not invent traffic numbers in the copy.
5. Show me the files.

This uses the same ticks as the checklist page, saved in this browser.