Each answer engine reads the web a little differently. These are the specific requirements per engine.
ChatGPT
GPTBot, OAI-SearchBot, ChatGPT-User
ChatGPT Search pulls live pages through Bing style indexes and its own crawlers, then quotes the passage that answers the prompt most directly.
- •Allow GPTBot, OAI-SearchBot and ChatGPT-User in robots.txt
- •Put a two sentence answer directly under every question style heading
- •Publish an llms.txt that maps your key pages and pricing
- •Keep pricing, plan names and claims consistent across every page so the model repeats one version of the facts
Perplexity
PerplexityBot, Perplexity-User
Perplexity runs a live search for each query, reads the top results and cites the sources it quoted, so freshness and clean extractable structure win.
- •Allow PerplexityBot and Perplexity-User, and keep a current lastmod in your sitemap
- •Use short paragraphs, lists and comparison tables that lift cleanly into a citation
- •Add FAQPage and Product schema so facts and prices are machine readable
- •Update key pages regularly, Perplexity favors recently changed sources
Google Gemini and AI Overviews
Googlebot, Google-Extended
Gemini and AI Overviews are built on the Google index, so classic technical SEO still decides whether you are eligible, and structure decides whether you are quoted.
- •Allow Google-Extended so your content can be used in AI answers
- •Ship server rendered HTML, not client only content, so the crawler sees full text
- •Cover the whole question set for a topic on one page with clear headings
- •Use Organization, WebSite, Product, Article and FAQPage schema for entity clarity
Claude
ClaudeBot, Claude-Web, Claude-SearchBot, anthropic-ai
Claude web search fetches pages at answer time and prefers documents that read like reference material with plain language and no clutter.
- •Allow ClaudeBot, Claude-Web, Claude-SearchBot and anthropic-ai
- •Lead with the answer, then support it, never bury the conclusion
- •Name entities explicitly instead of using pronouns, so passages stand alone
- •Keep an llms.txt and an llms-full.txt so the model can read a clean version of your site