User-agent: * Allow: / # AI answer engines — real-time, user-triggered retrieval User-agent: ChatGPT-User Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / # AI training crawlers — periodic ingestion for model training User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: anthropic-ai Allow: / User-agent: Claude-Web Allow: / User-agent: Google-Extended Allow: / User-agent: CCBot Allow: / User-agent: Bytespider Allow: / User-agent: meta-externalagent Allow: / User-agent: AI2Bot Allow: / User-agent: Diffbot Allow: / User-agent: YandexGPT Allow: / User-agent: MistralBot Allow: / User-agent: cohere-ai Allow: / User-agent: youBot Allow: / User-agent: NeevaAI Allow: / User-agent: Amazon Q Allow: / User-agent: Timpibot Allow: / User-agent: SBIntuitionsBot Allow: / # SEO scraper tools — unrelated to AI visibility, keep blocked User-agent: SemrushBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: PetalBot Disallow: / Sitemap: https://tgr.co/sitemap.xml Sitemap: https://tgr.co/sitemap.xml