User-agent: * Disallow: # AI CRAWLERS: ALL ALLOWED # OpenAI / ChatGPT User-agent: GPTBot Allow: / # OpenAI browsing via ChatGPT User-agent: ChatGPT-User Allow: / # Anthropic / Claude User-agent: ClaudeBot Allow: / # Anthropic / Claude API User-agent: Claude-Web Allow: / # Google Gemini / Bard User-agent: Google-Extended Allow: / # Meta AI User-agent: FacebookBot Allow: / # Meta AI (future) User-agent: meta-externalagent Allow: / # Apple User-agent: Applebot Allow: / # Apple Extended User-agent: Applebot-Extended Allow: / # Microsoft Copilot / Bing AI User-agent: Bingbot Allow: / # Microsoft Copilot User-agent: msnbot Allow: / # Cohere AI User-agent: cohere-ai Allow: / # Perplexity AI User-agent: PerplexityBot Allow: / # You.com User-agent: YouBot Allow: / # Common Crawl (used by many AI training datasets) User-agent: CCBot Allow: / # Diffbot (AI data extraction) User-agent: Diffbot Allow: / # Bytedance / TikTok AI User-agent: Bytespider Allow: / # Amazon Alexa User-agent: ia_archiver Allow: / # AI2 (Allen Institute) User-agent: AI2Bot Allow: / # Timpibot (AI search) User-agent: Timpibot Allow: / # Omgilibot (AI data) User-agent: omgilibot Allow: / Sitemap: https://www.urbanturbannc.com/sitemap.xml