AI 爬虫目录

名字说明用途；Cloudflare 的核验结果会单独显示，不把 UA 自报名当成身份验证。

返回看板

实时取页

  • ChatGPT-User chatgpt-user
  • Claude-User claude-user
  • Perplexity-User perplexity-user
  • MistralAI-User mistralai-user
  • DuckAssistBot duckassistbot
  • Meta-ExternalFetcher meta-externalfetcher
  • Google-Agent google-agent
  • Amzn-User amzn-user
  • Kimi-User kimi-user
  • GrokAgent grokagent

建索引

  • OAI-SearchBot oai-searchbot
  • Claude-SearchBot claude-searchbot
  • PerplexityBot perplexitybot
  • MistralAI-Index mistralai-index
  • Applebot applebot
  • Amzn-SearchBot amzn-searchbot
  • YouBot youbot
  • Meta-WebIndexer meta-webindexer

训练抓取

  • GPTBot gptbot
  • ClaudeBot claudebot
  • Bytespider bytespider
  • Meta-ExternalAgent meta-externalagent
  • CCBot ccbot
  • Amazonbot amazonbot
  • Google-CloudVertexBot google-cloudvertexbot
  • MistralAI-Training mistralai-training
  • Cohere Training Crawler cohere-training-data-crawler