vlt.se
robots.txt

Groups

*

ccbot

chatgpt-user

gptbot

google-extended

google-cloudvertexbot

omgilibot

omgili

facebookbot

claudebot

diffbot

duckassistbot

perplexitybot

cohere-ai

cohere-training-data-crawler

meta-externalagent

meta-externalfetcher

timpibot

webzio-extended

youbot

amazonbot

bytespider

anthropic-ai

oai-searchbot

velenpublicwebcrawler

Comments

• Common Crawl robot, the resulting dataset is the primary training corpus in every LLM.
• ChatGPT robot, used to improve the ChatGPT LLM.
• ChatGPT robot, may be used to improve the ChatGPT LLM.
• Robot used to improve Bard and Vertex AI LLMs.
• Associated with Google Vertex AI agents
• webz.io robot, the resulting dataset can and is purchased to train LLMs.
• webz.io robot, the resulting dataset can and is purchased to train LLMs.
• FacebookBot crawls public web pages to improve LLMs for Facebook's speech recognition technology.
• Another agent used by Anthropic that is more specifically related to Claude
• Diffbot crawls the web in or others to train their LLMs.
• Uses scraped data on-the-fly to create answers for DuckAssist.
• Used by perplexity.ai. Generates text based on scraped material.
• Cohere’s chatbot.
• Cohere’s chatbot.
• Use cases such as training AI models or improving products by indexing content directly.
• Crawler performs user-initiated fetches of individual links in support of some AI tools.
• Used by Timpi to scrape data for training their Large Language Models.
• Used by Webz.io to indicate that your site should not be included those using it to train AI models.
• Crawler behind You.com’s AI search and browser assistant, indexing content for real-time answers.
• Amazonbot is used to train Amazon services such as Alexa.
• Bytespider is ByteDance's bot and may not respect robots.txt.
• Robot used to improve Anthropic AI LLMs.
• OpenAI search bot
• Velen.io/Hunter.io "build business datasets and machine learning models to better understand the web" - seems to focus on collecting email adresses for spam though.