strawberry.gay
robots.txt

Groups

*

*

Other Records

adsbot
adsbot-google
adsbot-google-mobile

*

peer39_crawler
peer39_crawler/1.0

turnitinbot

npbot

slysearch

blexbot

checkmarknetwork/1.0 (+https://www.checkmarknetwork.com/spider.html)

brandverity/1.0

piplbot

mj12bot

No rules defined. All paths allowed.

Other Records

chatgpt-user
gptbot

anthropic-ai
claude-web

claudebot

google-extended

facebookbot
meta-externalagent

cotoyogi

webzio-extended

img2dataset
omgili
omgilibot
timpibot
velenpublicwebcrawler
facebookexternalhit
icc-crawler
imagesiftbot
petalbot
scrapy
bytespider
amazonbot
diffbot
friendlycrawler
oai-searchbot
applebot-extended
academicbotrtu
kangaroo bot
semrushbot-ocob
semrushbot-ft
genai
webzio-extended
bigsur.ai
cohere-ai
youbot
perplexitybot

Comments

• Please see <https://seirdy.one/meta/scrapers-i-block/> for an explanation on almost every entry, including intentionally-excluded entries.
• Also, a lot of these are additional entries added by me.
• git.girlcock.ceo stuff
• from https://git.gay/gitgay/assets/src/branch/main/public/robots.txt
• I opt out of online advertising so malware that injects ads on my site won't
• get paid. You should do the same. my ads.txt file contains a standard
• placeholder to forbid any compliant ad networks from paying for ad placement
• on my domain.
• Enabling our crawler to access your site offers several significant benefits
• to you as a publisher. By allowing us access, you enable the maximum number
• of advertisers to confidently purchase advertising space on your pages. Our
• comprehensive data insights help advertisers understand the suitability and
• context of your content, ensuring that their ads align with your audience's
• interests and needs. This alignment leads to improved user experiences,
• increased engagement, and ultimately, higher revenue potential for your
• publication. (https://www.peer39.com/crawler-notice)
• --> fuck off.
• IP-violation scanners
• The next three are borrowed from https://www.videolan.org/robots.txt
• > This robot collects content from the Internet for the sole purpose of
• helping educational institutions prevent plagiarism. [...] we compare student
• papers against the content we find on the Internet to see if we # can find
• similarities. (http://www.turnitin.com/robot/crawlerinfo.html)
• --> fuck off.
• > NameProtect engages in crawling activity in search of a wide range of brand
• and other intellectual property violations that may be of interest to our
• clients. (http://www.nameprotect.com/botinfo.html)
• --> fuck off.
• iThenticate is a new service we have developed to combat the piracy of
• intellectual property and ensure the originality of written work for
• publishers, non-profit agencies, corporations, and newspapers.
• (http://www.slysearch.com/)
• --> fuck off.
• BLEXBot assists internet marketers to get information on the link structure
• of sites and their interlinking on the web, to avoid any technical and
• possible legal issues and improve overall online experience.
• (http://webmeup-crawler.com/)
• --> fuck off.
• Providing Intellectual Property professionals with superior brand protection
• services by artfully merging the latest technology with expert analysis.
• (https://www.checkmarknetwork.com/spider.html/)
• "The Internet is just way to big to effectively police alone." (ACTUAL quote)
• --> fuck off.
• Stop trademark violations and affiliate non-compliance in paid search.
• Automatically monitor your partner and affiliates’ online marketing to
• protect yourself from harmful brand violations and regulatory risks. We
• regularly crawl websites on behalf of our clients to ensure content
• compliance with brand and regulatory guidelines.
• (https://www.brandverity.com/why-is-brandverity-visiting-me)
• --> fuck off.
• Misc. icky stuff
• Pipl assembles online identity information from multiple independent sources
• to create the most complete picture of a digital identity and connect it to
• real people and their offline identity records. When all the fragments of
• online identity data are collected, connected, and corroborated, the result
• is a more trustworthy identity.
• --> fuck off.
• Well-known overly-aggressive bot that claims to respect robots.txt: http://mj12bot.com/
• Gen-AI data scrapers
• Eat shit, OpenAI.
• There isn't any public documentation for this AFAICT.
• Reuters thinks this works so I might as well give it a shot.
• Extremely aggressive crawling with no documentation. people had to email the
• company about this for robots.txt guidance.
• Official way to opt-out of Google's generative AI training:
• <https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers>
• FacebookBot crawls public web pages to improve language models for our speech
• recognition technology.
• <https://developers.facebook.com/docs/sharing/bot/?_fb_noscript=1>
• UPDATE: The Meta-ExternalAgent crawler crawls the web for use cases such as training AI models or improving products by indexing content directly.
• <https://developers.facebook.com/docs/sharing/webmasters/web-crawlers>
• This one doesn't support robots.txt: https://www.allenai.org/crawler
• block it with your reverse-proxy or WAF or something.
• See <https://ds.rois.ac.jp/center8/crawler/>
• Parent page says it builds LLMs in the infographic: <https://ds.rois.ac.jp/center8/>
• https://webz.io/bot.html
• Other AI/hostile shit