itic.org
robots.txt

Groups

ahrefsbot

semrushbot

mj12bot

newspaper/0.2.8

dotbot

sogou web spider

petalbot

ltx71 - (http://ltx71.com/)

mozilla/5.0 (compatible; ahrefsbot/7.0; +http://ahrefs.com/robot/)

mozilla/5.0 (compatible; semrushbot/7~bl; +http://www.semrush.com/bot.html)

siteauditbot

semrushbot-ba

semrushbot-si

semrushbot-swa

semrushbot-ct

semrushbot-bm

splitsignalbot

semrushbot-coub

mozilla/5.0 (compatible; dubbotbot/0.2; +http://dubbot.com)

applebot

mozilla/5.0 (compatible; mj12bot/v1.4.8; http://mj12bot.com/)

blp_bbot/0.1

mozilla/5.0 (compatible; linux x86_64; mail.ru_bot/2.0; +http://go.mail.ru/help/robots)

sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm

eyemonit_bot_version_0.1_(http://www.eyemon.it/)

mozilla/5.0 (compatible; blexbot/1.0; +http://webmeup-crawler.com/)

mozilla/5.0 (compatible; barkrowler/0.9; +https://babbar.tech/crawler)

mozilla/5.0 (compatible; dataforseobot/1.0; +https://dataforseo.com/dataforseo-bot)

mozilla/5.0 (compatible; adsbot/3.1; +https://seostar.co/robot/)

mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)

mozilla/5.0 (compatible;petalbot;+https://webmaster.petalsearch.com/site/petalbot)

mozilla/5.0 (compatible; neevabot/1.0; +https://neeva.com/neevabot)

mozilla/5.0 (compatible; mojeekbot/0.10; +https://www.mojeek.com/bot.html)

mozilla/5.0 (compatible; coccocbot-image/1.0; +http://help.coccoc.com/searchengine)

zoombot (linkbot 1.0 http://suite.seozoom.it/bot.html)

mozilla/5.0 (compatible; megaindex.ru/2.0; +http://megaindex.com/crawler)

pyspider/0.3.10 (+http://pyspider.org/)

mozilla/5.0 (compatible; dotbot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)

academicbotrtu (https://academicbot.rtu.lv; mailto:caps@rtu.lv)

mauibot (crawler.feedback+gamma@gmail.com)

the knowledge ai

inetdex-bot/1.5 (mozilla/5.0; https://inetdex.com/; info at inetdex dot com)

screaming frog seo spider/16.6

mozilla/5.0 (compatible; seznambot/3.2-test1; +http://napoveda.seznam.cz/en/seznambot-intro/)

mozilla/5.0 (compatible; linespider/1.1; +https://lin.ee/4dwxkth)

zoominfobot (zoominfobot at zoominfo dot com)

mozilla/5.0 (compatible; qwantify/2.4w; +https://www.qwant.com/)

screaming frog seo spider/16.7

mozilla/5.0 (compatible; mojeekbot/0.11; +https://www.mojeek.com/bot.html)

mozilla/5.0 (compatible; seznambot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)

awariosmartbot/1.0 (+https://awario.com/bots.html; bots@awario.com)

mozilla/5.0 (compatible; siteauditbot/0.97; +http://www.semrush.com/bot.html)

legistorm bot (http://www.legistorm.com/legibot.html)

mozilla/5.0 (compatible; infotigerbot/1.9; +https://infotiger.com/bot)

mozilla/5.0 (compatible; baiduspider/2.0; +http://www.baidu.com/search/spider.html)

ccbot/2.0 (https://commoncrawl.org/faq/)

buck/2.3.2; (+https://app.hypefactors.com/media-monitoring/about.html)

turnitin (https://bit.ly/2uvnfoq)

turnitinbot

awariosmartbot/1.0 (+https://awario.com/bots.html; bots@awario.com)

awariorssbot/1.0 (+https://awario.com/bots.html; bots@awario.com)

mozilla/5.0 (compatible; obot/2.3.1; http://www.xforce-security.com/crawler/)

mozilla/5.0 (compatible) semanticscholarbot (+https://www.semanticscholar.org/crawler)

mozilla/5.0 (compatible; coccocbot-web/1.0; +http://help.coccoc.com/searchengine)

expanse, a palo alto networks company, searches across the global ipv4 space multiple times per day to identify customers&

screaming frog seo spider/17.2

linabot

scrapy/1.7.4 (+https://scrapy.org)

python-requests/2.28.2

mozilla/5.0 (linux; android 5.0) applewebkit/537.36 (khtml, like gecko) mobile safari/537.36 (compatible; bytespider; https://zhanzhang.toutiao.com/)

mozilla/5.0 (windows nt 10.0; win64; x64; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) gecko/20170101 firefox/67.0

mozilla/5.0 (linux; android 5.0) applewebkit/537.36 (khtml, like gecko) mobile safari/537.36 (compatible; bytespider; https://zhanzhang.toutiao.com/)

mozilla/5.0 (linux; android 5.0) applewebkit/537.36 (khtml, like gecko) mobile safari/537.36 (compatible; bytespider; spider-feedback@bytedance.com)

screaming frog seo spider/18.4

screaming frog seo spider/19.0

mozilla/5.0 (compatible; semrushbot; +http://www.semrush.com/bot.html)

mozilla/5.0 (compatible; dataprovider.com)

mozilla/5.0 (compatible; seznambot/4.0; +http://napoveda.seznam.cz/seznambot-intro/)

mozilla/5.0 (compatible; dataforseobot/1.0; +https://dataforseo.com/dataforseo-bot)

gulper web bot 0.2.4 (www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/link/gulperbot)

mozilla/5.0 (compatible; paqlebot/2.0; +http://www.paqle.dk/about/paqlebot)

mediatoolkitbot (complaints@mediatoolkit.com)

mozilla/5.0 (macintosh; intel mac os x 10_10_1) applewebkit/600.2.5 (khtml, like gecko) version/8.0.2 safari/600.2.5 (amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)

claudebot

mozilla/5.0 (compatible; seekportbot; +https://bot.seekport.com)

mozilla/5.0 (compatible; yeti/1.1; +https://naver.me/spd)

mozilla/5.0 applewebkit/537.36 (khtml, like gecko; compatible; claudebot/1.0; +claudebot@anthropic.com)

mozilla/5.0 (compatible; openindexspider; +https://www.openindex.io/saas/about-our-spider/)

domcopbot (https://www.domcop.com/bot)

mozilla/5.0 (compatible; awariobot/1.0; +https://awario.com/bots.html)

mozilla/5.0 applewebkit/537.36 (khtml, like gecko; compatible; gptbot/1.0; +https://openai.com/gptbot)