sheerwhenwet.com
robots.txt

Groups

*

Other Records

mediapartners-google

slurp

yahoo pipes 1.0

mozilla/5.0 (compatible; macocu; +https://www.clarin.si/info/macocu-massive-collection-and-curation-of-monolingual-and-bilingual-data/)

macocu

mozilla/5.0 (compatible; dataforseobot/1.0; +https://dataforseo.com/dataforseo-bot)

dataforseobot

barkrowler

barkrowler

mozilla/5.0+(compatible;+barkrowler/0.9;++https://babbar.tech/crawler)

mozilla/5.0 (compatible; barkrowler/0.9; +https://babbar.tech/crawler)

mozilla/5.0+(macintosh;+intel+mac+os+x+10_10_1)+applewebkit/600.2.5+(khtml,+like+gecko)+version/8.0.2+safari/600.2.5+(amazonbot/0.1;++https://developer.amazon.com/support/amazonbot)

mozilla/5.0 (macintosh; intel mac os x 10_15_5) applewebkit/605.1.15 (khtml, like gecko) version/13.1.1 safari/605.1.15 (applebot/0.1; +http://www.apple.com/go/applebot)

rollcrawl

baiduspider

yandexbot

sogou web spider

junglee bot

pinterestbot

mj12bot

semrushbot-bm

semrushbot-sa

ahrefsbot

naverbot

aspiegelbot

semrushbot

mozilla/5.0 (compatible; megaindex.ru/2.0; +http://megaindex.com/crawler)

bingbot

mozilla/5.0+(compatible;+megaindex.ru/2.0;++http://megaindex.com/crawler)

megaindex.ru/2.0

megaindex.ru

megaindex.ru

ccbot

googlebot

Other Records

Comments

• disallow adsense bot, as we no longer do adsense.
• Yahoo bot is evil.
• Yahoo Pipes is for feeds not web pages.
• disallow barkrowler bot, as we no longer do barkrowler.
• disallow barkrowler bot, as we no longer do barkrowler.
• Block MegaIndex.ru