Log In Sign Up

bitdefender.in
robots.txt

Robots Exclusion Standard data for bitdefender.in

Resource Scan

Scan Details

Site Domain bitdefender.in
Base Domain bitdefender.in
Scan Status Ok
Last Scan2025-12-30T16:41:11+00:00
Next Scan 2026-01-29T16:41:11+00:00

Last Scan

Scanned2025-12-30T16:41:11+00:00
URL https://bitdefender.in/robots.txt
Domain IPs 104.21.81.117, 172.67.159.208, 2606:4700:3030::6815:5175, 2606:4700:3035::ac43:9fd0
Response IP 104.21.81.117
Found Yes
Hash 5b006a9cf5911f25fb900a810dda59469a6790baa4ed8fa908eda88c8c8a6aa1
SimHash 107d702087c8

Groups

friendlycrawler
orbbot
phpcrawl
nutch
magpie-crawler
heritrix
indy library
go-http-client

Rule Path
Disallow /

ahrefsbot
mj12bot
blexbot
dotbot
zoominfobot
maxpointcrawler
dataforseobot
seoscanners.net

Rule Path
Disallow /

gptbot
claudebot
bytespider
petalbot
barkrowler

Rule Path
Disallow /

yandexbot
seznambot
mail.ru
coccocbot-image

Rule Path
Disallow /

amazonbot
admantx
buzzbot
trendictionbot
rogerbot
zumbot
alphabot
aspiegelbot

Rule Path
Disallow /

duckduckbot

Rule Path
Disallow /
Back to top

Comments

  • Aggressive crawlers & scrapers
  • SEO & data mining bots
  • User-agent: SemrushBot
  • AI / LLM crawlers
  • Regional / foreign search engines
  • Social / marketing crawlers
  • Privacy search engines
Back to top