av19.tech
robots.txt

Robots Exclusion Standard data for av19.tech

Resource Scan

Scan Details

Site Domain av19.tech
Base Domain av19.tech
Scan Status Ok
Last Scan2026-02-27T00:45:00+00:00
Next Scan 2026-03-29T00:45:00+00:00

Last Scan

Scanned2026-02-27T00:45:00+00:00
URL https://av19.tech/robots.txt
Redirect https://av19.is/robots.txt
Redirect Domain av19.is
Redirect Base av19.is
Domain IPs 2.57.91.174, 2a02:4780:84:fa93:26e9:e052:2cb9:fcfd, 2a02:4780:85:d0a5:5d57:aa95:be71:2895, 84.32.84.235
Redirect IPs 104.21.8.66, 172.67.138.161, 2606:4700:3036::ac43:8aa1, 2606:4700:3037::6815:842
Response IP 172.67.138.161
Found Yes
Hash 9e0d22b62cc6f77f998f9d25ec5bdcad9128ffe252107e3bf41d78a3c6f5896d
SimHash 50904a00e575

Groups

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

yandex

Rule Path
Allow /

naverbot

Rule Path
Allow /

yeti

Rule Path
Allow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

*

Rule Path
Allow /

Comments

  • Allow all SEO crawlers
  • Block AI training crawlers
  • Default allow