archinform.net
robots.txt

Robots Exclusion Standard data for archinform.net

Resource Scan

Scan Details

Site Domain archinform.net
Base Domain archinform.net
Scan Status Ok
Last Scan2024-10-11T04:14:10+00:00
Next Scan 2024-10-18T04:14:10+00:00

Last Scan

Scanned2024-10-11T04:14:10+00:00
URL https://archinform.net/robots.txt
Redirect https://www.archinform.net/robots.txt
Redirect Domain www.archinform.net
Redirect Base archinform.net
Domain IPs 62.67.209.19
Redirect IPs 62.67.209.19
Response IP 62.67.209.19
Found Yes
Hash 8146d8ba0f4e50f7c9c28ffb82cb7834719c9397c75f4a22d9d7a97401f14030
SimHash 926e7142c10b

Groups

ia_archiver

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider-video

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

bubing

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sogou inst spider

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yeti

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

yodaobot

Rule Path
Disallow /

zoombot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

moget

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

*

Rule Path
Disallow /cgi-bin/
Disallow /errors/
Disallow /asin/
Disallow /user/
Disallow /plugins/
Disallow /system/extlink.htm

Warnings

  • 2 invalid lines.