foreignpolicy-com-staging-01.go-vip.net
robots.txt

Robots Exclusion Standard data for foreignpolicy-com-staging-01.go-vip.net

Resource Scan

Scan Details

Site Domain foreignpolicy-com-staging-01.go-vip.net
Base Domain go-vip.net
Scan Status Ok
Last Scan2024-09-12T02:04:39+00:00
Next Scan 2024-09-26T02:04:39+00:00

Last Scan

Scanned2024-09-12T02:04:39+00:00
URL https://foreignpolicy-com-staging-01.go-vip.net/robots.txt
Domain IPs 192.0.66.65, 2a04:fa87:fffd::c000:4241
Response IP 192.0.66.65
Found Yes
Hash be4fd05fab1218e8fc6ffeb58c05deeb4c858140c62d990122547ca9b3d1643e
SimHash 0042d9c44374

Groups

gptbot
ccbot
anthropic-ai
google-extended

Rule Path
Disallow /

*

Rule Path
Disallow /*?*wp_login_redirect=
Disallow /*?*view_post_comments=
Disallow */feed/

*

Rule Path
Disallow /

Comments

  • LLM Bots
  • General Disallow
  • Crawling is blocked for go-vip.co and go-vip.net domains