foreignpolicy-com.go-vip.co
robots.txt

Robots Exclusion Standard data for foreignpolicy-com.go-vip.co

Resource Scan

Scan Details

Site Domain foreignpolicy-com.go-vip.co
Base Domain go-vip.co
Scan Status Ok
Last Scan2024-11-06T17:15:25+00:00
Next Scan 2024-11-20T17:15:25+00:00

Last Scan

Scanned2024-11-06T17:15:25+00:00
URL https://foreignpolicy-com.go-vip.co/robots.txt
Domain IPs 192.0.66.2
Response IP 192.0.66.2
Found Yes
Hash 360371aa19e2e3796b88092e63b1bbeed8fce7714f8fb3ba67b594ef343df52b
SimHash 484669c56376

Groups

gptbot
ccbot
anthropic-ai
google-extended

Rule Path
Disallow /

*

Rule Path
Disallow /*?*wp_login_redirect=
Disallow /*?*view_post_comments=
Disallow */feed/

*

Rule Path
Disallow /

Other Records

Field Value
sitemap https://foreignpolicy.com/sitemap.xml
sitemap https://foreignpolicy.com/news-sitemap.xml
sitemap https://foreignpolicy.com/sitemap.xml

Comments

  • LLM Bots
  • General Disallow
  • Crawling is blocked for go-vip.co and go-vip.net domains
  • Sitemap archive