Log In Sign Up

unglobalcompact.org
robots.txt

Robots Exclusion Standard data for unglobalcompact.org

Resource Scan

Scan Details

Site Domain unglobalcompact.org
Base Domain unglobalcompact.org
Scan Status Ok
Last Scan2024-11-05T17:16:57+00:00
Next Scan 2024-11-19T17:16:57+00:00

Last Scan

Scanned2024-11-05T17:16:57+00:00
URL https://unglobalcompact.org/robots.txt
Domain IPs 13.227.74.107, 13.227.74.49, 13.227.74.5, 13.227.74.55
Response IP 3.164.143.9
Found Yes
Hash a06d72ec813b331471d4075bf68a56bf60b616250ee684ddcd9461276b9321d5
SimHash 22847e816470

Groups

*

Rule Path
Disallow /COP/analyzing_progress/
Disallow /Cops/
Disallow /docs/
Disallow /news/
Disallow /participant/
Disallow /participants/
Disallow /participants/search
Disallow /search

Other Records

Field Value
crawl-delay 3

seokicks-robot

Rule Path
Disallow /

cfnetwork

Rule Path
Disallow /
Back to top

Comments

  • See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
  • To ban all spiders from the entire site uncomment the next two lines:
Back to top