Log In Sign Up

canada.travel
robots.txt

Robots Exclusion Standard data for canada.travel

Resource Scan

Scan Details

Site Domain canada.travel
Base Domain canada.travel
Scan Status Ok
Last Scan2024-05-10T08:54:40+00:00
Next Scan 2024-06-09T08:54:40+00:00

Last Scan

Scanned2024-05-10T08:54:40+00:00
URL https://canada.travel/robots.txt
Redirect https://info.destinationcanada.com:443/robots.txt
Redirect Domain info.destinationcanada.com
Redirect Base destinationcanada.com
Domain IPs 142.44.217.176
Redirect IPs 2600:9000:2003:2000:3:b612:40c0:93a1, 2600:9000:2003:5c00:3:b612:40c0:93a1, 2600:9000:2003:7400:3:b612:40c0:93a1, 2600:9000:2003:7e00:3:b612:40c0:93a1, 2600:9000:2003:8800:3:b612:40c0:93a1, 2600:9000:2003:a000:3:b612:40c0:93a1, 2600:9000:2003:c800:3:b612:40c0:93a1, 2600:9000:2003:f600:3:b612:40c0:93a1, 52.84.229.11, 52.84.229.120, 52.84.229.123, 52.84.229.13
Response IP 52.84.229.11
Found Yes
Hash 4e5b2ebecbce24f61c0faa9bc065c80209c4d4bf080f1ea9205f7fa031d463be
SimHash e815d81365a0

Groups

*

Rule Path
Disallow

gsa-crawler

Rule Path
Disallow

akamai-sitesnapshot/*

Rule Path
Disallow
Back to top

Comments

  • To remove the staging sites from all user-agent from and prevent them crawling the staging sites, with the exceptions of the user-agents of ctc google search appliance and Akamai.
  • Allow the CTC's google search appliance to access the sites
  • Any empty value, indicates that all URLs can be retrieved.
  • Allow Akamai crawler to access the sites
  • Any empty value, indicates that all URLs can be retrieved.
Back to top