Log In Sign Up

info.destinationcanada.com
robots.txt

Robots Exclusion Standard data for info.destinationcanada.com

Resource Scan

Scan Details

Site Domain info.destinationcanada.com
Base Domain destinationcanada.com
Scan Status Ok
Last Scan2024-11-06T16:06:00+00:00
Next Scan 2024-12-06T16:06:00+00:00

Last Scan

Scanned2024-11-06T16:06:00+00:00
URL https://info.destinationcanada.com/robots.txt
Domain IPs 34.111.187.154
Response IP 34.111.187.154
Found Yes
Hash 4e5b2ebecbce24f61c0faa9bc065c80209c4d4bf080f1ea9205f7fa031d463be
SimHash e815d81365a0

Groups

*

Rule Path
Disallow

gsa-crawler

Rule Path
Disallow

akamai-sitesnapshot/*

Rule Path
Disallow
Back to top

Comments

  • To remove the staging sites from all user-agent from and prevent them crawling the staging sites, with the exceptions of the user-agents of ctc google search appliance and Akamai.
  • Allow the CTC's google search appliance to access the sites
  • Any empty value, indicates that all URLs can be retrieved.
  • Allow Akamai crawler to access the sites
  • Any empty value, indicates that all URLs can be retrieved.
Back to top