globeandmail.ca
robots.txt

Robots Exclusion Standard data for globeandmail.ca

Resource Scan

Scan Details

Site Domain globeandmail.ca
Base Domain globeandmail.ca
Scan Status Ok
Last Scan2024-06-14T05:26:45+00:00
Next Scan 2024-06-21T05:26:45+00:00

Last Scan

Scanned2024-06-14T05:26:45+00:00
URL http://globeandmail.ca/robots.txt
Redirect https://www.theglobeandmail.com/robots.txt
Redirect Domain www.theglobeandmail.com
Redirect Base theglobeandmail.com
Domain IPs 199.198.138.250
Redirect IPs 23.55.39.156, 23.55.39.180, 2600:1413:b000:14::b857:c151, 2600:1413:b000:14::b857:c15b
Response IP 42.99.140.146
Found Yes
Hash df76993373568f2934914b67670be03dcd5af10864e15fde51a39f228bd9a7de
SimHash 886ed852e093

Groups

googlebot-news

Rule Path
Disallow /feeds/
Disallow /incoming/
Disallow /test/
Disallow /partners/
Disallow /search/
Disallow /business/adv/appointmentnotices/search/

adsbot-google

Rule Path
Disallow /feeds/
Disallow /incoming/
Disallow /test/
Disallow /search/
Disallow /business/adv/appointmentnotices/search/

*

Rule Path
Disallow /feeds/
Disallow /incoming/
Disallow /test/
Disallow /search/
Disallow /business/adv/appointmentnotices/search/
Disallow /marketing-containers/
Disallow /coupons/

gptbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.theglobeandmail.com/sitemap.xml?outputType=xml