globeandmail.ca
robots.txt

Robots Exclusion Standard data for globeandmail.ca

Resource Scan

Scan Details

Site Domain globeandmail.ca
Base Domain globeandmail.ca
Scan Status Ok
Last Scan2024-06-21T05:26:48+00:00
Next Scan 2024-06-28T05:26:48+00:00

Last Scan

Scanned2024-06-21T05:26:48+00:00
URL http://globeandmail.ca/robots.txt
Redirect https://www.theglobeandmail.com/robots.txt
Redirect Domain www.theglobeandmail.com
Redirect Base theglobeandmail.com
Domain IPs 199.198.138.250
Redirect IPs 23.45.207.207, 23.45.207.208, 2600:1413:a000::17d2:fa91, 2600:1413:a000::17d2:fab8
Response IP 42.99.140.153
Found Yes
Hash ba72f5ef7d92a7063e647da0be7462b9c32f885f95d94f72443cdf9ae77d02ce
SimHash 743c595ae481

Groups

googlebot-news

Rule Path
Disallow /feeds/
Disallow /incoming/
Disallow /test/
Disallow /partners/
Disallow /search/
Disallow /business/adv/appointmentnotices/search/

adsbot-google

Rule Path
Disallow /feeds/
Disallow /incoming/
Disallow /test/
Disallow /search/
Disallow /business/adv/appointmentnotices/search/

*

Rule Path
Disallow /feeds/
Disallow /incoming/
Disallow /test/
Disallow /search/
Disallow /business/adv/appointmentnotices/search/
Disallow /marketing-containers/
Disallow /coupons/

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

awariorssbot
awariosmartbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

newsnow

Rule Path
Disallow /

news-please

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

peer39_crawler
peer39_crawler/1.0

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.theglobeandmail.com/sitemap.xml?outputType=xml