cooperators.ca
robots.txt

Robots Exclusion Standard data for cooperators.ca

Resource Scan

Scan Details

Site Domain cooperators.ca
Base Domain cooperators.ca
Scan Status Ok
Last Scan2024-09-18T08:29:09+00:00
Next Scan 2024-10-18T08:29:09+00:00

Last Scan

Scanned2024-09-18T08:29:09+00:00
URL https://cooperators.ca/robots.txt
Redirect https://www.cooperators.ca/robots.txt
Redirect Domain www.cooperators.ca
Redirect Base cooperators.ca
Domain IPs 118.215.84.109
Redirect IPs 118.215.84.109
Response IP 104.76.131.206
Found Yes
Hash 6d2dc99d1254fc52429b811d934dbe36dda6c18ad27a413478914ebdd8f5f711
SimHash 6b5e1250c131

Groups

baiduspider
compspybot
curious george
cybeye.com
daumoa
docomo
exb language crawler
ezooms
flamingo_searchengine
genieo
genio
gsa-crawler
lexxebot
libcrawl
linkdex
lwnutch
magpie-crawler
openwebindex
rediffnewsbot
repparser
scanmine
screaming frog seo spider
seoengworldbot
shopwiki
showyoubot
sindice-site-manager
sogou
sogou spider
sosospider
wocbot
woriobot
yacybot
yandex
yeti
yolinkbot
youdaobot

Rule Path
Disallow /

*

Rule Path
Disallow /sitecore/
Disallow /shared/
Disallow /recreational-vehicles/
Disallow /mobile/
Disallow /nqb/
Disallow /libs/
Disallow /disable-feature/

Other Records

Field Value
sitemap https://www.cooperators.ca/sitemap.xml

Comments

  • Updated February 27 2024 base on content from local copy should be revisited
  • This NEEDS to be updated has lots of stuff we do not want crawled