Log In Sign Up

canadianunderwriter.ca
robots.txt

Robots Exclusion Standard data for canadianunderwriter.ca

Resource Scan

Scan Details

Site Domain canadianunderwriter.ca
Base Domain canadianunderwriter.ca
Scan Status Ok
Last Scan2024-10-28T12:14:40+00:00
Next Scan 2024-11-27T12:14:40+00:00

Last Scan

Scanned2024-10-28T12:14:40+00:00
URL https://canadianunderwriter.ca/robots.txt
Redirect https://www.canadianunderwriter.ca/robots.txt
Redirect Domain www.canadianunderwriter.ca
Redirect Base canadianunderwriter.ca
Domain IPs 2600:1f11:793:c400:b63d:2b6c:262f:48c1, 52.60.183.88
Redirect IPs 2600:1f11:793:c400:b63d:2b6c:262f:48c1, 52.60.183.88
Response IP 52.60.183.88
Found Yes
Hash c7f408a9bcdbad34180eb41848ebbf2342f1ffd0f1cdf5105589e4737a224f80
SimHash fc00111127b5

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /insurance/man-accused-of-ontario-healthcare-clinic-fraud-pleads-guilty-pays-50-000-to-auto-insurers-1002598744/
Disallow /news/man-arrested-charged-with-fraud-in-alleged-ontario-auto-insurance-related-scam/1002194172/
Allow /wp-admin/admin-ajax.php

amazonbot
anthropic-ai
applebot
awariorssbot
awariosmartbot
bytespider
ccbot
chatgpt-user
claudebot
claude-web
cohere-ai
dataforseobot
diffbot
facebookbot
friendlycrawler
googleother
img2dataset
imagesiftbot
magpie-crawler
meltwater
omgili
omgilibot
peer39_crawler
peer39_crawler/1.0
perplexitybot
piplbot
scoop.it
seekr
youbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.canadianunderwriter.ca/sitemap_index.xml
Back to top

Comments

  • Canadian Underwriter content is made available for your personal, non-commercial
  • use subject to our Terms of Service here:
  • https://www.newcom.ca/terms-of-use.
  • Use of any device, tool, or process designed to data mine or scrape the content
  • using automated means is prohibited without prior written permission from
  • Newcom Media Inc. Prohibited uses include but are not limited to:
  • (1) text and data mining activities under Art. 4 of the EU Directive on Copyright in
  • the Digital Single Market;
  • (2) the development of any software, machine learning, artificial intelligence (AI),
  • and/or large language models (LLMs);
  • (3) creating or providing archived or cached data sets containing our content to others; and/or
  • (4) any commercial purposes.
  • Contact https://www.canadianunderwriter.ca/contacts/ for assistance.
  • Disallow Rules
Back to top