Log In Sign Up

crainalerts.com
robots.txt

Robots Exclusion Standard data for crainalerts.com

Resource Scan

Scan Details

Site Domain crainalerts.com
Base Domain crainalerts.com
Scan Status Ok
Last Scan2025-09-26T09:45:58+00:00
Next Scan 2025-10-26T09:45:58+00:00

Last Scan

Scanned2025-09-26T09:45:58+00:00
URL https://crainalerts.com/robots.txt
Domain IPs 2600:9000:2795:4200:10:3b45:4980:93a1, 2600:9000:2795:8000:10:3b45:4980:93a1, 2600:9000:2795:9200:10:3b45:4980:93a1, 2600:9000:2795:a800:10:3b45:4980:93a1, 2600:9000:2795:c800:10:3b45:4980:93a1, 2600:9000:2795:ce00:10:3b45:4980:93a1, 2600:9000:2795:e00:10:3b45:4980:93a1, 2600:9000:2795:fe00:10:3b45:4980:93a1, 3.164.85.107, 3.164.85.113, 3.164.85.6, 3.164.85.67
Response IP 18.244.87.87
Found Yes
Hash e140411918b6734dafe222b9e2bf14d868c8f5cae8b80501e7cf5b5f9e1b67ad
SimHash 618888c0a096

Groups

*

Rule Path
Disallow

ccbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

gptbot
claudebot
claude-web
ccbot
applebot-extended
facebookbot
meta-externalagent
diffbot
perplexitybot
omgili
omgilibot
imagesiftbot
bytespider
amazonbot
youbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.crain.com/sitemap_index.xml
Back to top

Comments

  • START YOAST BLOCK
  • ---------------------------
  • ---------------------------
  • END YOAST BLOCK
  • START CRAIN BLOCK
  • ---------------------------
  • ---------------------------
  • END CRAIN BLOCK
Back to top