marketsmith.investors.com
robots.txt

Robots Exclusion Standard data for marketsmith.investors.com

Resource Scan

Scan Details

Site Domain marketsmith.investors.com
Base Domain investors.com
Scan Status Ok
Last Scan2024-09-23T02:14:00+00:00
Next Scan 2024-10-07T02:14:00+00:00

Last Scan

Scanned2024-09-23T02:14:00+00:00
URL https://marketsmith.investors.com/robots.txt
Redirect https://marketsurge.investors.com/robots.txt
Redirect Domain marketsurge.investors.com
Redirect Base investors.com
Domain IPs 104.16.64.225, 104.18.159.173
Redirect IPs 104.16.64.225, 104.18.159.173
Response IP 104.18.159.173
Found Yes
Hash 09633123d38101acde489435a7291a91e7abbcd941da451ccaf3c512ad29f8d0
SimHash 100d744acc90

Groups

*

Rule Path
Disallow /Tool/
Disallow /Secure/
Disallow /Disclaimer.aspx
Disallow /404

blp_bbot

Rule Path
Disallow /

blp_bbot/0.1

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

newsnow

Rule Path
Disallow /

news-please

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

perplexity-ai

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://marketsurge.investors.com/sitemap.xml

Comments

  • robots.txt file for marketsurge.investors.com