csimarket.com
robots.txt

Robots Exclusion Standard data for csimarket.com

Resource Scan

Scan Details

Site Domain csimarket.com
Base Domain csimarket.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-10-13T00:21:17+00:00
Next Scan 2024-12-12T00:21:17+00:00

Last Successful Scan

Scanned2024-08-14T14:11:28+00:00
URL https://csimarket.com/robots.txt
Domain IPs 144.217.77.115
Response IP 144.217.77.115
Found Yes
Hash f858ea9e62cfef24c3cabab2b2fa484bafe6683e6f22dc08c51f4171e73813d4
SimHash 5d04eb33ed02

Groups

*

Rule Path
Allow /
Disallow /xbrl/
Disallow /API/
Disallow /test/
Disallow /AutoComplete/
Disallow /.well-known/
Disallow /error/
Disallow /excel2sql/
Disallow /scripts/

baiduspider
baiduspider-video
baiduspider-image

Rule Path
Disallow /

amazonbot

Rule Path
Allow /

bytespider

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

ahrefsbot

Rule Path
Allow /

semrushbot

Rule Path
Allow /

yak

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

python-urllib/3.9

Rule Path
Disallow /

muckrack

Rule Path
Disallow /
Allow /news/

Other Records

Field Value
sitemap https://csimarket.com/sitemap_index.xml