canidev.com
robots.txt

Robots Exclusion Standard data for canidev.com

Resource Scan

Scan Details

Site Domain canidev.com
Base Domain canidev.com
Scan Status Ok
Last Scan2025-09-23T11:45:21+00:00
Next Scan 2025-09-30T11:45:21+00:00

Last Scan

Scanned2025-09-23T11:45:21+00:00
URL https://canidev.com/robots.txt
Domain IPs 85.215.157.91
Response IP 85.215.157.91
Found Yes
Hash f949767a7bdee36c46c83bbb86e7106ae2d674ec3c9220d0f4b9ae7346b1f5bc
SimHash 911ccda4ae12

Groups

amazonbot

Rule Path
Disallow /

baidu

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

*

Rule Path
Disallow /adm/
Disallow /cache/
Disallow /includes/
Disallow /language/
Disallow /common.php
Disallow /config.php
Disallow /mcp.php
Disallow /posting.php
Disallow /report.php
Disallow /*.html