minebittoken.com
robots.txt

Robots Exclusion Standard data for minebittoken.com

Resource Scan

Scan Details

Site Domain minebittoken.com
Base Domain minebittoken.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2025-11-18T22:44:43+00:00
Next Scan 2026-02-16T22:44:43+00:00

Last Successful Scan

Scanned2025-07-22T22:43:12+00:00
URL https://minebittoken.com/robots.txt
Domain IPs 104.21.69.111, 172.67.207.188, 2606:4700:3033::6815:456f, 2606:4700:3034::ac43:cfbc
Response IP 104.21.69.111
Found Yes
Hash eb501977cd034e920976b5b83caed2d22a24bb51eaee90a60df1486c57fc541f
SimHash 205a89d076b4

Groups

*

Rule Path
Disallow /home/
Disallow /process/
Disallow /bittoken/
Disallow /cron/
Disallow /PHPMailer/
Disallow /assets/
Disallow /register/
Disallow /auth/
Disallow /telegram/
Disallow /vendor/

Other Records

Field Value
crawl-delay 10

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

googlebot

Rule Path
Disallow

Comments

  • Prevent aggressive crawling
  • Block unnecessary directories
  • Block specific bots
  • Allow Googlebot full access (no crawl-delay for it)
  • Specify sitemap