Log In Sign Up

cloudhi.io
robots.txt

Robots Exclusion Standard data for cloudhi.io

Resource Scan

Scan Details

Site Domain cloudhi.io
Base Domain cloudhi.io
Scan Status Ok
Last Scan2024-09-15T03:42:47+00:00
Next Scan 2024-10-15T03:42:47+00:00

Last Scan

Scanned2024-09-15T03:42:47+00:00
URL https://cloudhi.io/robots.txt
Domain IPs 108.156.133.107, 108.156.133.123, 108.156.133.51, 108.156.133.91
Response IP 108.156.133.51
Found Yes
Hash 8a31c4423ea75fcc590e643d57d6b143964c6f81d72c7464291a9550113df6a4
SimHash 78bec97283b1

Groups

*

Rule Path
Disallow

ahrefsbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 20

yandex

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 20

trovitbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 20

mj12bot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 20

baiduspider

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 20

blexbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 20

dotbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 20

ccbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 20
Back to top

Comments

  • Allow non-bad bots to index site.
  • Stop the following specific bots from scanning site and limit scanning to 20seconds intervals.
  • Need to add a site map.
Back to top