Log In Sign Up

nearbyph.com
robots.txt

Robots Exclusion Standard data for nearbyph.com

Resource Scan

Scan Details

Site Domain nearbyph.com
Base Domain nearbyph.com
Scan Status Ok
Last Scan2024-09-18T13:34:48+00:00
Next Scan 2024-09-25T13:34:48+00:00

Last Scan

Scanned2024-09-18T13:34:48+00:00
URL https://nearbyph.com/robots.txt
Domain IPs 104.21.33.175, 172.67.147.170, 2606:4700:3032::ac43:93aa, 2606:4700:3036::6815:21af
Response IP 172.67.147.170
Found Yes
Hash ee15abfe7aa6f469eb4d603ffe17151c1f95c8bcbb40139902c9156134fe84c4
SimHash 820ccb02a363

Groups

yandex
moget
ichiro
naverbot
yeti
baiduspider
baiduspider-video
baiduspider-image
sogou spider
youdaobot
semrushbot
semrushbot-sa
proximic
ahrefsbot
a6
admantx

Rule Path
Disallow /

*

Rule Path
Disallow /admin
Disallow /scraper
Disallow /user_guide
Disallow /click
Disallow /ajax
Back to top

Comments

  • The following spiders are considered aggressive and /or non-desirable
Back to top