Log In Sign Up

discoverjamaica.com
robots.txt

Robots Exclusion Standard data for discoverjamaica.com

Resource Scan

Scan Details

Site Domain discoverjamaica.com
Base Domain discoverjamaica.com
Scan Status Ok
Last Scan2024-11-08T22:13:45+00:00
Next Scan 2024-11-15T22:13:45+00:00

Last Scan

Scanned2024-11-08T22:13:45+00:00
URL https://discoverjamaica.com/robots.txt
Domain IPs 162.251.80.39
Response IP 162.251.80.39
Found Yes
Hash a2c0a69f8d89003f5de67d3bad9eb2f79eb4b2f0682bbaa324e9f55f3f84e090
SimHash 4012de52d4db

Groups

*

Rule Path
Disallow /images/
Disallow /cgi-bin/
Disallow /cgi-gallery/
Disallow /cgi-store/
Disallow /cgi-local/
Disallow /cgi-aj/
Disallow /imanager/
Disallow /phpwork/
Disallow /yl/

googlebot-image

Rule Path
Disallow /images/
Disallow /yl/
Disallow /cgi-gallery/

googlebot

Rule Path
Disallow /images/
Disallow /imanager/
Disallow /phpwork/
Disallow /cgi-gallery/
Disallow /yl/

bingbot

Rule Path
Disallow /cgi-gallery/
Disallow /yl/

Other Records

Field Value
crawl-delay 1

ahrefsbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /
Back to top

Comments

  • Bing
  • Totally Block these
  • contacted
  • Crawl-Delay: 2
  • Slurp
  • Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
  • Control These bots
  • Baiduspider,AhrefsBot,dotbot
  • https://moz.com/researchtools/ose/dotbot
  • https://ahrefs.com/robot
Back to top