Log In Sign Up

berryhilldrip.com
robots.txt

Robots Exclusion Standard data for berryhilldrip.com

Resource Scan

Scan Details

Site Domain berryhilldrip.com
Base Domain berryhilldrip.com
Scan Status Ok
Last Scan2024-09-24T07:42:23+00:00
Next Scan 2024-10-24T07:42:23+00:00

Last Scan

Scanned2024-09-24T07:42:23+00:00
URL https://berryhilldrip.com/robots.txt
Redirect https://www.berryhilldrip.com/robots.txt
Redirect Domain www.berryhilldrip.com
Redirect Base berryhilldrip.com
Domain IPs 70.60.131.105
Redirect IPs 70.60.131.105
Response IP 70.60.131.105
Found Yes
Hash 1e9379f5a2bf7a6ce2cc635d005b8d9953c6a74e464bae72213354439e6b80b0
SimHash 8ad8dd00c3f0

Groups

googlebot

Rule Path
Disallow /*printable%3DY*
Disallow /*js%3D*
Disallow /*print_cat%3D*

Other Records

Field Value
crawl-delay 3

ahrefsbot

Rule Path
Disallow /

alexibot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

becomebot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

clickagy intelligence bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

httrack

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

omniexplorer_bot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

shim-crawler

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

tinytestbot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

xenu's link sleuth 1.1c

Rule Path
Disallow /

yandex

Rule Path
Disallow /

*

Rule Path
Disallow /login.php
Disallow /register.php
Disallow /cart.php
Disallow /*.php?xid_*=*

Other Records

Field Value
crawl-delay 10
Back to top

Comments

  • See: https://www.robotstxt.org/
  • Bots that can cause issues
  • http://webmeup-crawler.com
  • https://megaindex.com/crawler
  • Defaults
Back to top

Warnings

  • 2 invalid lines.
Back to top