Log In Sign Up

yelp.com.au
robots.txt

Robots Exclusion Standard data for yelp.com.au

Resource Scan

Scan Details

Site Domain yelp.com.au
Base Domain yelp.com.au
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-05-15T01:56:40+00:00
Next Scan 2024-07-14T01:56:40+00:00

Last Successful Scan

Scanned2024-03-08T17:01:58+00:00
URL https://yelp.com.au/robots.txt
Redirect https://www.yelp.com.au/robots.txt
Redirect Domain www.yelp.com.au
Redirect Base yelp.com.au
Domain IPs 108.157.254.105, 108.157.254.51, 108.157.254.78, 108.157.254.80
Redirect IPs 108.157.254.105, 108.157.254.51, 108.157.254.78, 108.157.254.80
Response IP 108.157.254.105
Found Yes
Hash c6d6fa0b891dd25dcd1b168fa273212436b709982e3af60a3e263d9a216891e9
SimHash ad17b531efde

Groups

*
bingbot
adsbot-google
adsbot-google-mobile
adsbot-google-mobile-apps
mediapartners-google

Rule Path
Disallow /
Back to top

Comments

  • By accessing Yelp's website you agree to Yelp's Terms of Service, available at
  • https://www.yelp.com/static?country=US&p=tos
  • If you would like to inquire about crawling Yelp, please contact us at
  • https://www.yelp.com/contact
  • As always, Asimov's Three Laws are in effect:
  • 1. A robot may not injure a human being or, through inaction, allow a human
  • being to come to harm.
  • 2. A robot must obey orders given it by human beings except where such
  • orders would conflict with the First Law.
  • 3. A robot must protect its own existence as long as such protection does
  • not conflict with the First or Second Law.
Back to top