Log In Sign Up

biz.yelp.com
robots.txt

Robots Exclusion Standard data for biz.yelp.com

Resource Scan

Scan Details

Site Domain biz.yelp.com
Base Domain yelp.com
Scan Status Ok
Last Scan2024-09-22T17:42:44+00:00
Next Scan 2024-10-06T17:42:44+00:00

Last Scan

Scanned2024-09-22T17:42:44+00:00
URL https://biz.yelp.com/robots.txt
Domain IPs 151.101.0.116, 151.101.128.116, 151.101.192.116, 151.101.64.116
Response IP 199.232.44.116
Found Yes
Hash 1f38f144e2bdb43819643fa346afa7b5cff621b46edd63b017f45ce01bd4a178
SimHash 181537716ddd

Groups

adsbot-google
adsbot-google-mobile

Rule Path
Allow /?*campaign=
Allow /signup/?*campaign=
Allow /landing/
Disallow /ad_syndication_user_tracking
Disallow /block_notifications
Disallow /category_suggest/json
Disallow /forgot
Disallow /home
Disallow /location_suggest
Disallow /login/add_owner_comment?
Disallow /preview
Disallow /r2r
Disallow /signup_phone_confirm
Disallow /clover
Disallow /signup_business
Disallow /sit_rep

applebot
bingpreview
deepcrawl
googlebot
googlebot-image
googlebot-mobile
mediapartners-google
stc-bot
scoutjet
searchmetricsbot
seznambot
telegrambot
twitterbot
yahoo! slurp
yandex
archive.org_bot
bingbot
ia_archiver
msnbot

Rule Path
Disallow /ad_syndication_user_tracking
Disallow /block_notifications
Disallow /category_suggest/json
Disallow /claim/*/verify/sms/wait
Disallow /claim/*/signup
Disallow /claim/*/*/verify/
Disallow /clover
Disallow /confirm_email/
Disallow /forgot
Disallow /home
Disallow /landing/
Disallow /location_suggest
Disallow /login/add_owner_comment?
Disallow /preview
Disallow /r2r
Disallow /signup/
Disallow /signup_business
Disallow /signup_phone_confirm
Disallow /sit_rep
Disallow /verify

*

Rule Path
Disallow /
Back to top

Comments

  • By accessing Yelp's website (© 2024) you agree to Yelp's Terms of Service, available at
  • https://www.yelp.com/static?country=US&p=tos
  • If you would like to inquire about crawling Yelp, please contact us at
  • https://www.yelp.com/contact
  • As always, Asimov's Three Laws are in effect:
  • 1. A robot may not injure a human being or, through inaction, allow a human
  • being to come to harm.
  • 2. A robot must obey orders given it by human beings except where such
  • orders would conflict with the First Law.
  • 3. A robot must protect its own existence as long as such protection does
  • not conflict with the First or Second Law.
Back to top