Log In Sign Up

yellowbot.com
robots.txt

Robots Exclusion Standard data for yellowbot.com

Resource Scan

Scan Details

Site Domain yellowbot.com
Base Domain yellowbot.com
Scan Status Ok
Last Scan2025-08-11T10:09:25+00:00
Next Scan 2025-08-18T10:09:25+00:00

Last Scan

Scanned2025-08-11T10:09:25+00:00
URL https://yellowbot.com/robots.txt
Redirect https://www.yellowbot.com/robots.txt
Redirect Domain www.yellowbot.com
Redirect Base yellowbot.com
Domain IPs 104.21.86.47, 172.67.215.18, 2606:4700:3030::ac43:d712, 2606:4700:3036::6815:562f
Redirect IPs 104.21.86.47, 172.67.215.18, 2606:4700:3030::ac43:d712, 2606:4700:3036::6815:562f
Response IP 104.21.86.47
Found Yes
Hash 6d751b72b9ec57a64519e0c85cb5007f021360993aaa5793770d282b9b1ee0f9
SimHash 0c375f36f5d9

Groups

mediapartners-google

Rule Path
Disallow

*

Rule Path
Disallow /api/
Disallow /messages/
Disallow /invite
Disallow /signout
Disallow /signin
Disallow /pictures/upload
Disallow /owner/
Disallow /map/
Disallow /submit/
Disallow /call/
Disallow /nfredirect/
Disallow /merchant-verification
Disallow /search
Disallow /brands/
Disallow /css/
Disallow /user/
Disallow /*?inline=
Disallow /*?map=inline
Disallow /*?pictures=inline
Disallow /*?pictures=1
Disallow /*?videos=inline
Disallow /*?videos=1
Disallow /reviews-*

fasterfox

Rule Path
Disallow /

bender

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /
Back to top

Comments

  • YellowBot
  • these generally need logins, javascript or at least humans to do
  • something. It's pointless for the crawlers to crawl them...
  • http://www.google.com/support/webmasters/bin/answer.py?answer=35303
  • Googlebot supports wildcards - the have a tool in their "webmaster tools"
  • to check the rules (these work)
  • http://www.edochan.com/programming/pf.htm
  • http://sites.google.com/site/bendercrawler
  • http://ahrefs.com/robot/
Back to top