static.hypebeast.com
robots.txt

Robots Exclusion Standard data for static.hypebeast.com

Resource Scan

Scan Details

Site Domain static.hypebeast.com
Base Domain hypebeast.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-04-06T15:31:23+00:00
Next Scan 2024-07-05T15:31:23+00:00

Last Successful Scan

Scanned2023-12-09T08:29:24+00:00
URL https://static.hypebeast.com/robots.txt
Domain IPs 52.84.251.107, 52.84.251.116, 52.84.251.120, 52.84.251.93
Response IP 52.222.174.108
Found Yes
Hash 3bb048bc12dc837826d790681eb318aeaf183b25ad34e75789509e62f9123d00
SimHash 4634d5327c75

Groups

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

twitterbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

baiduspider

Rule Path
Allow /search/page/2
Allow /search/page/3
Disallow /search/page/

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

grapeshot

Rule Path
Disallow

*

Rule Path
Disallow /app.php
Disallow /pageviews.php
Disallow /wp-admin
Disallow /administration
Disallow /logout
Disallow /giveaway-form-submit
Disallow /form-submit
Disallow /bookmarks
Disallow /firebase-subscribe
Disallow /account
Disallow /api
Disallow /search-suggest
Disallow /amp-disqus-embed
Disallow /comments-section
Disallow /hypeindex/graph
Disallow /hypeindex/performance-graph
Disallow /forum/
Disallow /mailto
Disallow /next-posts
Disallow /.well-known
Disallow /zh/next-posts
Disallow /jp/next-posts
Disallow /embed
Disallow /_ajax
Disallow /_fragment
Allow /

rogerbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

linguee bot

Rule Path
Disallow /

domain re-animator bot

Rule Path
Disallow /

mauibot (crawler.feedback+wc@gmail.com)

Rule Path
Disallow /
Disallow /search
Disallow /search/

Comments

  • www.robotstxt.org/
  • www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449
  • blocking bad bots