hypebeast.cc
robots.txt

Robots Exclusion Standard data for hypebeast.cc

Resource Scan

Scan Details

Site Domain hypebeast.cc
Base Domain hypebeast.cc
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-05-26T08:10:30+00:00
Next Scan 2024-08-24T08:10:30+00:00

Last Successful Scan

Scanned2023-10-30T07:51:33+00:00
URL https://hypebeast.cc/robots.txt
Redirect https://hypebeast.cn/robots.txt
Redirect Domain hypebeast.cn
Redirect Base hypebeast.cn
Domain IPs 35.163.28.206, 44.240.35.18
Redirect IPs 18.155.68.11, 18.155.68.118, 18.155.68.27, 18.155.68.61
Response IP 99.84.108.98
Found Yes
Hash 3bb048bc12dc837826d790681eb318aeaf183b25ad34e75789509e62f9123d00
SimHash 4634d5327c75

Groups

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

twitterbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

baiduspider

Rule Path
Allow /search/page/2
Allow /search/page/3
Disallow /search/page/

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

grapeshot

Rule Path
Disallow

*

Rule Path
Disallow /app.php
Disallow /pageviews.php
Disallow /wp-admin
Disallow /administration
Disallow /logout
Disallow /giveaway-form-submit
Disallow /form-submit
Disallow /bookmarks
Disallow /firebase-subscribe
Disallow /account
Disallow /api
Disallow /search-suggest
Disallow /amp-disqus-embed
Disallow /comments-section
Disallow /hypeindex/graph
Disallow /hypeindex/performance-graph
Disallow /forum/
Disallow /mailto
Disallow /next-posts
Disallow /.well-known
Disallow /zh/next-posts
Disallow /jp/next-posts
Disallow /embed
Disallow /_ajax
Disallow /_fragment
Allow /

rogerbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

linguee bot

Rule Path
Disallow /

domain re-animator bot

Rule Path
Disallow /

mauibot (crawler.feedback+wc@gmail.com)

Rule Path
Disallow /
Disallow /search
Disallow /search/

Comments

  • www.robotstxt.org/
  • www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449
  • blocking bad bots