jp.hypebeast.com
robots.txt

Robots Exclusion Standard data for jp.hypebeast.com

Resource Scan

Scan Details

Site Domain jp.hypebeast.com
Base Domain hypebeast.com
Scan Status Ok
Last Scan2024-05-05T13:16:34+00:00
Next Scan 2024-05-12T13:16:34+00:00

Last Scan

Scanned2024-05-05T13:16:34+00:00
URL https://jp.hypebeast.com/robots.txt
Domain IPs 18.155.68.11, 18.155.68.118, 18.155.68.27, 18.155.68.61
Response IP 18.155.68.118
Found Yes
Hash 8c0bb0acfa4537d5276065aa19d83f4f4c37c20ab8fea7a470a679d860d59511
SimHash 0234d130ec77

Groups

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

twitterbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

baiduspider

Rule Path
Allow /search/page/2
Allow /search/page/3
Disallow /search/page/

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

grapeshot

Rule Path
Disallow

*

Rule Path
Disallow /app.php
Disallow /pageviews.php
Disallow /wp-admin
Disallow /administration
Disallow /logout
Disallow /giveaway-form-submit
Disallow /form-submit
Disallow /bookmarks
Disallow /firebase-subscribe
Disallow /account
Disallow /api
Disallow /search-suggest
Disallow /amp-disqus-embed
Disallow /comments-section
Disallow /hypeindex/graph
Disallow /hypeindex/performance-graph
Disallow /forum/
Disallow /mailto
Disallow /next-posts
Disallow /.well-known
Disallow /zh/next-posts
Disallow /jp/next-posts
Disallow /embed
Disallow /_ajax
Disallow /_fragment
Allow /

rogerbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

linguee bot

Rule Path
Disallow /

domain re-animator bot

Rule Path
Disallow /

mauibot (crawler.feedback+wc@gmail.com)

Rule Path
Disallow /
Disallow /search
Disallow /search/

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

Comments

  • www.robotstxt.org/
  • www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449
  • blocking bad bots
  • ChatGPT
  • Google Bard