popbee.com
robots.txt

Robots Exclusion Standard data for popbee.com

Resource Scan

Scan Details

Site Domain popbee.com
Base Domain popbee.com
Scan Status Ok
Last Scan2024-11-15T14:40:17+00:00
Next Scan 2024-11-22T14:40:17+00:00

Last Scan

Scanned2024-11-15T14:40:17+00:00
URL https://popbee.com/robots.txt
Domain IPs 13.35.210.113, 13.35.210.37, 13.35.210.7, 13.35.210.90
Response IP 13.35.210.113
Found Yes
Hash 3fc59664b34736efa1bac8495c53bad4628e2afe3253624697cf76089317a90d
SimHash 5634d5306c77

Groups

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

twitterbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

twitterbot/ 1.0
baiduspider

Rule Path
Allow /search/page/2
Allow /search/page/3
Disallow /search/page/

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

grapeshot

Rule Path
Disallow

*

Rule Path
Disallow /app.php
Disallow /pageviews.php
Disallow /wp-admin
Disallow /administration
Disallow /logout
Disallow /giveaway-form-submit
Disallow /form-submit
Disallow /bookmarks
Disallow /firebase-subscribe
Disallow /account
Disallow /api
Disallow /search-suggest
Disallow /amp-disqus-embed
Disallow /comments-section
Disallow /hypeindex/graph
Disallow /hypeindex/performance-graph
Disallow /forum/
Disallow /mailto
Disallow /next-posts
Disallow /.well-known
Disallow /zh/next-posts
Disallow /jp/next-posts
Disallow /embed
Disallow /_ajax
Disallow /_fragment
Disallow /post-content
Allow /

rogerbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

linguee bot

Rule Path
Disallow /

domain re-animator bot

Rule Path
Disallow /

mauibot (crawler.feedback+wc@gmail.com)

Rule Path
Disallow /
Disallow /search
Disallow /search/

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

mozilla/5.0 (compatible; echoboxbot/1.0; hash/w4mwnpbx£3mfabxokjrw; +http://www.echobox.com)

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

Comments

  • www.robotstxt.org/
  • www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449
  • blocking bad bots
  • Google Bard

Warnings

  • `create-delay` is not a known field.