popbee.com
robots.txt

Robots Exclusion Standard data for popbee.com

Resource Scan

Scan Details

Site Domain popbee.com
Base Domain popbee.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-08-19T14:38:49+00:00
Next Scan 2024-10-18T14:38:49+00:00

Last Successful Scan

Scanned2024-06-21T14:36:45+00:00
URL https://popbee.com/robots.txt
Domain IPs 18.155.68.11, 18.155.68.118, 18.155.68.27, 18.155.68.61
Response IP 18.155.68.27
Found Yes
Hash 6efda1df2703401850e01985e134946f0468b0d47a3a9cb81155db9e7eb2aded
SimHash 0234d130fc75

Groups

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

twitterbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

baiduspider

Rule Path
Allow /search/page/2
Allow /search/page/3
Disallow /search/page/

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

grapeshot

Rule Path
Disallow

*

Rule Path
Disallow /app.php
Disallow /pageviews.php
Disallow /wp-admin
Disallow /administration
Disallow /logout
Disallow /giveaway-form-submit
Disallow /form-submit
Disallow /bookmarks
Disallow /firebase-subscribe
Disallow /account
Disallow /api
Disallow /search-suggest
Disallow /amp-disqus-embed
Disallow /comments-section
Disallow /hypeindex/graph
Disallow /hypeindex/performance-graph
Disallow /forum/
Disallow /mailto
Disallow /next-posts
Disallow /.well-known
Disallow /zh/next-posts
Disallow /jp/next-posts
Disallow /embed
Disallow /_ajax
Disallow /_fragment
Disallow /post-content
Allow /

rogerbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

linguee bot

Rule Path
Disallow /

domain re-animator bot

Rule Path
Disallow /

mauibot (crawler.feedback+wc@gmail.com)

Rule Path
Disallow /
Disallow /search
Disallow /search/

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

Comments

  • www.robotstxt.org/
  • www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449
  • blocking bad bots
  • ChatGPT
  • Google Bard