beyondeast.com
robots.txt

Robots Exclusion Standard data for beyondeast.com

Resource Scan

Scan Details

Site Domain beyondeast.com
Base Domain beyondeast.com
Scan Status Ok
Last Scan2025-11-27T05:53:49+00:00
Next Scan 2025-12-11T05:53:49+00:00

Last Scan

Scanned2025-11-27T05:53:49+00:00
URL https://beyondeast.com/robots.txt
Redirect https://www.beyondeast.com/robots.txt
Redirect Domain www.beyondeast.com
Redirect Base beyondeast.com
Domain IPs 23.227.38.32
Redirect IPs 23.227.38.74, 2620:127:f00f:e::
Response IP 23.227.38.74
Found Yes
Hash 7ab09888fd945c291e59529e18dfb0e3d94372e3355fbdff4aea4e019de502c5
SimHash 665d1872f5c9

Groups

googlebot

Rule Path
Disallow

googlebot-image

Rule Path
Disallow

bingbot

Rule Path
Disallow /admin
Disallow /cart
Disallow /carts
Disallow /orders
Disallow /account
Disallow /checkouts

Other Records

Field Value
crawl-delay 5

applebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

adsbot-google

Rule Path
Disallow /cart
Disallow /checkouts
Disallow /orders
Disallow /account
Disallow /*?*utm_
Disallow /preview_theme_id
Disallow /preview_script_id

gptbot

Rule Path
Allow /

google-extended

Rule Path
Allow /

anthropicbot

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

grokbot

Rule Path
Allow /

ccbot

Rule Path
Allow /

*

Rule Path
Disallow /admin
Disallow /cart
Disallow /carts
Disallow /orders
Disallow /account
Disallow /checkouts
Disallow /checkout
Disallow /.well-known/shopify/monorail
Disallow /wpm%40*
Disallow /custom/web-pixel-shopify-custom-pixel*
Disallow /collections/sort_by
Disallow /collections/%2B
Disallow /collections/%2B
Disallow /collections/filter
Disallow /collections/tag%3D
Disallow /blogs/%2B
Disallow /blogs/%2B
Disallow /*?*utm_
Disallow /*?variant=
Disallow /*?pr_prod_strat=
Disallow /*?pr_rec_id=
Disallow /*?pr_rec_pid=
Disallow /*?pr_ref_pid=
Disallow /*?pr_seq=
Disallow /*?srsltid=
Disallow /*?start=
Disallow /*?sz=
Disallow /*?cgid=
Disallow /pmin
Disallow /pmax
Disallow /prefn1
Disallow /prefv1
Disallow /srule
Disallow /selectedUrl
Disallow /search?q=
Disallow /en-us/
Disallow /search
Disallow /search-result-page
Disallow /*.atom$

facebookexternalhit/1.1

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 0

facebookexternalhit/1.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 15

facebookexternalhit/*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 15

ahrefsbot

Rule Path
Disallow /admin
Disallow /cart
Disallow /checkouts
Disallow /account

Other Records

Field Value
crawl-delay 10

mj12bot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

exabot

Rule Path
Disallow /

trovitbot

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linko

Rule Path
Disallow /

httrack

Rule Path
Disallow /

xenu

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

zumbot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.beyondeast.com/sitemap.xml

Comments

  • ===== Sitemap =====
  • ===== Allow Major Search Bots =====
  • ===== Allow AI Bots =====
  • ===== All Other Bots =====
  • --- Duplicate/Thin URLs (filters, sorting, pagination) ---
  • --- Parameter-based Duplicate URLs ---
  • --- Locale duplication ---
  • --- Search Pages ---
  • --- Atom Feeds ---
  • ===== Throttle or Block Specific Bots =====
  • ===== Block Known Bad Bots =====
  • ===== Nutch Crawler =====