Log In Sign Up

sendegaro.com
robots.txt

Robots Exclusion Standard data for sendegaro.com

Resource Scan

Scan Details

Site Domain sendegaro.com
Base Domain sendegaro.com
Scan Status Ok
Last Scan2026-03-07T15:24:02+00:00
Next Scan 2026-03-21T15:24:02+00:00

Last Scan

Scanned2026-03-07T15:24:02+00:00
URL https://sendegaro.com/robots.txt
Domain IPs 104.26.4.131, 104.26.5.131, 172.67.72.232, 2606:4700:20::681a:483, 2606:4700:20::681a:583, 2606:4700:20::ac43:48e8
Response IP 172.67.72.232
Found Yes
Hash 456d84e8e8ad695d5216c4e4e6bd480b6ce37b28d16f41545b8642f78ca2a432
SimHash fd18e0124d79

Groups

*

Rule Path
Disallow /a/downloads/-/*
Disallow /admin
Disallow /cart
Disallow /orders
Disallow /checkouts/
Disallow /checkout
Disallow /25881542733/checkouts
Disallow /25881542733/orders
Disallow /carts
Disallow /account
Disallow /collections/*sort_by*
Disallow /*/collections/*sort_by*
Disallow /collections/*%2B*
Disallow /collections/*%2B*
Disallow /collections/*%2B*
Disallow /*/collections/*%2B*
Disallow /*/collections/*%2B*
Disallow /*/collections/*%2B*
Disallow */collections/*filter*%26*filter*
Disallow /blogs/*%2B*
Disallow /blogs/*%2B*
Disallow /blogs/*%2B*
Disallow /*/blogs/*%2B*
Disallow /*/blogs/*%2B*
Disallow /*/blogs/*%2B*
Disallow /*?*oseid=*
Disallow /*preview_theme_id*
Disallow /*preview_script_id*
Disallow /policies/
Disallow /*/policies/
Disallow /*/*?*ls=*&ls=*
Disallow /*/*?*ls%3D*%3Fls%3D*
Disallow /*/*?*ls%3D*%3Fls%3D*
Disallow /search
Disallow /apple-app-site-association
Disallow /.well-known/shopify/monorail
Disallow /cdn/wpm/*.js
Disallow /recommendations/products
Disallow /*/recommendations/products
Disallow /products/*-%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D-remote
Disallow /*/products/*-%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D-remote
Disallow /collections/*/products/*-%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D-remote
Disallow /*/collections/*/products/*-%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D-remote
Disallow /collections/all*
Disallow /collections/vendors?q=*
Disallow /collections/types?q=*
Disallow /collections/*?*constraint*
Disallow /collections/*?*filter*
Disallow /collections/*?*pf_*
Disallow /collections/*?*grid_list*
Disallow /collections/*?view=*
Disallow /collections/*?*view=*
Disallow /*/collections/all*
Disallow /*/collections/vendors?q=*
Disallow /*/collections/types?q=*
Disallow /*/collections/*?*constraint*
Disallow /*/collections/*?*filter*
Disallow /*/collections/*?*pf_*
Disallow /*/collections/*?*grid_list*
Disallow /*/collections/*?view=*
Disallow /*/collections/*?*view=*
Disallow /*?*sort_by=*
Disallow /search
Disallow /*/search
Disallow /*?*utm_*
Disallow /*?*fbclid=*
Disallow /*?*gclid=*
Disallow /*?*gbraid=*
Disallow /*?*wbraid=*
Disallow /*?*srsltid=*
Disallow /*?*preview_theme_id=*
Disallow /*?*section_id=*
Disallow /products/*?variant=*
Disallow /*?*variant=*

adsbot-google

Rule Path
Disallow /checkouts/
Disallow /checkout
Disallow /carts
Disallow /orders
Disallow /25881542733/checkouts
Disallow /25881542733/orders
Disallow /*?*oseid=*
Disallow /*preview_theme_id*
Disallow /*preview_script_id*
Disallow /cdn/wpm/*.js
Disallow /products/*-%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D-remote
Disallow /*/products/*-%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D-remote
Disallow /collections/*/products/*-%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D-remote
Disallow /*/collections/*/products/*-%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D-remote

nutch

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /a/downloads/-/*
Disallow /admin
Disallow /cart
Disallow /orders
Disallow /checkouts/
Disallow /checkout
Disallow /25881542733/checkouts
Disallow /25881542733/orders
Disallow /carts
Disallow /account
Disallow /collections/*sort_by*
Disallow /*/collections/*sort_by*
Disallow /collections/*%2B*
Disallow /collections/*%2B*
Disallow /collections/*%2B*
Disallow /*/collections/*%2B*
Disallow /*/collections/*%2B*
Disallow /*/collections/*%2B*
Disallow */collections/*filter*%26*filter*
Disallow /blogs/*%2B*
Disallow /blogs/*%2B*
Disallow /blogs/*%2B*
Disallow /*/blogs/*%2B*
Disallow /*/blogs/*%2B*
Disallow /*/blogs/*%2B*
Disallow /*?*oseid=*
Disallow /*preview_theme_id*
Disallow /*preview_script_id*
Disallow /policies/
Disallow /*/policies/
Disallow /*/*?*ls=*&ls=*
Disallow /*/*?*ls%3D*%3Fls%3D*
Disallow /*/*?*ls%3D*%3Fls%3D*
Disallow /search
Disallow /apple-app-site-association
Disallow /.well-known/shopify/monorail
Disallow /cdn/wpm/*.js
Disallow /products/*-%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D-remote
Disallow /*/products/*-%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D-remote
Disallow /collections/*/products/*-%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D-remote
Disallow /*/collections/*/products/*-%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D-remote

Other Records

Field Value
crawl-delay 10

ahrefssiteaudit

Rule Path
Disallow /a/downloads/-/*
Disallow /admin
Disallow /cart
Disallow /orders
Disallow /checkouts/
Disallow /checkout
Disallow /25881542733/checkouts
Disallow /25881542733/orders
Disallow /carts
Disallow /account
Disallow /collections/*sort_by*
Disallow /*/collections/*sort_by*
Disallow /collections/*%2B*
Disallow /collections/*%2B*
Disallow /collections/*%2B*
Disallow /*/collections/*%2B*
Disallow /*/collections/*%2B*
Disallow /*/collections/*%2B*
Disallow */collections/*filter*%26*filter*
Disallow /blogs/*%2B*
Disallow /blogs/*%2B*
Disallow /blogs/*%2B*
Disallow /*/blogs/*%2B*
Disallow /*/blogs/*%2B*
Disallow /*/blogs/*%2B*
Disallow /*?*oseid=*
Disallow /*preview_theme_id*
Disallow /*preview_script_id*
Disallow /policies/
Disallow /*/policies/
Disallow /*/*?*ls=*&ls=*
Disallow /*/*?*ls%3D*%3Fls%3D*
Disallow /*/*?*ls%3D*%3Fls%3D*
Disallow /search
Disallow /apple-app-site-association
Disallow /.well-known/shopify/monorail
Disallow /cdn/wpm/*.js
Disallow /products/*-%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D-remote
Disallow /*/products/*-%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D-remote
Disallow /collections/*/products/*-%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D-remote
Disallow /*/collections/*/products/*-%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D%5Ba-f0-9%5D-remote

Other Records

Field Value
crawl-delay 10

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

oai-searchbot

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

gptbot

Rule Path
Allow /

google-extended

Rule Path
Allow /

ahrefsbot

Rule Path
Disallow /

ahrefssiteaudit

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

googlebot

Rule Path
Allow /products/
Allow /products/*?variant=
Allow /collections/
Allow /cdn/
Allow /cdn/shop/
Allow /

googlebot-image

Rule Path
Allow /cdn/
Allow /cdn/shop/
Allow /

adsbot-google

Rule Path
Allow /products/
Allow /collections/
Allow /cdn/
Allow /cdn/shop/
Allow /
Back to top

Other Records

Field Value
sitemap https://sendegaro.com/sitemap.xml
sitemap https://sendegaro.com/sitemap.xml
sitemap https://sendegaro.com/sitemap.xml
Back to top

Comments

  • ----------- Custom rules for all crawlers -----------
  • Faceted / thin archives we don't want indexed (ROOT)
  • Faceted / thin archives we don't want indexed (MARKETS SUBFOLDERS)
  • Sorting params anywhere
  • Block site search results (root + markets)
  • Strip common tracking/noise parameters
  • Shopify section rendering duplicates
  • Variant-param duplicates (product pages)
  • ChatGPT / OpenAI crawlers
  • Note: ChatGPT-User is user-initiated and robots.txt may not apply,
  • but allowing doesn't hurt.
  • Optional: allow OpenAI training crawler too (only if you want that use-case)
  • Google Gemini training/grounding control token
  • (Does not affect Google Search ranking/inclusion)
  • Custom crawler rules – adjust to taste
  • Note: blocking these won't harm Google, but it disables some 3rd-party audits.
  • Explicit allowances for Google crawlers (Merchant Center & Images)
Back to top