Log In Sign Up

engravedrinks.com
robots.txt

Robots Exclusion Standard data for engravedrinks.com

Resource Scan

Scan Details

Site Domain engravedrinks.com
Base Domain engravedrinks.com
Scan Status Ok
Last Scan2026-02-27T01:48:15+00:00
Next Scan 2026-03-29T01:48:15+00:00

Last Scan

Scanned2026-02-27T01:48:15+00:00
URL https://engravedrinks.com/robots.txt
Redirect https://www.engravedrinks.com/robots.txt
Redirect Domain www.engravedrinks.com
Redirect Base engravedrinks.com
Domain IPs 172.66.40.245, 172.66.43.11
Redirect IPs 172.66.40.245, 172.66.43.11
Response IP 172.66.43.11
Found Yes
Hash c4670ec1d605474b428c46e672be9bc71a7e6e3c08210eee827e758535cc795c
SimHash 8e1dcb03cecb

Groups

*

Rule Path
Disallow /index.php/
Disallow /catalogsearch/
Disallow /checkout/
Disallow /customer/
Disallow /wishlist/
Disallow /multishipping/
Disallow /search/
Disallow /admin/
Disallow /dev/
Disallow /errors/
Disallow /lib/
Disallow /var/
Disallow /setup/
Disallow /pub/
Disallow /rest/
Disallow /graphql
Disallow /*?SID=
Disallow /*?___store=
Disallow /*?dir=
Disallow /*?order=
Disallow /*?p=
Disallow /*?q=
Disallow /*?limit=

Other Records

Field Value
crawl-delay 15

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

applebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 20

applebot-extended

Rule Path Comment
Disallow / opt-out of AI training only. Applebot will still crawl for search.

meta-externalagent

Rule Path
Disallow /catalogsearch/
Disallow /search/
Disallow /rest/
Disallow /graphql

facebookexternalhit

Rule Path
Allow /

facebot

Rule Path
Allow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.engravedrinks.com/sitemap.xml
sitemap https://www.engravedrinks.com/sitemap.xml
Back to top

Comments

  • Sitemap
  • Default for all bots
  • Magento API endpoints that can get hammered
  • Block common URL parameters that explode crawl space
  • A conservative default crawl delay for engines that respect it
  • Google (ignores crawl-delay; keep scope tight via disallows above)
  • Bing (respects crawl-delay + can be tuned in Bing Webmaster Tools)
  • Apple
  • (Optional but recommended) Opt out of Apple's AI training while still allowing Applebot search
  • See Applebot-Extended docs if you prefer this:
  • (Safe to remove if you want to allow AI training.)
  • refs: Apple confirms robots.txt is honoured and Applebot-Extended exists.
  • Meta (does NOT honour crawl-delay; restrict heavy areas explicitly)
  • Meta preview bots (usually light traffic; allow but still respect global disallows)
  • Block SEO crawlers & scrapers
Back to top