Log In Sign Up

hunting-store.eu
robots.txt

Robots Exclusion Standard data for hunting-store.eu

Resource Scan

Scan Details

Site Domain hunting-store.eu
Base Domain hunting-store.eu
Scan Status Ok
Last Scan2025-10-02T03:04:55+00:00
Next Scan 2025-11-01T03:04:55+00:00

Last Scan

Scanned2025-10-02T03:04:55+00:00
URL https://hunting-store.eu/robots.txt
Redirect https://www.hunting-trade.eu/robots.txt
Redirect Domain www.hunting-trade.eu
Redirect Base hunting-trade.eu
Domain IPs 104.26.10.66, 104.26.11.66, 172.67.68.39, 2606:4700:20::681a:a42, 2606:4700:20::681a:b42, 2606:4700:20::ac43:4427
Redirect IPs 104.21.47.108, 172.67.147.51, 2606:4700:3036::6815:2f6c, 2606:4700:3036::ac43:9333
Response IP 172.67.147.51
Found Yes
Hash 27b99b3dae5403d7758003680e15535a2eafd362be621a7c925dfa610df42d57
SimHash 5268c85b83fb

Groups

*

Rule Path Comment
Disallow /app/ -
Disallow /bin/ -
Disallow /dev/ -
Disallow /lib/ -
Disallow /phpserver/ -
Disallow /setup/ -
Disallow /update/ -
Disallow /generated/ -
Disallow /var/ -
Disallow /pub/static/adminhtml/ -
Disallow /pub/static/_requirejs/ -
Disallow /pub/static/frontend/ -
Disallow /pub/media/captcha/ -
Disallow /pub/media/downloadable/ -
Disallow /pub/media/import/ -
Disallow /pub/media/tmp/ -
Disallow /pub/media/catalog/category/cache/ -
Disallow /pub/media/catalog/product/cache/ -
Disallow /checkout/ -
Disallow /catalogsearch/ -
Disallow /customer/ -
Disallow /wishlist/ -
Disallow /review/ -
Disallow /sendfriend/ -
Disallow /downloadable/ -
Disallow /*?dir= -
Disallow /*?order= -
Disallow /*?mode= -
Disallow /*?price= -
Disallow /*?filter_ Catch any attribute-based filters
Disallow /cms/noroute/ -
Disallow /errors/ -
Allow /pub/static/frontend/**.css$ -
Allow /pub/static/frontend/**.js$ -
Allow /pub/media/catalog/product/ -
Allow /pub/media/wysiwyg/ -

Other Records

Field Value
crawl-delay 10

googlebot

Rule Path
Allow /pub/static/frontend/**.css$
Allow /pub/static/frontend/**.js$

Other Records

Field Value
crawl-delay 5

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.hunting-trade.eu/en/sitemap
sitemap https://www.hunting-trade.eu/us/sitemap
sitemap https://www.hunting-trade.eu/ca/sitemap
sitemap https://www.hunting-trade.eu/nz/sitemap
sitemap https://www.hunting-trade.eu/uk/sitemap
sitemap https://www.hunting-trade.eu/au/sitemap
sitemap https://www.hunting-trade.eu/ch/sitemap
sitemap https://www.hunting-trade.eu/int/sitemap
sitemap https://www.hunting-trade.eu/no/sitemap
sitemap https://www.hunting-trade.eu/is/sitemap
Back to top

Comments

  • Global directives for all crawlers
  • -----------------------------------------------------
  • A) Crawl-delay (optional; use only if server load is a concern)
  • Note: Google ignores this; Bing and others respect it.
  • Here we set 10 seconds between requests.
  • B) Disallow common Magento 2 system and backend paths
  • Prevent indexing of sensitive, duplicate, or non–SEO-friendly areas
  • Disallow generated and runtime files
  • Disallow Magento customer/account pages (not indexable)
  • Disallow layered navigation filters (potentially infinite URLs)
  • Disallow internal CMS pages not meant for indexing
  • -----------------------------------------------------
  • C) Allow CSS, JS, image files needed for rendering pages
  • (Important: Magento’s CSS/JS files must be crawlable for proper page rendering)
  • Specific directives for Googlebot (example of user-agent–specific rules)
  • Give Google more frequent access to CSS/JS
  • If you want to give Google a shorter crawl-delay:
  • Specific directives for Bingbot
  • Block any known aggressive or unwanted bots
Back to top