parfumerie.be
robots.txt

Robots Exclusion Standard data for parfumerie.be

Resource Scan

Scan Details

Site Domain parfumerie.be
Base Domain parfumerie.be
Scan Status Ok
Last Scan2025-10-18T09:14:53+00:00
Next Scan 2025-11-17T09:14:53+00:00

Last Scan

Scanned2025-10-18T09:14:53+00:00
URL https://parfumerie.be/robots.txt
Redirect https://www.parfumerie.be/robots.txt
Redirect Domain www.parfumerie.be
Redirect Base parfumerie.be
Domain IPs 104.26.14.52, 104.26.15.52, 172.67.71.148, 2606:4700:20::681a:e34, 2606:4700:20::681a:f34, 2606:4700:20::ac43:4794
Redirect IPs 104.26.14.52, 104.26.15.52, 172.67.71.148, 2606:4700:20::681a:e34, 2606:4700:20::681a:f34, 2606:4700:20::ac43:4794
Response IP 172.67.71.148
Found Yes
Hash 2dbbaefeba39740d0ef550a21628abf646f810b410f221af9a48ea3faf3222d5
SimHash 6b16d953c8b0

Groups

*

Rule Path
Allow /*?p=
Allow /media/catalog/
Allow /*?size=
Allow /*?sku=
Disallow /*?
Disallow /index.php/
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /catalog/product/view/
Disallow /wishlist/
Disallow /catalogsearch/
Disallow /review/product/
Disallow /sendfriend/
Disallow /enable-cookies/
Disallow /LICENSE.txt
Disallow /LICENSE.html
Disallow /directory/
Disallow /checkout/
Disallow /onestepcheckout/
Disallow /customer/
Disallow /customer/account/
Disallow /customer/account/login/
Disallow /catalogsearch/
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /catalog/product/view/
Disallow /*?dir*
Disallow /*?dir=desc
Disallow /*?dir=asc
Disallow /*?limit=all
Disallow /*?mode*
Disallow /tag/
Disallow /review/
Disallow /*.php$
Disallow /*?*SID=
Disallow /*?*dir=
Disallow /*?*order=
Disallow /*?*limit=
Disallow /*?*price=
Disallow /*?*cat=
Disallow /*?*acc=
Disallow /rss*
Disallow /*PHPSESSID
Disallow /catalog/product/gallery/id/
Disallow /catalog/product/gallery/image/
Disallow /returns/*

adsbot-google

Rule Path
Disallow

baiduspider

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

fasterfox

Rule Path
Disallow /

teracent-feed-processing

Rule Path
Disallow /

ingrid

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

jyxobot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

yahoo! slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

rogerbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

ahrefsbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.parfumerie.nl/media/sitemap/parfumerienl.xml

Comments

  • Crawler setup
  • Allowable Index
  • Directories
  • Account & checkout pages
  • Catalog & search pages
  • Filtered pages
  • Duplicate content
  • Paths (no clean URLs)
  • For specific bots (on all domains)
  • Backlink crawler (https://ahrefs.com/robot)