morhane.nl
robots.txt

Robots Exclusion Standard data for morhane.nl

Resource Scan

Scan Details

Site Domain morhane.nl
Base Domain morhane.nl
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-12-25T08:07:36+00:00
Next Scan 2026-03-25T08:07:36+00:00

Last Successful Scan

Scanned2025-05-23T00:24:59+00:00
URL https://morhane.nl/robots.txt
Domain IPs 104.21.11.119, 172.67.166.2, 2606:4700:3032::6815:b77, 2606:4700:3033::ac43:a602
Response IP 104.21.11.119
Found Yes
Hash 7f245909681fa1fb2d2abd4f103d7a93e02b6b286806282b9aac7333007b2b74
SimHash 384cb971c8b8

Groups

*

Rule Path Comment
Allow /*?p= -
Allow /media/catalog/ -
Allow /media/wysiwyg/ -
Disallow */404/ -
Disallow */app/ -
Disallow */cgi-bin/ -
Disallow */downloader/ -
Disallow */errors/ -
Disallow */includes/ -
Disallow */lib/ -
Disallow */magento/ -
Disallow */media/captcha/ -
Disallow */media/css_secure/ -
Disallow */media/customer/ -
Disallow */media/dhl/ -
Disallow */media/downloadable/ -
Disallow */media/import/ -
Disallow */media/pdf/ -
Disallow */media/sales/ -
Disallow */media/tmp/ -
Disallow */media/xmlconnect/ -
Disallow */pkginfo/ -
Disallow */report/ -
Disallow */scripts/ -
Disallow */shell/ -
Disallow */stats/ -
Disallow */var/ -
Disallow */*index.php* -
Disallow */catalog/product_compare/ -
Disallow */catalog/category/view/ -
Disallow */catalog/product/view/ -
Disallow */catalog/product/gallery/ -
Disallow */catalogsearch/ -
Disallow */checkout/ -
Disallow */control/ -
Disallow */contacts/ -
Disallow */customer/ -
Disallow */customize/ -
Disallow */newsletter/ -
Disallow */poll/ -
Disallow */review/ -
Disallow */sendfriend/ -
Disallow */tag/ -
Disallow */wishlist/ -
Disallow */*virtuemart* -
Disallow */*?pop=0* -
Disallow */*flypage.tpl* -
Disallow */*raamfolie* -
Disallow */*list-all-products* -
Disallow */*com_gk3* -
Disallow */*author* -
Disallow */*awcountdown* -
Disallow */advancedsearch/ -
Disallow */addtocart/ -
Disallow */cartquickpro/ -
Disallow */productalert/ -
Disallow */catalogsearch/advanced/ -
Disallow */search/ -
Disallow */sales/guest/form/ -
Disallow */my-account/ -
Disallow */vendor/ -
Disallow */update/ -
Disallow */setup/ -
Disallow */phpserver/ -
Disallow */dev/ -
Disallow */bin/ -
Disallow */page_cache/block/* -
Disallow */stores/store/redirect/* -
Disallow */stores/store/switch/* -
Disallow /cdn-cgi/l/email-protection -
Disallow */privacy-policy-cookie-restriction-mode/ -
Disallow */terms/ -
Disallow */cron.php -
Disallow */cron.sh -
Disallow */error_log -
Disallow */install.php -
Disallow */LICENSE.html -
Disallow */LICENSE.txt -
Disallow */LICENSE_AFL.txt -
Disallow */STATUS.txt -
Disallow */get.php Magento 1.5+
Disallow /*?price= -
Disallow /shopby/ -
Disallow /*.php$ -
Disallow /*?p=*& -
Disallow /*?p=* -
Disallow /*?pagenr=* -
Disallow /*?pagenr=*& -
Disallow /*?SID= -
Disallow /?___from_store= -
Disallow /*___from_store%3D -
Disallow /?dir= -
Disallow /?mode= -
Disallow /?limit= -
Disallow /rss* -
Disallow /*PHPSESSID -
Disallow */sort-by/* -
Disallow */show/* -
Disallow */?gclid=* -
Disallow /*is_ajax -
Disallow /*?page= -
Disallow /*?customFilters= -
Disallow /*?pile_type=* -
Disallow /*?size=* -
Disallow /*?cat=* -
Disallow /*?price=* -

oai-searchbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

yahoo! slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

duckduckbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

yandexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

baiduspider

Rule Path
Disallow /

bspider

Rule Path
Disallow /

sogou

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

vagabondo

Rule Path
Disallow /

exabot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

moz

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

fasterfox

Rule Path
Disallow /

jyxobot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

geedoproductsearch

Rule Path
Disallow /

gptbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

gptbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

chatgpt-user

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

claudebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

anthropic-ai

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

ccbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

facebookbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

meta-externalagent

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

google-extended

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

googleai

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

gemini

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

deepmind-explore

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

bytespider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

amazonbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

applebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

cohere-ai

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

youbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

airesearchbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

seekrbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

neevabot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

perplexitybot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

writer

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

screaming frog seo spider

Rule Path
Allow /
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.png$
Disallow /*.bmp$
Disallow /*.xml$
Disallow /*.css$
Disallow /*.js$

Comments

  • Crawlers SetupF
  • Allowable Index
  • Mind that Allow is not an official standard
  • Directories
  • Paths (clean URLs)
  • Efex
  • Pages
  • Files
  • filters
  • Paths (no clean URLs)
  • Disallow: /*.js$
  • Disallow: /*.css$
  • Efex
  • Disallow: /*.map$
  • Crawler bots
  • AI bots
  • Sitemaps