Log In Sign Up

morhane.nl
robots.txt

Robots Exclusion Standard data for morhane.nl

Resource Scan

Scan Details

Site Domain morhane.nl
Base Domain morhane.nl
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-12-25T08:07:36+00:00
Next Scan 2026-03-25T08:07:36+00:00

Last Successful Scan

Scanned2025-05-23T00:24:59+00:00
URL https://morhane.nl/robots.txt
Domain IPs 104.21.11.119, 172.67.166.2, 2606:4700:3032::6815:b77, 2606:4700:3033::ac43:a602
Response IP 104.21.11.119
Found Yes
Hash 7f245909681fa1fb2d2abd4f103d7a93e02b6b286806282b9aac7333007b2b74
SimHash 384cb971c8b8

Groups

*

Rule Path Comment
Allow /*?p= -
Allow /media/catalog/ -
Allow /media/wysiwyg/ -
Disallow */404/ -
Disallow */app/ -
Disallow */cgi-bin/ -
Disallow */downloader/ -
Disallow */errors/ -
Disallow */includes/ -
Disallow */lib/ -
Disallow */magento/ -
Disallow */media/captcha/ -
Disallow */media/css_secure/ -
Disallow */media/customer/ -
Disallow */media/dhl/ -
Disallow */media/downloadable/ -
Disallow */media/import/ -
Disallow */media/pdf/ -
Disallow */media/sales/ -
Disallow */media/tmp/ -
Disallow */media/xmlconnect/ -
Disallow */pkginfo/ -
Disallow */report/ -
Disallow */scripts/ -
Disallow */shell/ -
Disallow */stats/ -
Disallow */var/ -
Disallow */*index.php* -
Disallow */catalog/product_compare/ -
Disallow */catalog/category/view/ -
Disallow */catalog/product/view/ -
Disallow */catalog/product/gallery/ -
Disallow */catalogsearch/ -
Disallow */checkout/ -
Disallow */control/ -
Disallow */contacts/ -
Disallow */customer/ -
Disallow */customize/ -
Disallow */newsletter/ -
Disallow */poll/ -
Disallow */review/ -
Disallow */sendfriend/ -
Disallow */tag/ -
Disallow */wishlist/ -
Disallow */*virtuemart* -
Disallow */*?pop=0* -
Disallow */*flypage.tpl* -
Disallow */*raamfolie* -
Disallow */*list-all-products* -
Disallow */*com_gk3* -
Disallow */*author* -
Disallow */*awcountdown* -
Disallow */advancedsearch/ -
Disallow */addtocart/ -
Disallow */cartquickpro/ -
Disallow */productalert/ -
Disallow */catalogsearch/advanced/ -
Disallow */search/ -
Disallow */sales/guest/form/ -
Disallow */my-account/ -
Disallow */vendor/ -
Disallow */update/ -
Disallow */setup/ -
Disallow */phpserver/ -
Disallow */dev/ -
Disallow */bin/ -
Disallow */page_cache/block/* -
Disallow */stores/store/redirect/* -
Disallow */stores/store/switch/* -
Disallow /cdn-cgi/l/email-protection -
Disallow */privacy-policy-cookie-restriction-mode/ -
Disallow */terms/ -
Disallow */cron.php -
Disallow */cron.sh -
Disallow */error_log -
Disallow */install.php -
Disallow */LICENSE.html -
Disallow */LICENSE.txt -
Disallow */LICENSE_AFL.txt -
Disallow */STATUS.txt -
Disallow */get.php Magento 1.5+
Disallow /*?price= -
Disallow /shopby/ -
Disallow /*.php$ -
Disallow /*?p=*& -
Disallow /*?p=* -
Disallow /*?pagenr=* -
Disallow /*?pagenr=*& -
Disallow /*?SID= -
Disallow /?___from_store= -
Disallow /*___from_store%3D -
Disallow /?dir= -
Disallow /?mode= -
Disallow /?limit= -
Disallow /rss* -
Disallow /*PHPSESSID -
Disallow */sort-by/* -
Disallow */show/* -
Disallow */?gclid=* -
Disallow /*is_ajax -
Disallow /*?page= -
Disallow /*?customFilters= -
Disallow /*?pile_type=* -
Disallow /*?size=* -
Disallow /*?cat=* -
Disallow /*?price=* -

oai-searchbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

yahoo! slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

duckduckbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

yandexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

baiduspider

Rule Path
Disallow /

bspider

Rule Path
Disallow /

sogou

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

vagabondo

Rule Path
Disallow /

exabot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

moz

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

fasterfox

Rule Path
Disallow /

jyxobot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

geedoproductsearch

Rule Path
Disallow /

gptbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

gptbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

chatgpt-user

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

claudebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

anthropic-ai

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

ccbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

facebookbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

meta-externalagent

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

google-extended

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

googleai

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

gemini

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

deepmind-explore

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

bytespider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

amazonbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

applebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

cohere-ai

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

youbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

airesearchbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

seekrbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

neevabot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

perplexitybot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

writer

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

screaming frog seo spider

Rule Path
Allow /
Disallow /*.gif$
Disallow /*.jpg$
Disallow /*.png$
Disallow /*.bmp$
Disallow /*.xml$
Disallow /*.css$
Disallow /*.js$
Back to top

Comments

  • Crawlers SetupF
  • Allowable Index
  • Mind that Allow is not an official standard
  • Directories
  • Paths (clean URLs)
  • Efex
  • Pages
  • Files
  • filters
  • Paths (no clean URLs)
  • Disallow: /*.js$
  • Disallow: /*.css$
  • Efex
  • Disallow: /*.map$
  • Crawler bots
  • AI bots
  • Sitemaps
Back to top