Log In Sign Up

bouwkampioen.be
robots.txt

Robots Exclusion Standard data for bouwkampioen.be

Resource Scan

Scan Details

Site Domain bouwkampioen.be
Base Domain bouwkampioen.be
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-09-08T06:35:12+00:00
Next Scan 2025-12-07T06:35:12+00:00

Last Successful Scan

Scanned2024-10-21T05:28:38+00:00
URL https://bouwkampioen.be/robots.txt
Redirect https://www.bouwkampioen.be/robots.txt
Redirect Domain www.bouwkampioen.be
Redirect Base bouwkampioen.be
Domain IPs 104.21.62.220, 172.67.139.210, 2606:4700:3031::6815:3edc, 2606:4700:3037::ac43:8bd2
Redirect IPs 104.21.62.220, 172.67.139.210, 2606:4700:3031::6815:3edc, 2606:4700:3037::ac43:8bd2
Response IP 104.21.62.220
Found Yes
Hash 182760b27b1cf54184b95cc192b78516893268369a5da45b2bb466735fa191b7
SimHash 2216df52eef3

Groups

*

Rule Path
Allow /*.js
Allow /*.css

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

*

Rule Path
Disallow /404/
Disallow /app/
Disallow /cgi-bin/
Disallow /downloader/
Disallow /errors/
Disallow /includes/
Disallow /js/
Disallow /lib/
Disallow /magento/
Disallow /pkginfo/
Disallow /report/
Disallow /scripts/
Disallow /shell/
Disallow /stats/
Disallow /var/
Disallow /nl/searchautocomplete/suggest/result/*
Disallow /searchautocomplete/suggest/result/*
Allow /skin/frontend/default/construction/

*

Rule Path
Disallow /cron.php
Disallow /cron.sh
Disallow /error_log
Disallow /install.php
Disallow /LICENSE.html
Disallow /LICENSE.txt
Disallow /LICENSE_AFL.txt
Disallow /STATUS.txt
Disallow /*.php$
Disallow /*?SID=
Disallow /nl/*.php$
Disallow /nl/*?SID=
Back to top

Other Records

Field Value
sitemap https://www.bouwkampioen.be/sitemap/sitemap.xml
Back to top

Comments

  • Google Image Crawler Setup
  • User-agent: Googlebot-Image
  • Disallow:
  • Limits Bot's activity
  • User-agent: msnbot
  • Crawl-delay: 10
  • Crawlers Setup
  • Disallow: /media/
  • Disallow: /skin/
  • Files
  • Disallow: /nl/extradownloads/*
  • Disallow: /downloads/*.pdf
  • Sitemap
  • Paths that can be safely ignored (no clean URLs)
  • Disallow: /*?p=*&
  • Disallow: /nl/*?p=*&
  • These are more specific, pick what you need - and do not forget to add your custom filters!
  • Disallow: /*?dir*
  • Disallow: /*?limit*
  • Disallow: /*?mode*
  • Disallow: /*?___from_store=*
  • Disallow: /*?___store=*
  • Disallow: /*?cat=*
  • Disallow: /*?q=*
  • Disallow: /*?price=*
  • Disallow: /*?availability=*
  • Disallow: /*?brand=*
  • Disallow: /nl/quickrfq?pid=*
  • Disallow: /quickrfq?pid=*
  • Disallow: /nl/*.html?*
  • Disallow: /nl/catalogsearch/result/index/?*
  • Disallow: /*.html?*
  • Disallow: /catalogsearch/result/index/?*
Back to top