bergfreunde.nl
robots.txt

Robots Exclusion Standard data for bergfreunde.nl

Resource Scan

Scan Details

Site Domain bergfreunde.nl
Base Domain bergfreunde.nl
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-04-25T17:57:58+00:00
Next Scan 2024-07-24T17:57:58+00:00

Last Successful Scan

Scanned2023-09-06T10:29:10+00:00
URL https://bergfreunde.nl/robots.txt
Redirect https://www.bergfreunde.nl:443/robots.txt
Redirect Domain www.bergfreunde.nl
Redirect Base bergfreunde.nl
Domain IPs 34.111.216.220
Redirect IPs 96.17.96.13, 96.17.96.24
Response IP 104.88.70.136
Found Yes
Hash a01d46d7b3ace6e2bfbb29dbd039c1ac514a3d82ca0e878b7d940bb006ee2dea
SimHash 6915d555ed85

Groups

*

Rule Path
Disallow /*/beoordeling--
Disallow /*/kleur--
Disallow /*/maat--
Disallow /*?cl=
Disallow /*?bfc-
Disallow /*?streamfilter
Disallow /*?listorderby
Disallow /*?_artperpage
Disallow /*?rev
Disallow /*?plain=true

pinterestbot

Rule Path
Disallow /*/beoordeling--
Disallow /*/kleur--
Disallow /*/maat--
Disallow /*?cl=
Disallow /*?bfc-
Disallow /*?streamfilter
Disallow /*?listorderby
Disallow /*?_artperpage
Disallow /*?rev
Disallow /*?plain=true

Other Records

Field Value
crawl-delay 1

bingbot

Rule Path
Disallow /*/beoordeling--
Disallow /*/kleur--
Disallow /*/maat--
Disallow /*?cl=
Disallow /*?bfc-
Disallow /*?streamfilter
Disallow /*?listorderby
Disallow /*?_artperpage
Disallow /*?rev
Disallow /*?plain=true

Other Records

Field Value
crawl-delay 1

Other Records

Field Value
sitemap https://www.bergfreunde.nl/sitemap/sitemap.xml.gz
sitemap https://www.bergfreunde.nl/blog/sitemap_index.xml