bergfreunde.de
robots.txt

Robots Exclusion Standard data for bergfreunde.de

Resource Scan

Scan Details

Site Domain bergfreunde.de
Base Domain bergfreunde.de
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-04-07T14:24:45+00:00
Next Scan 2024-07-06T14:24:45+00:00

Last Successful Scan

Scanned2023-08-17T03:24:01+00:00
URL https://bergfreunde.de/robots.txt
Redirect https://www.bergfreunde.de:443/robots.txt
Redirect Domain www.bergfreunde.de
Redirect Base bergfreunde.de
Domain IPs 34.111.216.220
Redirect IPs 96.17.96.13, 96.17.96.24
Response IP 23.33.126.19
Found Yes
Hash e9d4b597f27949d47da6277c8e01ea8b69e90af93580606d9b0b95c8d1300ae4
SimHash 61931754f583

Groups

*

Rule Path
Disallow /*/bewertung--
Disallow /*/farbe--
Disallow /*?cl=
Disallow /*?bfc-
Disallow /*?streamfilter
Disallow /*?listorderby
Disallow /*?_artperpage
Disallow /*?rev
Disallow /*?plain=true

pinterestbot

Rule Path
Disallow /*/bewertung--
Disallow /*/farbe--
Disallow /*?cl=
Disallow /*?bfc-
Disallow /*?streamfilter
Disallow /*?listorderby
Disallow /*?_artperpage
Disallow /*?rev
Disallow /*?plain=true

Other Records

Field Value
crawl-delay 1

bingbot

Rule Path
Disallow /*/bewertung--
Disallow /*/farbe--
Disallow /*?cl=
Disallow /*?bfc-
Disallow /*?streamfilter
Disallow /*?listorderby
Disallow /*?_artperpage
Disallow /*?rev
Disallow /*?plain=true

Other Records

Field Value
crawl-delay 1

Other Records

Field Value
sitemap https://www.bergfreunde.de/sitemap/sitemap.xml.gz
sitemap https://www.bergfreunde.de/data/om/sitemap/sitemap_videos.xml
sitemap https://www.bergfreunde.de/blog/sitemap_index.xml