holdnyt.dk
robots.txt

Robots Exclusion Standard data for holdnyt.dk

Resource Scan

Scan Details

Site Domain holdnyt.dk
Base Domain holdnyt.dk
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a server error.
Last Scan2024-06-12T15:35:00+00:00
Next Scan 2024-09-10T15:35:00+00:00

Last Successful Scan

Scanned2023-11-16T07:03:21+00:00
URL https://holdnyt.dk/robots.txt
Redirect https://www.holdnyt.dk/robots.txt
Redirect Domain www.holdnyt.dk
Redirect Base holdnyt.dk
Domain IPs 104.21.27.147, 172.67.142.240, 2606:4700:3036::6815:1b93, 2606:4700:3036::ac43:8ef0
Redirect IPs 104.21.27.147, 172.67.142.240, 2606:4700:3036::6815:1b93, 2606:4700:3036::ac43:8ef0
Response IP 104.21.27.147
Found Yes
Hash caba6c2a9251f90add6c5f320a7e27d029bb774d88a9548fcd2ee67b2b88e38c
SimHash 6d895045c3b1

Groups

bingbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 20

bingbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 20

yahoo! slurp

Rule Path
Disallow

Other Records

Field Value
crawl-delay 20

*

Rule Path
Disallow /cgi-bin/
Disallow /admin/
Disallow /nyheder/fodbold/danmark/superligaen/midtjylland

Other Records

Field Value
sitemap http://www.holdnyt.dk/sitemap.php