highbeer.nl
robots.txt

Robots Exclusion Standard data for highbeer.nl

Resource Scan

Scan Details

Site Domain highbeer.nl
Base Domain highbeer.nl
Scan Status Ok
Last Scan5/19/2025, 6:29:07 AM
Next Scan 5/26/2025, 6:29:07 AM

Last Scan

Scanned5/19/2025, 6:29:07 AM
URL https://highbeer.nl/robots.txt
Redirect https://www.highbeer.nl/robots.txt
Redirect Domain www.highbeer.nl
Redirect Base highbeer.nl
Domain IPs 104.21.42.165, 172.67.163.174, 2606:4700:3031::6815:2aa5, 2606:4700:3035::ac43:a3ae
Redirect IPs 104.21.42.165, 172.67.163.174, 2606:4700:3031::6815:2aa5, 2606:4700:3035::ac43:a3ae
Response IP 172.67.163.174
Found Yes
Hash 2dccd0077389206c099d3b8e2d1cd5a7af565c5f80cdb37db3d60d0bf77ba6d3
SimHash 4a04dc108333

Groups

scrapy

Rule Path
Allow /

*

Rule Path
Disallow /wp-admin/
Disallow /portal/

Other Records

Field Value
crawl-delay 10

Other Records

Field Value
sitemap https://www.hightea.nl/sitemap.xml