hiclub.bg
robots.txt

Robots Exclusion Standard data for hiclub.bg

Resource Scan

Scan Details

Site Domain hiclub.bg
Base Domain hiclub.bg
Scan Status Ok
Last Scan2024-11-16T18:16:08+00:00
Next Scan 2024-11-23T18:16:08+00:00

Last Scan

Scanned2024-11-16T18:16:08+00:00
URL https://hiclub.bg/robots.txt
Redirect https://www.hiclub.bg/robots.txt
Redirect Domain www.hiclub.bg
Redirect Base hiclub.bg
Domain IPs 104.21.82.92, 172.67.167.248, 2606:4700:3032::ac43:a7f8, 2606:4700:3037::6815:525c
Redirect IPs 104.21.82.92, 172.67.167.248, 2606:4700:3032::ac43:a7f8, 2606:4700:3037::6815:525c
Response IP 172.67.167.248
Found Yes
Hash 37a26a18b45db59b4ac5f79e6f1019f9d0ee3bfd41944e7af85b5b1bbb1e6db6
SimHash 6510db727792

Groups

scrapy

Rule Path
Disallow /

*

Rule Path
Disallow /Search
Disallow /Article/RecommendArticle/
Allow /