dotbg.bg
robots.txt

Robots Exclusion Standard data for dotbg.bg

Resource Scan

Scan Details

Site Domain dotbg.bg
Base Domain dotbg.bg
Scan Status Ok
Last Scan2024-09-23T13:01:48+00:00
Next Scan 2024-09-30T13:01:48+00:00

Last Scan

Scanned2024-09-23T13:01:48+00:00
URL https://dotbg.bg/robots.txt
Redirect https://www.dotbg.bg/robots.txt
Redirect Domain www.dotbg.bg
Redirect Base dotbg.bg
Domain IPs 104.21.73.64, 172.67.188.230, 2606:4700:3031::6815:4940, 2606:4700:3031::ac43:bce6
Redirect IPs 104.21.73.64, 172.67.188.230, 2606:4700:3031::6815:4940, 2606:4700:3031::ac43:bce6
Response IP 104.21.73.64
Found Yes
Hash 37a26a18b45db59b4ac5f79e6f1019f9d0ee3bfd41944e7af85b5b1bbb1e6db6
SimHash 6510db727792

Groups

scrapy

Rule Path
Disallow /

*

Rule Path
Disallow /Search
Disallow /Article/RecommendArticle/
Allow /