mama24.bg
robots.txt

Robots Exclusion Standard data for mama24.bg

Resource Scan

Scan Details

Site Domain mama24.bg
Base Domain mama24.bg
Scan Status Ok
Last Scan2024-11-15T11:32:44+00:00
Next Scan 2024-11-22T11:32:44+00:00

Last Scan

Scanned2024-11-15T11:32:44+00:00
URL https://mama24.bg/robots.txt
Redirect https://www.mama24.bg/robots.txt
Redirect Domain www.mama24.bg
Redirect Base mama24.bg
Domain IPs 104.21.49.83, 172.67.160.110, 2606:4700:3032::6815:3153, 2606:4700:3036::ac43:a06e
Redirect IPs 104.21.49.83, 172.67.160.110, 2606:4700:3032::6815:3153, 2606:4700:3036::ac43:a06e
Response IP 104.21.49.83
Found Yes
Hash 37a26a18b45db59b4ac5f79e6f1019f9d0ee3bfd41944e7af85b5b1bbb1e6db6
SimHash 6510db727792

Groups

scrapy

Rule Path
Disallow /

*

Rule Path
Disallow /Search
Disallow /Article/RecommendArticle/
Allow /