mama24.bg
robots.txt

Robots Exclusion Standard data for mama24.bg

Resource Scan

Scan Details

Site Domain mama24.bg
Base Domain mama24.bg
Scan Status Ok
Last Scan2024-06-14T03:42:03+00:00
Next Scan 2024-06-21T03:42:03+00:00

Last Scan

Scanned2024-06-14T03:42:03+00:00
URL https://mama24.bg/robots.txt
Redirect https://www.mama24.bg/robots.txt
Redirect Domain www.mama24.bg
Redirect Base mama24.bg
Domain IPs 104.21.49.83, 172.67.160.110, 2606:4700:3032::6815:3153, 2606:4700:3036::ac43:a06e
Redirect IPs 104.21.49.83, 172.67.160.110, 2606:4700:3032::6815:3153, 2606:4700:3036::ac43:a06e
Response IP 172.67.160.110
Found Yes
Hash 37a26a18b45db59b4ac5f79e6f1019f9d0ee3bfd41944e7af85b5b1bbb1e6db6
SimHash 6510db727792

Groups

scrapy

Rule Path
Disallow /

*

Rule Path
Disallow /Search
Disallow /Article/RecommendArticle/
Allow /