bgdnes.bg
robots.txt

Robots Exclusion Standard data for bgdnes.bg

Resource Scan

Scan Details

Site Domain bgdnes.bg
Base Domain bgdnes.bg
Scan Status Ok
Last Scan2024-05-26T22:43:28+00:00
Next Scan 2024-06-02T22:43:28+00:00

Last Scan

Scanned2024-05-26T22:43:28+00:00
URL https://bgdnes.bg/robots.txt
Redirect https://www.bgdnes.bg/robots.txt
Redirect Domain www.bgdnes.bg
Redirect Base bgdnes.bg
Domain IPs 104.21.31.31, 172.67.174.232, 2606:4700:3031::6815:1f1f, 2606:4700:3032::ac43:aee8
Redirect IPs 104.21.31.31, 172.67.174.232, 2606:4700:3031::6815:1f1f, 2606:4700:3032::ac43:aee8
Response IP 172.67.174.232
Found Yes
Hash bcffae9d8ae3f3fe182afff68479fea5ede30694e78602937d1a5228f605921a
SimHash 64109b534192

Groups

scrapy

Rule Path
Disallow /

*

Rule Path
Disallow /search
Disallow /Search
Disallow /Article/RecommendArticle/
Allow /

Other Records

Field Value
sitemap https://www.bgdnes.bg/sitemap.xml
sitemap https://www.bgdnes.bg/sitemap-rubrics.xml
sitemap https://www.bgdnes.bg/sitemap-static.xml