b.dk
robots.txt
Robots Exclusion Standard data for b.dk
Resource Scan
Scan Details
Site Domain | b.dk |
Base Domain | b.dk |
Scan Status | Ok |
Last Scan | 2024-05-19T11:07:50+00:00 |
Next Scan | 2024-05-26T11:07:50+00:00 |
Last Scan
Scanned | 2024-05-19T11:07:50+00:00 |
URL | https://b.dk/robots.txt |
Redirect | https://www.berlingske.dk/robots.txt |
Redirect Domain | www.berlingske.dk |
Redirect Base | berlingske.dk |
Domain IPs | 13.33.30.24, 13.33.30.38, 13.33.30.65, 13.33.30.81 |
Redirect IPs | 96.17.96.21, 96.17.96.24 |
Response IP | 23.44.4.160 |
Found | Yes |
Hash | 1f7e7b39e9c48ae4580a3cc6682990bb343494b331915d1dc1d759acd326ca78 |
SimHash | 7e44de043833 |
Groups
*
Rule | Path |
---|---|
Disallow | /api |
Disallow | /redirect |
Disallow | /logout |
Disallow | /login |
Disallow | /register |
Disallow | /user |
Disallow | /opener-reload-and-window-close |
Disallow | /showMore |
Disallow | /autocomplete |
Disallow | /image_gallery/ |
Disallow | /tracking/image_gallery/ |
Disallow | /artikel-arkiv |
Disallow | /search |
Disallow | /sonata/ |
Other Records
Field | Value |
---|---|
crawl-delay | 10 |
Other Records
Field | Value |
---|---|
sitemap | https://www.berlingske.dk/sitemap.xml |
Comments