berlingske.dk
robots.txt
Robots Exclusion Standard data for berlingske.dk
Resource Scan
Scan Details
| Site Domain | berlingske.dk |
| Base Domain | berlingske.dk |
| Scan Status | Ok |
| Last Scan | 2024-04-30T18:40:14+00:00 |
| Next Scan | 2024-05-07T18:40:14+00:00 |
Last Scan
| Scanned | 2024-04-30T18:40:14+00:00 |
| URL | https://berlingske.dk/robots.txt |
| Redirect | https://www.berlingske.dk/robots.txt |
| Redirect Domain | www.berlingske.dk |
| Redirect Base | berlingske.dk |
| Domain IPs | 13.33.30.24, 13.33.30.38, 13.33.30.65, 13.33.30.81 |
| Redirect IPs | 96.17.96.21, 96.17.96.24 |
| Response IP | 23.59.168.114 |
| Found | Yes |
| Hash | 1f7e7b39e9c48ae4580a3cc6682990bb343494b331915d1dc1d759acd326ca78 |
| SimHash | 7e44de043833 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /api |
| Disallow | /redirect |
| Disallow | /logout |
| Disallow | /login |
| Disallow | /register |
| Disallow | /user |
| Disallow | /opener-reload-and-window-close |
| Disallow | /showMore |
| Disallow | /autocomplete |
| Disallow | /image_gallery/ |
| Disallow | /tracking/image_gallery/ |
| Disallow | /artikel-arkiv |
| Disallow | /search |
| Disallow | /sonata/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 10 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.berlingske.dk/sitemap.xml |
Comments