bestel.beren.nl
robots.txt
Robots Exclusion Standard data for bestel.beren.nl
Resource Scan
Scan Details
| Site Domain | bestel.beren.nl |
| Base Domain | beren.nl |
| Scan Status | Ok |
| Last Scan | 2024-11-03T16:33:18+00:00 |
| Next Scan | 2024-12-03T16:33:18+00:00 |
Last Scan
| Scanned | 2024-11-03T16:33:18+00:00 |
| URL | https://bestel.beren.nl/robots.txt |
| Domain IPs | 13.226.2.111, 13.226.2.125, 13.226.2.13, 13.226.2.36, 2600:9000:20fe:0:1a:8287:1c40:93a1, 2600:9000:20fe:2200:1a:8287:1c40:93a1, 2600:9000:20fe:2c00:1a:8287:1c40:93a1, 2600:9000:20fe:9400:1a:8287:1c40:93a1, 2600:9000:20fe:a400:1a:8287:1c40:93a1, 2600:9000:20fe:c400:1a:8287:1c40:93a1, 2600:9000:20fe:e600:1a:8287:1c40:93a1, 2600:9000:20fe:fc00:1a:8287:1c40:93a1 |
| Response IP | 3.164.230.92 |
| Found | Yes |
| Hash | a8538397aafc05d68d13d0b39e82a9e801cd8e8db38dfab38c5ea54db608abaa |
| SimHash | 28586086a672 |
Groups
webreaper
webcopier
offline explorer
httrack
microsoft.url.control
emailcollector
penthesilea
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Disallow | /dataPrivacy |
| Disallow | /imprint |
| Disallow | /apilogindenied |
| Disallow | /autologout |
| Disallow | /commonerror |
| Disallow | /devdata |
| Disallow | /permissiondenied |
| Disallow | /nojs |
| Disallow | /newerBrowser |
| Disallow | /setIFrameSession |
| Disallow | /health |
| Disallow | /systemdata |
| Disallow | /saveCookieChoise |
| Disallow | /simulateWebsiteWithIframe |
Other Records
| Field | Value |
|---|---|
| sitemap | https://bestel.beren.nl/sitemap |