dustin.be
robots.txt
Robots Exclusion Standard data for dustin.be
Resource Scan
Scan Details
| Site Domain | dustin.be |
| Base Domain | dustin.be |
| Scan Status | Ok |
| Last Scan | 2024-09-21T10:41:40+00:00 |
| Next Scan | 2024-09-28T10:41:40+00:00 |
Last Scan
| Scanned | 2024-09-21T10:41:40+00:00 |
| URL | https://dustin.be/robots.txt |
| Redirect | https://www.dustin.be:443/robots.txt |
| Redirect Domain | www.dustin.be |
| Redirect Base | dustin.be |
| Domain IPs | 2a02:2858:310::11, 75.2.6.158, 99.83.130.12 |
| Redirect IPs | 2600:9000:2022:3e00:17:1937:4cc0:93a1, 2600:9000:2022:4800:17:1937:4cc0:93a1, 2600:9000:2022:600:17:1937:4cc0:93a1, 2600:9000:2022:6800:17:1937:4cc0:93a1, 2600:9000:2022:9600:17:1937:4cc0:93a1, 2600:9000:2022:a800:17:1937:4cc0:93a1, 2600:9000:2022:bc00:17:1937:4cc0:93a1, 2600:9000:2022:f600:17:1937:4cc0:93a1, 54.230.112.13, 54.230.112.28, 54.230.112.32, 54.230.112.4 |
| Response IP | 52.85.49.78 |
| Found | Yes |
| Hash | fcdbb6c6786e91d2a939fe3c64dac41f14a1c9f7f7fa70ab2a5bd1a582b08b53 |
| SimHash | 7a13db56f731 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | *network%3D |
| Disallow | /ajax.php |
| Disallow | /?page=cart |
| Disallow | /nl/winkelwagen/ |
| Disallow | /fr/panier/ |
| Disallow | *request%3DminiProduct |
| Disallow | /kiss/ |
| Disallow | /*?page=compare |
| Disallow | /nl/rechercher/ |
| Disallow | /nl/zoeken/ |
| Disallow | /fr/zoeken/ |
| Disallow | /fr/rechercher/ |
| Disallow | /*?page=search |
| Disallow | /*?search= |
| Disallow | /*/review/ |
| Disallow | /*/review |
| Disallow | /*?request=stockReminder |
| Disallow | /*?request=getYoutube |
| Disallow | /*?article= |
| Disallow | /*?famid= |
| Disallow | /*?page= |
| Disallow | /*?pld0page= |
| Disallow | /*?rtv= |
| Disallow | /*?cid= |
| Disallow | /*?ref= |
| Disallow | /*?delete= |
| Disallow | /*?limit_direction_ |
| Disallow | /*?t= |
| Disallow | /frame.php?url= |
| Disallow | /*?special= |
| Disallow | /*?Sorting= |
| Disallow | /*?sorting= |
| Disallow | /*?author= |
| Disallow | /*?prodid= |
| Disallow | /*?cat= |
| Disallow | /protected/ |
| Disallow | /tracker/ |
| Disallow | /*?lang= |
| Disallow | /*?refine |
| Disallow | /*?fromblog= |
| Disallow | /*?review_sort= |
| Disallow | /*?promo= |
| Disallow | /*?oid= |
| Disallow | /*?cameFrom= |
| Disallow | /*?Nr= |
| Disallow | /*?print= |
| Disallow | /*?sms |
| Disallow | /*?amount |
| Disallow | /*?fss= |
| Disallow | /api2/ |
| Disallow | /*?facet |
| Disallow | /*?reset |
| Disallow | /static-pages/* |
| Disallow | /error-pages/* |
| Disallow | /node-api/ |
| Disallow | /node-elastic/ |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.dustin.be/sitemap_index.xml |