hogrefe.com
robots.txt
Robots Exclusion Standard data for hogrefe.com
Resource Scan
Scan Details
| Site Domain | hogrefe.com |
| Base Domain | hogrefe.com |
| Scan Status | Ok |
| Last Scan | 2024-06-09T10:04:31+00:00 |
| Next Scan | 2024-07-09T10:04:31+00:00 |
Last Scan
| Scanned | 2024-06-09T10:04:31+00:00 |
| URL | https://hogrefe.com/robots.txt |
| Redirect | https://www.hogrefe.com/robots.txt |
| Redirect Domain | www.hogrefe.com |
| Redirect Base | hogrefe.com |
| Domain IPs | 194.6.208.155, 2a00:f48:1008::208:155:10 |
| Redirect IPs | 194.6.208.155 |
| Response IP | 194.6.208.155 |
| Found | Yes |
| Hash | 5ff78aa0f412d5839bbd25550e7869f1fb1bcfc8508e2d39924b6c080c4d2827 |
| SimHash | 530f60477206 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /typo3/ |
| Disallow | /shop/hog-mage/ |
| Disallow | *tx_news_pi* |
| Disallow | *tx_wwproject_seminar* |
| Disallow | *tx_bdsearch_searchresult* |
| Disallow | *.html?type* |
| Disallow | /shop/catalogsearch/ |
| Disallow | /shop/customer/ |
| Disallow | /shop/shoppinglist/ |
| Disallow | /shop/checkout/ |
| Disallow | /shop/catalog/product_compare/ |
| Disallow | /shop/catalog/category/ |
| Disallow | /shop/media/catalog/product/cache/ |
| Disallow | /shop/*?* |
| Disallow | /*/shop/catalogsearch/ |
| Disallow | /*/shop/customer/ |
| Disallow | /*/shop/shoppinglist/ |
| Disallow | /*/shop/checkout/ |
| Disallow | /*/shop/catalog/product_compare/ |
| Disallow | /*/shop/catalog/category/ |
| Disallow | /*/shop/media/catalog/product/cache/ |
| Disallow | /*/shop/*?* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 3 |
Warnings
- 4 invalid lines.
Comments