pepper.it
robots.txt
Robots Exclusion Standard data for pepper.it
Resource Scan
Scan Details
| Site Domain | pepper.it |
| Base Domain | pepper.it |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 2024-08-27T01:33:36+00:00 |
| Next Scan | 2024-11-25T01:33:36+00:00 |
Last Successful Scan
| Scanned | 2024-04-30T01:24:58+00:00 |
| URL | https://pepper.it/robots.txt |
| Redirect | https://www.pepper.it/robots.txt |
| Redirect Domain | www.pepper.it |
| Redirect Base | pepper.it |
| Domain IPs | 104.18.16.202, 104.18.17.202, 2606:4700::6812:10ca, 2606:4700::6812:11ca |
| Redirect IPs | 104.18.16.202, 104.18.17.202, 2606:4700::6812:10ca, 2606:4700::6812:11ca |
| Response IP | 104.18.17.202 |
| Found | Yes |
| Hash | 9986d2207d0c08fa9b14d3e9ecd2a3ce5873a2e5e93efc91f38685af527bb1dd |
| SimHash | ea08d202ee20 |
Groups
*
| Rule | Path |
|---|---|
| Allow | / |
| Allow | /search/price_comparison/geizhals |
| Disallow | /cdn-cgi/ |
| Disallow | /assets/js/share/thread-embed.js |
| Disallow | /selection |
| Disallow | /*?layout=* |
| Disallow | /login |
| Disallow | /visit/ |
| Disallow | /graphql |
| Disallow | /codici-sconto/api/* |
| Disallow | /codici-sconto/loading/* |
| Disallow | /codici-sconto/m/clickout |
| Disallow | /codici-sconto/m/click-* |
| Disallow | /codici-sconto/m/cn-clickout* |
| Disallow | /codici-sconto/clickout/* |
| Disallow | /codici-sconto/user/submit |
| Disallow | /codici-sconto/*?preview |
| Disallow | /codici-sconto/m/clickout/* |
| Disallow | /rest-api/ |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.pepper.it/sitemap/sitemap.xml.gz |
| sitemap | https://www.pepper.it/codici-sconto/sitemap.xml |