salespider.com
robots.txt
Robots Exclusion Standard data for salespider.com
Resource Scan
Scan Details
| Site Domain | salespider.com |
| Base Domain | salespider.com |
| Scan Status | Ok |
| Last Scan | 2024-11-07T20:38:18+00:00 |
| Next Scan | 2024-11-14T20:38:18+00:00 |
Last Scan
| Scanned | 2024-11-07T20:38:18+00:00 |
| URL | https://salespider.com/robots.txt |
| Redirect | http://www.salespider.com/robots.txt |
| Redirect Domain | www.salespider.com |
| Redirect Base | salespider.com |
| Domain IPs | 104.21.62.70, 172.67.221.16, 2606:4700:3036::6815:3e46, 2606:4700:3037::ac43:dd10 |
| Redirect IPs | 104.21.62.70, 172.67.221.16, 2606:4700:3036::6815:3e46, 2606:4700:3037::ac43:dd10 |
| Response IP | 104.21.62.70 |
| Found | Yes |
| Hash | 2e680a3372a2938e3d42dd5dbe46638e43fed207771c715153dbdaa812ad586f |
| SimHash | 4f0d503411b7 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /max |
| Disallow | /images/banners |
| Disallow | /bm |
| Disallow | /cpl |
| Disallow | /delivery |
| Disallow | /1054037 |
| Disallow | /selfserve |
| Disallow | /auto |
| Disallow | /directory-jobs |
| Disallow | /directory-businesses |
| Disallow | /directory-people |
| Disallow | /directory-classifieds |
| Disallow | /directory-biztools |
| Disallow | /directory-leads |
| Disallow | /directory-name |
| Disallow | /answers |
| Disallow | /cpid |
| Allow | / |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
Other Records
| Field | Value |
|---|---|
| sitemap | http://www.salespider.com/sitemap-index.xml |