spokesman.com
robots.txt
Robots Exclusion Standard data for spokesman.com
Resource Scan
Scan Details
| Site Domain | spokesman.com |
| Base Domain | spokesman.com |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 2024-10-31T03:10:05+00:00 |
| Next Scan | 2024-11-30T03:10:05+00:00 |
Last Successful Scan
| Scanned | 2024-10-02T02:30:15+00:00 |
| URL | https://spokesman.com/robots.txt |
| Redirect | https://www.spokesman.com/robots.txt |
| Redirect Domain | www.spokesman.com |
| Redirect Base | spokesman.com |
| Domain IPs | 2600:9000:2795:3c00:13:3ce1:5800:93a1, 2600:9000:2795:5000:13:3ce1:5800:93a1, 2600:9000:2795:5600:13:3ce1:5800:93a1, 2600:9000:2795:5e00:13:3ce1:5800:93a1, 2600:9000:2795:7200:13:3ce1:5800:93a1, 2600:9000:2795:8e00:13:3ce1:5800:93a1, 2600:9000:2795:e400:13:3ce1:5800:93a1, 2600:9000:2795:ea00:13:3ce1:5800:93a1, 3.164.85.107, 3.164.85.115, 3.164.85.52, 3.164.85.98 |
| Redirect IPs | 18.244.214.19, 18.244.214.5, 18.244.214.63, 18.244.214.9, 2600:9000:23d0:1200:0:b5fb:e6c0:93a1, 2600:9000:23d0:1a00:0:b5fb:e6c0:93a1, 2600:9000:23d0:200:0:b5fb:e6c0:93a1, 2600:9000:23d0:9400:0:b5fb:e6c0:93a1, 2600:9000:23d0:a600:0:b5fb:e6c0:93a1, 2600:9000:23d0:c200:0:b5fb:e6c0:93a1, 2600:9000:23d0:e200:0:b5fb:e6c0:93a1, 2600:9000:23d0:e800:0:b5fb:e6c0:93a1 |
| Response IP | 18.165.122.50 |
| Found | Yes |
| Hash | d2832e662edea6157f6669953df7971341a8aa237b8119fa0b6e817783ecf8f8 |
| SimHash | e810fc10e305 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /accounts/ |
| Disallow | /site/accounts/ |
| Disallow | /blockemail/ |
| Disallow | /circ/ |
| Disallow | /comments/ |
| Disallow | /contact/ |
| Disallow | /credit/ |
| Disallow | /customer-comment-form/ |
| Disallow | /digitalcredit/ |
| Disallow | /email/ |
| Disallow | /fragment/ |
| Allow | /newsletters/$ |
| Disallow | /newsletters/ |
| Disallow | /newstart/ |
| Disallow | /profiles/ |
| Disallow | /search/ |
| Disallow | /share/ |
| Disallow | /sradmin/ |
| Disallow | /tags/ |
| Disallow | /xml/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 10 |
bingbot
| Rule | Path |
|---|---|
| Disallow | /accounts/ |
| Disallow | /site/accounts/ |
| Disallow | /circ/ |
| Disallow | /comments/ |
| Disallow | /contact/ |
| Disallow | /credit/ |
| Disallow | /customer-comment-form/ |
| Disallow | /digitalcredit/ |
| Disallow | /email/ |
| Disallow | /fragment/ |
| Allow | /newsletters/$ |
| Disallow | /newsletters/ |
| Disallow | /newstart/ |
| Disallow | /profiles/ |
| Disallow | /search/ |
| Disallow | /share/ |
| Disallow | /sradmin/ |
| Disallow | /tags/ |
| Disallow | /xml/ |
twitterbot
| Rule | Path |
|---|---|
| Disallow | /accounts/ |
| Disallow | /site/accounts/ |
| Disallow | /circ/ |
| Disallow | /comments/ |
| Disallow | /contact/ |
| Disallow | /credit/ |
| Disallow | /customer-comment-form/ |
| Disallow | /digitalcredit/ |
| Disallow | /email/ |
| Disallow | /fragment/ |
| Allow | /newsletters/$ |
| Disallow | /newsletters/ |
| Disallow | /newstart/ |
| Disallow | /profiles/ |
| Disallow | /search/ |
| Disallow | /share/ |
| Disallow | /sradmin/ |
| Disallow | /tags/ |
| Disallow | /xml/ |
Comments