spokesman.com
robots.txt
Robots Exclusion Standard data for spokesman.com
Resource Scan
Scan Details
| Site Domain | spokesman.com |
| Base Domain | spokesman.com |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 2024-07-04T01:18:03+00:00 |
| Next Scan | 2024-10-02T01:18:03+00:00 |
Last Successful Scan
| Scanned | 2022-12-08T20:05:33+00:00 |
| URL | https://spokesman.com/robots.txt |
| Redirect | https://www.spokesman.com/robots.txt |
| Redirect Domain | www.spokesman.com |
| Redirect Base | spokesman.com |
| Domain IPs | 2600:9000:23d1:2000:13:3ce1:5800:93a1, 2600:9000:23d1:2a00:13:3ce1:5800:93a1, 2600:9000:23d1:4600:13:3ce1:5800:93a1, 2600:9000:23d1:4e00:13:3ce1:5800:93a1, 2600:9000:23d1:6e00:13:3ce1:5800:93a1, 2600:9000:23d1:da00:13:3ce1:5800:93a1, 2600:9000:23d1:ea00:13:3ce1:5800:93a1, 2600:9000:23d1:ec00:13:3ce1:5800:93a1, 65.9.66.101, 65.9.66.49, 65.9.66.79, 65.9.66.81 |
| Redirect IPs | 13.32.99.38, 13.32.99.43, 13.32.99.79, 13.32.99.96, 2600:9000:211e:1000:0:b5fb:e6c0:93a1, 2600:9000:211e:200:0:b5fb:e6c0:93a1, 2600:9000:211e:5000:0:b5fb:e6c0:93a1, 2600:9000:211e:9200:0:b5fb:e6c0:93a1, 2600:9000:211e:be00:0:b5fb:e6c0:93a1, 2600:9000:211e:c800:0:b5fb:e6c0:93a1, 2600:9000:211e:d000:0:b5fb:e6c0:93a1, 2600:9000:211e:e400:0:b5fb:e6c0:93a1 |
| Response IP | 18.161.97.10 |
| Found | Yes |
| Hash | d2832e662edea6157f6669953df7971341a8aa237b8119fa0b6e817783ecf8f8 |
| SimHash | e810fc10e305 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /accounts/ |
| Disallow | /site/accounts/ |
| Disallow | /blockemail/ |
| Disallow | /circ/ |
| Disallow | /comments/ |
| Disallow | /contact/ |
| Disallow | /credit/ |
| Disallow | /customer-comment-form/ |
| Disallow | /digitalcredit/ |
| Disallow | /email/ |
| Disallow | /fragment/ |
| Allow | /newsletters/$ |
| Disallow | /newsletters/ |
| Disallow | /newstart/ |
| Disallow | /profiles/ |
| Disallow | /search/ |
| Disallow | /share/ |
| Disallow | /sradmin/ |
| Disallow | /tags/ |
| Disallow | /xml/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 10 |
bingbot
| Rule | Path |
|---|---|
| Disallow | /accounts/ |
| Disallow | /site/accounts/ |
| Disallow | /circ/ |
| Disallow | /comments/ |
| Disallow | /contact/ |
| Disallow | /credit/ |
| Disallow | /customer-comment-form/ |
| Disallow | /digitalcredit/ |
| Disallow | /email/ |
| Disallow | /fragment/ |
| Allow | /newsletters/$ |
| Disallow | /newsletters/ |
| Disallow | /newstart/ |
| Disallow | /profiles/ |
| Disallow | /search/ |
| Disallow | /share/ |
| Disallow | /sradmin/ |
| Disallow | /tags/ |
| Disallow | /xml/ |
twitterbot
| Rule | Path |
|---|---|
| Disallow | /accounts/ |
| Disallow | /site/accounts/ |
| Disallow | /circ/ |
| Disallow | /comments/ |
| Disallow | /contact/ |
| Disallow | /credit/ |
| Disallow | /customer-comment-form/ |
| Disallow | /digitalcredit/ |
| Disallow | /email/ |
| Disallow | /fragment/ |
| Allow | /newsletters/$ |
| Disallow | /newsletters/ |
| Disallow | /newstart/ |
| Disallow | /profiles/ |
| Disallow | /search/ |
| Disallow | /share/ |
| Disallow | /sradmin/ |
| Disallow | /tags/ |
| Disallow | /xml/ |
Comments