tucsonsentinel.com
robots.txt
Robots Exclusion Standard data for tucsonsentinel.com
Resource Scan
Scan Details
| Site Domain | tucsonsentinel.com |
| Base Domain | tucsonsentinel.com |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 2024-08-26T20:57:43+00:00 |
| Next Scan | 2024-11-24T20:57:43+00:00 |
Last Successful Scan
| Scanned | 2023-11-01T20:40:01+00:00 |
| URL | https://tucsonsentinel.com/robots.txt |
| Redirect | https://www.tucsonsentinel.com/robots.txt |
| Redirect Domain | www.tucsonsentinel.com |
| Redirect Base | tucsonsentinel.com |
| Domain IPs | 172.66.40.173, 172.66.43.83, 2606:4700:3108::ac42:28ad, 2606:4700:3108::ac42:2b53 |
| Redirect IPs | 172.66.40.173, 172.66.43.83, 2606:4700:3108::ac42:28ad, 2606:4700:3108::ac42:2b53 |
| Response IP | 172.66.43.83 |
| Found | Yes |
| Hash | 7b163e2000c001e7e9d7b597dc14807cd2347e1d420339eb1aa894016e5b3be9 |
| SimHash | 2c1a5901e5b5 |
Groups
*
| Rule | Path |
|---|---|
| Allow | / |
*
| Rule | Path |
|---|---|
| Disallow | /events/ |
*
| Rule | Path |
|---|---|
| Disallow | /calendar/ |
*
| Rule | Path |
|---|---|
| Disallow | /forums/ |
*
| Rule | Path |
|---|---|
| Disallow | /reader/login/ |
*
No rules defined. All paths allowed.
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
yahoomobile
yahoocachesystem
yahoo! slurp/site explorer
mozilla/4.05 [en]
lti/lemurproject
yahoo-blogs
yahoo-blogs/v3.9
yahoo-mmcrawler
yahoo-mmcrawler/3.x
yahooysmcm
yahooysmcm/2.0.0
yahoo-test
yahoo! mindset
y!j-bsc
y!j-bsc/1.0
y!j-bsc/1.0(http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
y!j-bsc
y!j-bsc/1.0
y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
y!j
y!j/1.0
y!j/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
y!j
y!j/1.0
y!j/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
mozilla/4.0 (compatible; y!j; for robot study; keyoshid)
mozilla/4.0 (compatible; y!j; for robot study; keyoshid)
mozilla/5.0 (compatible; yahoo! slurp china; http://misc.yahoo.com.cn/help.html)
mozilla/5.0 (compatible; yahoo! de slurp; http://help.yahoo.com/help/us/ysearch/slurp)
mozilla/5.0 (yahoo-test/4.0 mailto:vertical-crawl-support@yahoo-inc.com)
mozilla/5.0 (compatible; yahoo! slurp; http://help.yahoo.com/help/us/ysearch/slurp)
| Rule | Path |
|---|---|
| Disallow | / |
Other Records
| Field | Value |
|---|---|
| sitemap | https://tucsonsentinel.com/news_sitemap.php |
| sitemap | https://tucsonsentinel.com/sitemap.php |
| sitemap | https://tucsonsentinel.com/sitemap_index.xml |
Warnings
- 2 invalid lines.
Comments