thestreet.com
robots.txt
Robots Exclusion Standard data for thestreet.com
Resource Scan
Scan Details
| Site Domain | thestreet.com |
| Base Domain | thestreet.com |
| Scan Status | Ok |
| Last Scan | 2024-05-05T17:29:40+00:00 |
| Next Scan | 2024-05-12T17:29:40+00:00 |
Last Scan
| Scanned | 2024-05-05T17:29:40+00:00 |
| URL | https://thestreet.com/robots.txt |
| Redirect | https://www.thestreet.com/robots.txt |
| Redirect Domain | www.thestreet.com |
| Redirect Base | thestreet.com |
| Domain IPs | 151.101.130.98, 151.101.194.98, 151.101.2.98, 151.101.66.98 |
| Redirect IPs | 151.101.130.98, 151.101.194.98, 151.101.2.98, 151.101.66.98 |
| Response IP | 151.101.130.98 |
| Found | Yes |
| Hash | d33489607519522301fe1773ab5d06bc1525cd8761a5f57a3fa13f34fa1fe347 |
| SimHash | a505895d8213 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /.preview/ |
| Disallow | /.context-results |
| Disallow | /.applenews/ |
| Disallow | /cgi-bin |
| Disallow | /pf/ |
| Disallow | /json/ |
| Disallow | /m/pf/ |
| Disallow | /p/pf/ |
| Disallow | /i/ |
| Disallow | /b/ |
| Disallow | /print/ |
| Disallow | /find/ |
| Disallow | /email/ |
| Disallow | /press-releases/ |
| Disallow | /partner/ |
| Disallow | /storycomments/ |
| Disallow | /newsanalysis/ |
| Disallow | /_yahoo/ |
| Disallow | /_googlepi/ |
| Disallow | /_aol/ |
| Disallow | /_cnnmoney/ |
| Disallow | /_catholic/ |
| Disallow | /p/ |
| Disallow | /commerce/subscribeToNewsletter.action |
| Disallow | /static-full-na/ |
| Disallow | /embed/ |
| Disallow | /_tsc* |
| Disallow | /_rms/* |
| Disallow | /*/gdprStatus |
| Disallow | /ci0* |
| Disallow | /comment/ |
| Disallow | /s/ |
| Disallow | /funds/ |
| Disallow | /bn/ |
| Disallow | /_dm/ |
| Disallow | /user/ |
| Disallow | /*/2018-* |
| Disallow | /keyword/ |
| Disallow | /signup/ |
| Disallow | /tsc/* |
| Disallow | /util/ |
| Disallow | /k/tscir/* |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.thestreet.com/sitemaps.xml |
Comments