washingtontimes.com
robots.txt
Robots Exclusion Standard data for washingtontimes.com
Resource Scan
Scan Details
| Site Domain | washingtontimes.com |
| Base Domain | washingtontimes.com |
| Scan Status | Ok |
| Last Scan | 2024-11-10T19:13:01+00:00 |
| Next Scan | 2024-11-17T19:13:01+00:00 |
Last Scan
| Scanned | 2024-11-10T19:13:01+00:00 |
| URL | https://washingtontimes.com/robots.txt |
| Redirect | https://www.washingtontimes.com:443/robots.txt |
| Redirect Domain | www.washingtontimes.com |
| Redirect Base | washingtontimes.com |
| Domain IPs | 107.22.3.120, 54.84.150.8 |
| Redirect IPs | 104.22.58.64, 104.22.59.64, 172.67.8.119 |
| Response IP | 104.22.59.64 |
| Found | Yes |
| Hash | c8004b5d24949b9048629ed418e1550e115abbe1b478f9505eab816b521182fa |
| SimHash | 2346d46281d0 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /apps/ |
| Disallow | /search/ |
| Disallow | /wire-* |
| Disallow | /budget/ |
| Disallow | /guns/ |
| Disallow | /taxes/ |
| Disallow | /users/ |
| Disallow | /accounts/ |
| Disallow | /upi-breaking/ |
| Disallow | /weather/ |
| Disallow | /account/ |
| Disallow | /voting/ |
| Disallow | /mailfriend |
| Disallow | /admin |
| Disallow | /comments |
| Disallow | /offensivecontent/ |
| Disallow | /accounts |
| Disallow | /ajax |
yandexbot
| Rule | Path |
|---|---|
| Disallow | /apps/ |
| Disallow | /search/ |
| Disallow | /wire-* |
| Disallow | /budget/ |
| Disallow | /guns/ |
| Disallow | /taxes/ |
| Disallow | /users/ |
| Disallow | /accounts/ |
| Disallow | /upi-breaking/ |
| Disallow | /weather/ |
| Disallow | /account/ |
| Disallow | /voting/ |
| Disallow | /mailfriend |
| Disallow | /admin |
| Disallow | /comments |
| Disallow | /offensivecontent/ |
| Disallow | /accounts |
| Disallow | /ajax |
spinn3r
| Rule | Path |
|---|---|
| Disallow | /apps/ |
| Disallow | /search/ |
| Disallow | /wire-* |
| Disallow | /budget/ |
| Disallow | /guns/ |
| Disallow | /taxes/ |
| Disallow | /users/ |
| Disallow | /accounts/ |
| Disallow | /upi-breaking/ |
| Disallow | /weather/ |
| Disallow | /account/ |
| Disallow | /voting/ |
| Disallow | /mailfriend |
| Disallow | /admin |
| Disallow | /comments |
| Disallow | /offensivecontent/ |
| Disallow | /accounts |
| Disallow | /ajax |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 30 |
mail.ru_bot
| Rule | Path |
|---|---|
| Disallow | /apps/ |
| Disallow | /search/ |
| Disallow | /wire-* |
| Disallow | /budget/ |
| Disallow | /guns/ |
| Disallow | /taxes/ |
| Disallow | /users/ |
| Disallow | /accounts/ |
| Disallow | /upi-breaking/ |
| Disallow | /weather/ |
| Disallow | /account/ |
| Disallow | /voting/ |
| Disallow | /mailfriend |
| Disallow | /admin |
| Disallow | /comments |
| Disallow | /offensivecontent/ |
| Disallow | /accounts |
| Disallow | /ajax |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 60 |
bingbot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /admin |
| Disallow | /newsletters/modal/ |
| Disallow | /atom/ |
| Disallow | /topics/ |
| Disallow | /media/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
msnbot
| Rule | Path |
|---|---|
| Allow | /news/ |
| Disallow | /apps/ |
| Disallow | /search/ |
| Disallow | /wire-* |
| Disallow | /budget/ |
| Disallow | /guns/ |
| Disallow | /taxes/ |
| Disallow | /users/ |
| Disallow | /accounts/ |
| Disallow | /upi-breaking/ |
| Disallow | /weather/ |
| Disallow | /account/ |
| Disallow | /voting/ |
| Disallow | /mailfriend |
| Disallow | /admin |
| Disallow | /comments |
| Disallow | /offensivecontent/ |
| Disallow | /accounts |
| Disallow | /ajax |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
applebot
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /atom/ |
| Disallow | /topics/ |
| Disallow | /media/ |
| Disallow | /news/2020/ |
| Disallow | /news/2021/ |
| Disallow | /news/2019/ |
| Disallow | /news/2018/ |
| Disallow | /news/2017/ |
| Disallow | /news/2016/ |
| Disallow | /news/2015/ |
| Disallow | /news/2014/ |
| Disallow | /news/2013/ |
| Disallow | /news/2012/ |
| Disallow | /news/2011/ |
| Disallow | /news/2010/ |
| Disallow | /news/2009/ |
| Disallow | /news/2008/ |
| Disallow | /news/2007/ |
| Disallow | /news/2006/ |
| Disallow | /news/2005/ |
| Disallow | /news/2004/ |
| Disallow | /news/2003/ |
| Disallow | /news/2002/ |
| Disallow | /news/2001/ |
| Disallow | /news/2000/ |
| Disallow | /news/1999/ |
| Disallow | /news/1998/ |
| Disallow | /news/1997/ |
| Disallow | /news/1996/ |
| Disallow | /news/1995/ |
| Disallow | /news/1994/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.washingtontimes.com/sitemap_index.xml |
| sitemap | https://www.washingtontimes.com/sitemap-stories.xml |
| sitemap | https://www.washingtontimes.com/sitemap-entries.xml |
Warnings
- `host` is not a known field.