widgets.hindustantimes.com
robots.txt
Robots Exclusion Standard data for widgets.hindustantimes.com
Resource Scan
Scan Details
| Site Domain | widgets.hindustantimes.com |
| Base Domain | hindustantimes.com |
| Scan Status | Ok |
| Last Scan | 2024-11-05T09:24:45+00:00 |
| Next Scan | 2024-11-12T09:24:45+00:00 |
Last Scan
| Scanned | 2024-11-05T09:24:45+00:00 |
| URL | https://widgets.hindustantimes.com/robots.txt |
| Domain IPs | 23.52.171.120, 23.52.171.145, 2600:1413:1::6011:4848, 2600:1413:1::7d38:db30 |
| Response IP | 23.45.207.173 |
| Found | Yes |
| Hash | 956deda4d48534eb814df120e3d4d19643b967587ff98906e3979a540e48bbf0 |
| SimHash | c34059634973 |
Groups
*
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /what-now/card-details/ |
| Disallow | /*/url |
| Disallow | /*/imageURL |
| Disallow | /fragment/ |
| Disallow | /Fragment/ |
| Disallow | /Error/ |
| Disallow | /error/ |
| Disallow | /intfeeds/ |
| Disallow | /Images/HTEditImages |
| Disallow | /images/HTPopups/ |
| Disallow | /Images/Popup/ |
| Disallow | /images/Popup/ |
| Disallow | /homenew |
| Disallow | /dummytestpage/* |
| Disallow | /brand-stories/international/* |
| Disallow | /origin-pre-prod/* |
| Disallow | /sponsored-stories/* |
| Disallow | /brand-stories/* |
| Disallow | /brand-post/* |
| Disallow | /static-content/10s/us-election.html |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.hindustantimes.com/sitemap/section.xml |
| sitemap | https://www.hindustantimes.com/sitemap/news.xml |
| sitemap | https://www.hindustantimes.com/sitemap/index.xml |