bengali.news18.com
robots.txt
Robots Exclusion Standard data for bengali.news18.com
Resource Scan
Scan Details
| Site Domain | bengali.news18.com |
| Base Domain | news18.com |
| Scan Status | Ok |
| Last Scan | 2024-05-08T11:22:29+00:00 |
| Next Scan | 2024-06-07T11:22:29+00:00 |
Last Scan
| Scanned | 2024-05-08T11:22:29+00:00 |
| URL | https://bengali.news18.com/robots.txt |
| Domain IPs | 23.199.151.73, 2600:1413:1:98a::3393 |
| Response IP | 23.199.151.73 |
| Found | Yes |
| Hash | 30d3933a7bb12d13bc68e48e6d385f2f07bae9555eb30bfef33f69975eb420aa |
| SimHash | e800d1d0ed97 |
Groups
*
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /events/general-election-2019/ |
| Disallow | /api/ |
| Disallow | /includes/ |
| Disallow | /SolrPhpClient/ |
| Disallow | /1039154/ |
| Disallow | /uc/ |
| Disallow | /mutual-funds/ |
| Disallow | /insurance/ |
| Disallow | /auto/ |
| Disallow | /footer_ajax_mobile.html |
| Disallow | /tag/%*/ |
| Disallow | /tag/*/page-*/ |
| Disallow | /tag/*/photogallery/page-*/ |
| Disallow | /tag/*/videos/page-*/ |
| Disallow | /tag/*/news/page-*/ |
| Disallow | /amp/tag/%*/ |
| Disallow | /amp/tag/*/page-*/ |
| Disallow | /amp/tag/*/photogallery/page-*/ |
| Disallow | /durga-puja-2017/* |
| Disallow | /byline/*/news/page-*/$ |
| Disallow | /byline/*/photogallery/page-*/$ |
| Disallow | /byline/*/videos/page-*/$ |
| Disallow | /byline/*/page-*/$ |
| Disallow | /byline/*/news/ |
| Disallow | /photogallery/*page-1$ |
| Disallow | /photogallery/page-* |
| Disallow | /photogallery/*/page-*/$ |
| Disallow | /durga-puja-2018/* |
| Disallow | /press-release/page-*/$ |
| Disallow | /poila-boisakh-celeb/page-*/$ |
| Disallow | /vod/ |
| Disallow | /*tag/*.* |
| Disallow | /*tag/*%3A* |
| Disallow | /*tag/*%2B* |
| Disallow | /*tag/*%29* |
| Disallow | /*tag/*%28* |
| Disallow | /*page-1page* |
| Disallow | /*%* |
| Disallow | /scorecard/ |
| Disallow | /*/page-*/privacy_policy.html |
| Disallow | /*/page-*/page-*/ |
| Disallow | /videos-sitemap |
| Disallow | /image-sitemap |
| Disallow | /articles-sitemap |
| Disallow | /*/privacy_policy.html/$ |
| Disallow | /news/*/page-* |
| Disallow | /videos/page-*/ |
| Disallow | /durga-puja-2023/ |
| Disallow | /embed/ |
| Disallow | /budget-2021/ |
| Disallow | /videos/*/page-*/$ |
| Disallow | /west-bengal-assembly-elections-2016/ |
| Disallow | /ampphotogallery/ |
| Disallow | /assembly-elections-2023/ |
| Disallow | /elections/assembly-elections-2018/ |
| Disallow | /photogallery/*/privacy_policy.html |
| Disallow | /ipl-2020/ |
| Disallow | /1 |
| Disallow | /2 |
| Disallow | /3 |
| Disallow | /4 |
| Disallow | /5 |
| Disallow | /6 |
| Disallow | /7 |
| Disallow | /8 |
| Disallow | /9 |
| Disallow | /0 |
| Disallow | /amp/tag/1 |
| Disallow | /amp/tag/2 |
| Disallow | /amp/tag/3 |
| Disallow | /amp/tag/4 |
| Disallow | /amp/tag/5 |
| Disallow | /amp/tag/6 |
| Disallow | /amp/tag/7 |
| Disallow | /amp/tag/8 |
| Disallow | /amp/tag/9 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://bengali.news18.com/commonfeeds/v1/ben/sitemap/today |
| sitemap | https://bengali.news18.com/commonfeeds/v1/ben/sitemap-index.xml |
| sitemap | https://bengali.news18.com/commonfeeds/v1/ben/sitemap/google-news.xml |
| sitemap | https://bengali.news18.com/commonfeeds/v1/ben/sitemap-image-index.xml |
| sitemap | https://bengali.news18.com/commonfeeds/v1/ben/sitemap-video-index.xml |
| sitemap | https://bengali.news18.com/commonfeeds/v1/ben/sitemap/webstories-sitemap-index.xml |