hindi.news18.com
robots.txt

Robots Exclusion Standard data for hindi.news18.com

Resource Scan

Scan Details

Site Domain hindi.news18.com
Base Domain news18.com
Scan Status Ok
Last Scan2024-09-19T07:12:12+00:00
Next Scan 2024-10-03T07:12:12+00:00

Last Scan

Scanned2024-09-19T07:12:12+00:00
URL https://hindi.news18.com/robots.txt
Domain IPs 23.36.49.159, 2600:1413:b000:389::3393, 2600:1413:b000:398::3393
Response IP 23.54.57.187
Found Yes
Hash 845ed6d2470b38f2ed0aedc1d464b2fb0e8888a66bf2577a3a77ca5bd25a504b
SimHash 6c0eb15bc4a3

Groups

*

Rule Path
Allow /
Disallow *.htmlnews/*/$
Disallow */%3Astate_slug%3A/*
Disallow /cricket/live-score/undefined-*.html
Disallow /hindi.news18.com/news/*.html$
Disallow /hindi.news18.com/amp/news/*.html$
Disallow /shows/
Disallow /events/general-election-2019/*
Disallow /amp/assembly-election-2022/*
Disallow /assembly-elections-2018/*
Disallow /image-sitemap-*.xml
Disallow /election-result-2013/*
Disallow /photogallery/*/*/page-*/
Disallow /api/*
Disallow /videos-sitemap-*.xml
Disallow /amp/assembly-elections-2022/*
Disallow /assembly-elections-2022/*
Disallow /conversations/
Disallow /marketing/
Disallow /1039154/
Disallow /test/
Disallow /uc/
Disallow /home.html
Disallow /jw_player/
Disallow /scroll/
Disallow /embedded/
Disallow /contest/
Disallow /inner-site/
Disallow /pwa/
Disallow /news/education/uc/
Disallow /history-18/
Disallow /uc-election-widget/
Disallow /mutual-funds/
Disallow /insurance/
Disallow /automobile/
Disallow /googlenewssitemaptest.xml
Disallow /nodeapi/
Disallow /tag/%*/
Disallow /tag/*/page-*/
Disallow /tag/*/photogallery/*
Disallow /tag/*/videos/*
Disallow /tag/*/news/*
Disallow /byline/%*/
Disallow /byline/*/photogallery/page-*/
Disallow /byline/*/videos/page-*/
Disallow /byline/*/news/page-*/
Disallow /agency/%*/
Disallow /agency/*/page-*/
Disallow /agency/*/photogallery/page-*/
Disallow /agency/*/videos/page-*/
Disallow /agency/*/news/page-*/
Disallow /etv-shows/
Disallow /videos/*/page-*/
Disallow /videos/*/*/page-*/
Disallow /byline/*htmlpage-*html/page-*/
Disallow /byline/*htmlpage-*/page-*/
Disallow /byline/*htmlpage-*/photogallery/
Disallow /byline/*/page-*/tag/*/
Disallow */access_log
Disallow */error_log
Disallow /tag/*void-0*/
Disallow /videos/videos/*
Disallow /news/*/*html/$
Disallow /astrology/
Disallow /byline/*page-*/$
Disallow /anchor/*page-*/$
Disallow /assembly-election-2020/*
Disallow /news/*page-*/$
Disallow /assembly-election-2021/*
Disallow /assembly-election-2019/*
Disallow /news/*htmlpage-*.html/$
Disallow /amp/astrology/*/
Disallow /tag/*/1*page--*/
Disallow /rajya-sabha-elections-2020/
Disallow /Tag/*/page-*/
Disallow /tag//*/
Disallow /union-budget-2018/
Disallow /amp/assembly-election-2020/
Disallow /assembly-elections/
Disallow /news/*/page-*/
Disallow /cricket/profile//*.html
Disallow /embed/videos/*%3D%3D/
Disallow /vod/*/livestreaming/
Disallow /madhya-pradesh/cricket/page-*/
Disallow /uttar-pradesh/page-*/
Disallow /app-download/
Disallow /podcast/news-bulletin/page-*/
Disallow /podcast/sports-buletin/page-*/
Disallow /agency/*/news/
Disallow /agency/*/photogallery/
Disallow /agency/*/videos/
Disallow /agency/*page-*/
Disallow /_next/image/
Disallow /cricket/series/*/page-*/
Disallow /cricket/profile/*/news-*/page-*/
Disallow /podcast/
Disallow /cricket/match-schedule/series/*
Disallow /lok-sabha-election-*/
Disallow /amp/assembly-election-*/
Disallow /states/blogs/page-*/
Disallow /rajasthan/*/page-*/
Disallow /cricket/live-score/team-news/*.html
Disallow /uttar-pradesh/*/page-*/
Disallow /bihar-assembly-election-*/
Disallow /amp/bihar-assembly-election-*/
Disallow /amp-bihar-assembly-election-*/
Disallow /news/uttar-pradesh/*/news
Disallow /byline/*/news
Disallow /amp/news/*/photogallery/
Disallow /news/*/photogallery/
Disallow /amp/states/news/
Disallow /states/news/
Disallow /asian-games/*/
Disallow /*/page-*/
Disallow /cricketnext/
Disallow /amp/cricketnext/
Disallow /states/static-redirect.php
Disallow /news/byline/*/
Disallow /sitemaps/daily/*.xml
Disallow /short-news/*a
Disallow /byline/*/photogallery/
Disallow /ampdesk-/news//*.html
Disallow /states/redirect.php?id=*

ccbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

mazbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

friendlycrawler

Rule Path
Disallow /

img2dataset

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

timpibot

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

Other Records

Field Value
sitemap https://hindi.news18.com/allstory-sitemap-data.xml
sitemap https://hindi.news18.com/commonfeeds/v1/hin/sitemap/google-news/yesterday.xml
sitemap https://hindi.news18.com/commonfeeds/v1/hin/sitemap/google-news/today.xml