hindi.cnbctv18.com
robots.txt

Robots Exclusion Standard data for hindi.cnbctv18.com

Resource Scan

Scan Details

Site Domain hindi.cnbctv18.com
Base Domain cnbctv18.com
Scan Status Ok
Last Scan2025-04-06T07:01:30+00:00
Next Scan 2025-04-20T07:01:30+00:00

Last Scan

Scanned2025-04-06T07:01:30+00:00
URL https://hindi.cnbctv18.com/robots.txt
Domain IPs 23.209.46.10, 23.209.46.16
Response IP 173.222.148.35
Found Yes
Hash 5e64c6e18a23fc67648d89cf1cf046491f9d5c89f023ba41f611a77d626416b3
SimHash 611d3975c121

Groups

*

Rule Path
Allow /
Disallow /contact/?is_app=1%2Famp%2F*
Disallow /*?amp=1
Disallow /*.htmamp
Disallow /*.htmlamp
Disallow /*.htm/amp/amp
Disallow /*//amp
Disallow /*?is_app=1
Disallow /*amp//
Disallow /*1%2Famp%2F%2F

ccbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

mazbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

friendlycrawler

Rule Path
Disallow /

img2dataset

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

timpibot

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://hindi.cnbctv18.com/commonfeeds/v1/cnh/sitemap/google-news.xml
sitemap https://hindi.cnbctv18.com/commonfeeds/v1/cnh/sitemap-index.xml
sitemap https://hindi.cnbctv18.com/commonfeeds/v1/cnh/sitemap-video.xml
sitemap https://hindi.cnbctv18.com/commonfeeds/v1/cnh/sitemap-image-index.xml
sitemap https://hindi.cnbctv18.com/commonfeeds/v1/cnh/sitemap/webstories-sitemap-index.xml