deshsanchar.com
robots.txt

Robots Exclusion Standard data for deshsanchar.com

Resource Scan

Scan Details

Site Domain deshsanchar.com
Base Domain deshsanchar.com
Scan Status Ok
Last Scan2024-05-17T22:45:29+00:00
Next Scan 2024-06-16T22:45:29+00:00

Last Scan

Scanned2024-05-17T22:45:29+00:00
URL https://deshsanchar.com/robots.txt
Domain IPs 13.33.30.34, 13.33.30.5, 13.33.30.76, 13.33.30.8, 2600:9000:229f:1000:0:b8c4:2d00:93a1, 2600:9000:229f:2c00:0:b8c4:2d00:93a1, 2600:9000:229f:3600:0:b8c4:2d00:93a1, 2600:9000:229f:9000:0:b8c4:2d00:93a1, 2600:9000:229f:9600:0:b8c4:2d00:93a1, 2600:9000:229f:be00:0:b8c4:2d00:93a1, 2600:9000:229f:d800:0:b8c4:2d00:93a1, 2600:9000:229f:f200:0:b8c4:2d00:93a1
Response IP 13.33.30.76
Found Yes
Hash 05f5f89ce35bcb03af38050696d4107cb4e60be46a5cad7dbbc7e5183e5fca56
SimHash 582c5118e1a1

Groups

*

Rule Path
Allow /

googlebot-news

Rule Path
Allow /

*

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/

bingbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

awariorssbot
awariosmartbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

peer39_crawler
peer39_crawler/1.0

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

facebookexternalhit

Rule Path
Allow /*?*smid=

twitterbot

Rule Path
Allow /*?*smid=

*

Rule Path
Disallow /?s=
Disallow /?mdrv=
Disallow /wp-admin/
Disallow /category/

Comments

  • Disallow Rules
  • Other Bot Rules