epaper.divyabhaskar.co.in
robots.txt

Robots Exclusion Standard data for epaper.divyabhaskar.co.in

Resource Scan

Scan Details

Site Domain epaper.divyabhaskar.co.in
Base Domain divyabhaskar.co.in
Scan Status Ok
Last Scan2024-05-23T20:34:54+00:00
Next Scan 2024-06-06T20:34:54+00:00

Last Scan

Scanned2024-05-23T20:34:54+00:00
URL https://epaper.divyabhaskar.co.in/robots.txt
Redirect https://www.divyabhaskar.co.in:443/epaper/robots.txt
Redirect Domain www.divyabhaskar.co.in
Redirect Base divyabhaskar.co.in
Domain IPs 108.157.254.100, 108.157.254.126, 108.157.254.79, 108.157.254.92
Redirect IPs 2600:9000:2003:200:1d:b19e:e900:93a1, 2600:9000:2003:4000:1d:b19e:e900:93a1, 2600:9000:2003:6400:1d:b19e:e900:93a1, 2600:9000:2003:6e00:1d:b19e:e900:93a1, 2600:9000:2003:b800:1d:b19e:e900:93a1, 2600:9000:2003:c00:1d:b19e:e900:93a1, 2600:9000:2003:d400:1d:b19e:e900:93a1, 2600:9000:2003:f800:1d:b19e:e900:93a1, 52.84.229.12, 52.84.229.124, 52.84.229.43, 52.84.229.67
Response IP 52.84.229.124
Found Yes
Hash 5ebd632b2630fce9ee6986c3bfd00430ec66497b6db9e603a249a123d8a48406
SimHash 4000c9488135

Groups

*

Rule Path
Allow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /