Log In Sign Up

yuvagalam.org
robots.txt

Robots Exclusion Standard data for yuvagalam.org

Resource Scan

Scan Details

Site Domain yuvagalam.org
Base Domain yuvagalam.org
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a server error.
Last Scan2026-02-13T06:41:36+00:00
Next Scan 2026-05-14T06:41:36+00:00

Last Successful Scan

Scanned2025-04-20T05:47:59+00:00
URL https://yuvagalam.org/robots.txt
Redirect https://www.yuvagalam.org/robots.txt
Redirect Domain www.yuvagalam.org
Redirect Base yuvagalam.org
Domain IPs 104.21.112.1, 104.21.16.1, 104.21.32.1, 104.21.48.1, 104.21.64.1, 104.21.80.1, 104.21.96.1, 2606:4700:3030::6815:1001, 2606:4700:3030::6815:2001, 2606:4700:3030::6815:3001, 2606:4700:3030::6815:4001, 2606:4700:3030::6815:5001, 2606:4700:3030::6815:6001, 2606:4700:3030::6815:7001
Redirect IPs 104.21.112.1, 104.21.16.1, 104.21.32.1, 104.21.48.1, 104.21.64.1, 104.21.80.1, 104.21.96.1, 2606:4700:3030::6815:1001, 2606:4700:3030::6815:2001, 2606:4700:3030::6815:3001, 2606:4700:3030::6815:4001, 2606:4700:3030::6815:5001, 2606:4700:3030::6815:6001, 2606:4700:3030::6815:7001
Response IP 104.21.32.1
Found Yes
Hash b9c4c76716b262927882f0423d2e021830b76d677ea61cb40cdf34231fdb18b9
SimHash beb7c920fca0

Groups

*

Rule Path
Disallow /wp-json/
Disallow /cdn-cgi/bm/cv/
Disallow /cdn-cgi/challenge-platform/

nuclei
wikido
riddler
petalbot
zoominfobot
go-http-client
node/simplecrawler
cazoodlebot
dotbot/1.0
gigabot
barkrowler
blexbot
magpie-crawler
gptbot
google-extended
chatgpt-user
ccbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.wbhrb.in/sitemap_index.xml
Back to top

Comments

  • Prevent crawling CF challenge URLs
  • Sitemap
  • -----------------
  • Ban bots that don't benefit us.
  • --------------------------------
Back to top