internship-application.muskurahat.org.in
robots.txt

Resource Scan

Scan Details

Site Domain internship-application.muskurahat.org.in
Base Domain muskurahat.org.in
Scan Status Ok
Last Scan2025-08-30T20:49:49+00:00
Next Scan 2025-09-29T20:49:49+00:00

Last Scan

Scanned2025-08-30T20:49:49+00:00
URL https://internship-application.muskurahat.org.in/robots.txt
Domain IPs 104.21.94.28, 172.67.218.140, 2606:4700:3033::ac43:da8c, 2606:4700:3037::6815:5e1c
Response IP 104.21.94.28
Found Yes
Hash 1dfacb75d287576bbef178f6ad1f37053a2004ac14cb1f7af02e9f84116e624f
SimHash 4804cf004753

Groups

*

Rule Path
Allow /

*

Rule Path
Disallow /intern-onboarding

Other Records

Field Value
sitemap https://www.muskurahat.org.in/sitemap.xml

Comments

  • Allow all crawlers
  • Block all crawlers for /intern-onboarding