Log In Sign Up

ladkibahin-yojana.org
robots.txt

Robots Exclusion Standard data for ladkibahin-yojana.org

Resource Scan

Scan Details

Site Domain ladkibahin-yojana.org
Base Domain ladkibahin-yojana.org
Scan Status Ok
Last Scan2025-10-29T02:21:27+00:00
Next Scan 2025-11-05T02:21:27+00:00

Last Scan

Scanned2025-10-29T02:21:27+00:00
URL https://ladkibahin-yojana.org/robots.txt
Domain IPs 104.21.38.80, 172.67.220.26, 2606:4700:3030::6815:2650, 2606:4700:3036::ac43:dc1a
Response IP 104.21.38.80
Found Yes
Hash 15889d89c1fc0071ca75ace8d9e1a2ab403c244a906ae2b07551565750f09fe9
SimHash 001bd8d0b6d2

Groups

googlebot

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

mediapartners-google

Rule Path
Allow /

adsbot-google

Rule Path
Allow /

bingbot

Rule Path
Allow /

slurp

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

yandex

Rule Path
Allow /

amazonbot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 15

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://ladkibahin-yojana.org/sitemap_index.xml
Back to top

Comments

  • robots.txt for ladkibahin-yojana.org
  • Purpose: Allow good crawlers, block bad bots, slow down heavy crawlers
  • ==============================
  • ​ Allow Important Search Engines & Ad Networks
  • ==============================
  • ==============================
  • ​ Block Known Scrapers & AI Training Bots
  • ==============================
  • ==============================
  • ​ Slow Down Aggressive Crawlers (Not Fully Blocked)
  • ==============================
  • ==============================
  • ​ Sitemap Location
  • ==============================
Back to top