Log In Sign Up

minecraftforfree.org
robots.txt

Robots Exclusion Standard data for minecraftforfree.org

Resource Scan

Scan Details

Site Domain minecraftforfree.org
Base Domain minecraftforfree.org
Scan Status Ok
Last Scan2025-11-28T18:38:17+00:00
Next Scan 2025-12-05T18:38:17+00:00

Last Scan

Scanned2025-11-28T18:38:17+00:00
URL https://minecraftforfree.org/robots.txt
Domain IPs 104.21.33.28, 172.67.140.187, 2606:4700:3030::ac43:8cbb, 2606:4700:3032::6815:211c
Response IP 172.67.140.187
Found Yes
Hash 5e1020a3d05d25513f3ae37c8330a1e7fd553756240c5473852c9147002fd212
SimHash 2e5a5b70e4c6

Groups

*

Rule Path
Allow /
Allow /play
Allow /features
Allow /downloads
Allow /news
Allow /news/*
Allow /MetaImage.png
Allow /news-images/
Allow /*.webp$
Allow /*.png$
Allow /*.jpg$
Allow /*.jpeg$
Disallow /_next/
Disallow /api/
Disallow /*.js$
Disallow /*.css$
Disallow /eaglerpocketmobile.user.js

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 0

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

chatgpt-user

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://minecraftforfree.org/sitemap.xml
Back to top

Comments

  • Robots.txt for Minecraft for Free
  • Priority crawling for main pages
  • Allow crawling of static assets
  • Disallow unnecessary files
  • Sitemap location
  • Crawl delay (be respectful)
  • Special rules for Google
  • Special rules for Bing
  • Block AI training bots if desired (optional)
Back to top