Log In Sign Up

bcmpedia.org
robots.txt

Robots Exclusion Standard data for bcmpedia.org

Resource Scan

Scan Details

Site Domain bcmpedia.org
Base Domain bcmpedia.org
Scan Status Ok
Last Scan2025-11-12T20:49:04+00:00
Next Scan 2025-12-12T20:49:04+00:00

Last Scan

Scanned2025-11-12T20:49:04+00:00
URL https://bcmpedia.org/robots.txt
Domain IPs 103.7.8.87
Response IP 103.7.8.87
Found Yes
Hash 69d4195ef9ed15f32720d6437d39c39ec9bc5dccd3ad397aa9adf8d35fec7299
SimHash 60148c522741

Groups

googlebot

Rule Path
Disallow

bingbot

Rule Path
Disallow

ahrefsbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

*

Rule Path
Disallow /wiki/Special%3ASearch
Disallow /wiki/Special%3ARandom
Disallow /wiki/Special%3A
Disallow /w/
Disallow /favicon.ico

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://bcmpedia.org/w/sitemap.xml
Back to top

Comments

  • User-agent: *
  • Disallow: /wiki/Special:Search
  • Disallow: /wiki/Special:Random
  • Allow major search engines full access
  • Restrict overly aggressive crawlers
  • Global rules for all other bots
  • Point crawlers to sitemap
Back to top