dbe-du.org
robots.txt

Robots Exclusion Standard data for dbe-du.org

Resource Scan

Scan Details

Site Domain dbe-du.org
Base Domain dbe-du.org
Scan Status Ok
Last Scan2025-09-27T01:46:25+00:00
Next Scan 2025-10-27T01:46:25+00:00

Last Scan

Scanned2025-09-27T01:46:25+00:00
URL https://dbe-du.org/robots.txt
Redirect https://www.dbe-du.org/robots.txt
Redirect Domain www.dbe-du.org
Redirect Base dbe-du.org
Domain IPs 216.198.79.1, 64.29.17.65
Redirect IPs 64.29.17.1, 64.29.17.65
Response IP 216.198.79.1
Found Yes
Hash 987c52f25732c9c37c4a4a5ce28733d344600ce14e5973d203d73a2c42d946b3
SimHash 4010cc906252

Groups

*

Rule Path
Allow /

Comments

  • www.robotstxt.org/
  • Allow all crawlers
  • Disallow crawling of specific folders (uncomment if needed)
  • Disallow: /admin/
  • Crawl-delay for all bots (optional, in seconds)
  • Crawl-delay: 10
  • Block specific bots (if needed)
  • User-agent: BadBot
  • Disallow: /