directorsbox.app
robots.txt

Robots Exclusion Standard data for directorsbox.app

Resource Scan

Scan Details

Site Domain directorsbox.app
Base Domain directorsbox.app
Scan Status Ok
Last Scan2025-12-31T21:40:09+00:00
Next Scan 2026-01-07T21:40:09+00:00

Last Scan

Scanned2025-12-31T21:40:09+00:00
URL https://directorsbox.app/robots.txt
Domain IPs 34.111.179.208
Response IP 34.111.179.208
Found Yes
Hash 58d5b10f34023598a8e6a3597fca6d49c05779d4a06181b670af0219477a3c9c
SimHash 440a1a51a5f1

Groups

*

Rule Path
Allow /

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

bingbot

Rule Path
Allow /
Disallow /api/
Disallow /admin

Other Records

Field Value
crawl-delay 1

Other Records

Field Value
sitemap https://directorsbox.app/sitemap.xml

Comments

  • SEO optimized robots.txt for news website
  • Block admin and API routes from crawlers
  • Sitemap location