craftr.app
robots.txt

Robots Exclusion Standard data for craftr.app

Resource Scan

Scan Details

Site Domain craftr.app
Base Domain craftr.app
Scan Status Ok
Last Scan2025-10-30T16:27:15+00:00
Next Scan 2025-11-06T16:27:15+00:00

Last Scan

Scanned2025-10-30T16:27:15+00:00
URL https://craftr.app/robots.txt
Domain IPs 65.109.99.134
Response IP 65.109.99.134
Found Yes
Hash cd5266fa444a19eeedaf45a760578da5c3185c4c241c530e86ac04795db22a1d
SimHash 6d105e75b421

Groups

*

Rule Path
Allow /
Disallow /admin/
Disallow /dashboard/
Disallow /api/
Disallow /storage/
Disallow /vendor/
Disallow /node_modules/
Disallow /tests/
Disallow /database/
Disallow /config/
Disallow /bootstrap/
Disallow /app/
Disallow /resources/
Disallow /routes/
Disallow /storage/
Disallow /temp/
Allow /css/
Allow /js/
Allow /images/
Allow /fonts/

openai

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

Other Records

Field Value
sitemap https://craftr.app/sitemap.xml
sitemap https://craftr.app/sitemap

Comments

  • Sitemaps
  • Disallow admin and private areas
  • Allow important files
  • AI User Agents
  • Crawl delay (optional)