hitcraft.ai
robots.txt

Robots Exclusion Standard data for hitcraft.ai

Resource Scan

Scan Details

Site Domain hitcraft.ai
Base Domain hitcraft.ai
Scan Status Ok
Last Scan2025-11-18T00:05:55+00:00
Next Scan 2025-12-18T00:05:55+00:00

Last Scan

Scanned2025-11-18T00:05:55+00:00
URL https://hitcraft.ai/robots.txt
Domain IPs 108.156.144.112, 108.156.144.13, 108.156.144.36, 108.156.144.75
Response IP 108.156.144.75
Found Yes
Hash 39ea1a7c1a097b32eb8c54b6b6d5f9e92fa17e54677c6ac0bb683a377b177497
SimHash a7981870c421

Groups

*

Rule Path
Allow /
Allow /about
Allow /why-hitcraft
Allow /pricing
Allow /contact
Allow /articles/terms-of-use
Disallow /login
Disallow /oauth/
Disallow /oauth/callback
Disallow /api/
Disallow /*.js$
Disallow /*.css$
Disallow /assets/js/
Disallow /assets/css/
Allow /assets/images/
Allow /sitemap.xml
Allow /favicon.ico

Other Records

Field Value
crawl-delay 1

Other Records

Field Value
sitemap https://hitcraft.ai/sitemap.xml

Comments

  • HitCraft Website Robots Configuration
  • Allow crawling of main pages
  • Prevent crawling of auth-related pages
  • Prevent crawling of development assets
  • Allow crawling of important assets
  • Sitemap location
  • Crawl-delay for rate limiting