Log In Sign Up

carchase.ai
robots.txt

Robots Exclusion Standard data for carchase.ai

Resource Scan

Scan Details

Site Domain carchase.ai
Base Domain carchase.ai
Scan Status Ok
Last Scan2025-12-02T16:09:03+00:00
Next Scan 2025-12-09T16:09:03+00:00

Last Scan

Scanned2025-12-02T16:09:03+00:00
URL https://carchase.ai/robots.txt
Domain IPs 104.21.15.118, 172.67.162.149, 2606:4700:3033::6815:f76, 2606:4700:3035::ac43:a295
Response IP 172.67.162.149
Found Yes
Hash ecdb6b412ca4fba3eecac09e35dd3d15b931ca3dedde07081b2994de47e98c53
SimHash 501c489ae5d3

Groups

*

Rule Path
Allow /
Disallow /admin/
Disallow /ejk4302ejk4302/
Disallow /api/
Disallow /static/flask_session/
Disallow /instance/
Disallow /*.log$
Disallow /flask_session/
Disallow /background_search_cache/
Disallow /*?
Allow /static/css/
Allow /static/js/
Allow /static/img/
Allow /static/lightning_search.css
Allow /static/lightning_search_frontend.js

semrushbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

facebookexternalhit/

Rule Path
Allow /

twitterbot

Rule Path
Allow /

linkedinbot

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://carchase.ai/cars-1.xml
Back to top

Comments

  • Block admin and private areas
  • Block duplicate/faceted search results
  • Allow important static assets
  • Block known scrapers and bad bots
  • Allow social media bots for link previews
  • Sitemap location (point to lean sitemap or sitemap index)
Back to top