Log In Sign Up

consolidata.ai
robots.txt

Robots Exclusion Standard data for consolidata.ai

Resource Scan

Scan Details

Site Domain consolidata.ai
Base Domain consolidata.ai
Scan Status Ok
Last Scan2025-10-13T06:16:00+00:00
Next Scan 2025-11-12T06:16:00+00:00

Last Scan

Scanned2025-10-13T06:16:00+00:00
URL https://consolidata.ai/robots.txt
Domain IPs 13.215.239.219, 2406:da18:b3d:e201::258, 2406:da18:b3d:e201::259, 52.74.6.109
Response IP 13.215.239.219
Found Yes
Hash 17f37ee30803c7a6cba802e059ab1416bbc8cc78b4b8a5ac3b031201ae8c0d97
SimHash ea00daaa24f1

Groups

*

Rule Path
Allow /
Disallow /admin-portal
Disallow /admin-login
Disallow /admin-portal/*
Disallow /thank-you
Disallow /purchase-thank-you
Allow /
Allow /identity-resolution
Allow /identity-resolution-funnel
Allow /gmb-tank-tracking
Allow /blog
Allow /blog/*
Allow /contact
Allow /privacy-policy
Allow /terms-of-service

Other Records

Field Value
crawl-delay 1
Back to top

Other Records

Field Value
sitemap https://yourdomain.com/sitemap.xml
Back to top

Comments

  • Disallow admin pages
  • Disallow thank you pages (no SEO value)
  • Allow important pages
  • Sitemap location
  • Crawl delay (optional - be respectful to search engines)
Back to top