Log In Sign Up

geofleet.ai
robots.txt

Robots Exclusion Standard data for geofleet.ai

Resource Scan

Scan Details

Site Domain geofleet.ai
Base Domain geofleet.ai
Scan Status Ok
Last Scan2025-09-23T13:07:05+00:00
Next Scan 2025-10-23T13:07:05+00:00

Last Scan

Scanned2025-09-23T13:07:05+00:00
URL https://geofleet.ai/robots.txt
Domain IPs 108.157.254.31, 108.157.254.59, 108.157.254.63, 108.157.254.91
Response IP 108.157.254.91
Found Yes
Hash 87b50bf59612de66424d4a2ed408eb348c6f3955d0c311a984a77c0028640fef
SimHash 61085e3365a2

Groups

*

Rule Path
Allow /
Disallow /admin/
Disallow /api/
Disallow /private/
Disallow /temp/
Disallow /_next/
Disallow /node_modules/
Disallow /*.json$
Allow /assets/
Allow /images/
Allow /css/
Allow /js/
Allow /fonts/

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

slurp

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

baiduspider

Rule Path
Allow /

semrushbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.geofleet.ai/sitemap.xml
sitemap https://www.geofleet.ai/blog-sitemap.xml
sitemap https://www.geofleet.ai/image-sitemap.xml
Back to top

Comments

  • Geofleet.ai Robots.txt - SEO Optimized
  • https://www.robotstxt.org/robotstxt.html
  • Disallow admin and private areas
  • Allow important directories
  • Crawl delay to be respectful to server resources
  • Sitemap location
  • Additional sitemaps (if you create them)
  • Allow all major search engines
  • Block bad bots and scrapers
Back to top