Log In Sign Up

asktodo.ai
robots.txt

Robots Exclusion Standard data for asktodo.ai

Resource Scan

Scan Details

Site Domain asktodo.ai
Base Domain asktodo.ai
Scan Status Ok
Last Scan2026-03-17T22:16:10+00:00
Next Scan 2026-03-24T22:16:10+00:00

Last Scan

Scanned2026-03-17T22:16:10+00:00
URL https://asktodo.ai/robots.txt
Domain IPs 89.116.170.5
Response IP 89.116.170.5
Found Yes
Hash 15d9bcd7a20699521d3d41db7750c7459085bb8db4278e2d28c8bd5054649e44
SimHash 6f805ad02ce2

Groups

*

Rule Path
Allow /
Disallow /api/
Disallow /admin/
Disallow /_next/
Disallow /static/
Allow /
Allow /tools/
Allow /blog/
Allow /blog/*
Allow /pricing
Allow /contact
Allow /privacy
Allow /terms
Allow /ads.txt
Allow /sitemap.xml
Allow /manifest.json

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Allow /
Allow /blog/
Allow /blog/*
Allow /tools/

Other Records

Field Value
crawl-delay 1

bingbot

Rule Path
Allow /
Allow /blog/
Allow /blog/*
Allow /tools/

Other Records

Field Value
crawl-delay 1

facebookexternalhit

Rule Path
Allow /
Allow /blog/
Allow /tools/

twitterbot

Rule Path
Allow /
Allow /blog/
Allow /tools/

linkedinbot

Rule Path
Allow /
Allow /blog/
Allow /tools/

mediapartners-google

Rule Path
Allow /
Allow /blog/
Allow /tools/
Back to top

Other Records

Field Value
sitemap https://asktodo.ai/sitemap.xml
Back to top

Comments

  • Sitemap location
  • Crawl-delay for respectful crawling
  • Block sensitive areas
  • Explicitly allow important pages and content
  • Block image generation folder (duplicate removed)
  • Search engine specific optimizations
  • Social media crawlers for better sharing
  • AdSense crawler
Back to top

Warnings

  • 2 invalid lines.
Back to top