Log In Sign Up

allaitools.dev
robots.txt

Robots Exclusion Standard data for allaitools.dev

Resource Scan

Scan Details

Site Domain allaitools.dev
Base Domain allaitools.dev
Scan Status Ok
Last Scan2025-12-16T20:31:08+00:00
Next Scan 2025-12-23T20:31:08+00:00

Last Scan

Scanned2025-12-16T20:31:08+00:00
URL https://allaitools.dev/robots.txt
Redirect https://www.allaitools.dev/robots.txt
Redirect Domain www.allaitools.dev
Redirect Base allaitools.dev
Domain IPs 216.24.57.1
Redirect IPs 216.24.57.251, 216.24.57.7
Response IP 216.24.57.7
Found Yes
Hash 54f247f123e0fb4cd75a0c6a2c132a4d3f1bcea60a2a44759196d71a1cae3360
SimHash 682018927461

Groups

*

Rule Path
Allow /
Allow /tools/
Allow /blog/
Allow /categories/
Allow /top-picks/
Disallow /admin/
Disallow /api/
Disallow /_next/
Disallow /*.json$
Disallow /submit-tool/thank-you

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Allow /api/og/

Other Records

Field Value
crawl-delay 0

bingbot

Rule Path
Allow /api/og/

Other Records

Field Value
crawl-delay 1

ccbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

ahrefsbot

Rule Path
Disallow /api/webhooks/

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://www.allaitools.dev/sitemap.xml
Back to top

Comments

  • AllAiTools.dev - AI Tools Directory
  • Updated: June 2025
  • Sitemap location (placed at top for better reliability)
  • Allow all crawlers to access content
  • Disallow admin and API routes
  • Disallow user-specific or temporary pages
  • Crawl delay (optional - be respectful)
  • Special rules for different bots
  • Block problematic bots (optional)
  • Disallow internal API endpoints
Back to top