Log In Sign Up

ctools.org
robots.txt

Robots Exclusion Standard data for ctools.org

Resource Scan

Scan Details

Site Domain ctools.org
Base Domain ctools.org
Scan Status Ok
Last Scan2025-09-25T02:52:20+00:00
Next Scan 2025-10-25T02:52:20+00:00

Last Scan

Scanned2025-09-25T02:52:20+00:00
URL https://ctools.org/robots.txt
Domain IPs 34.111.179.208
Response IP 34.111.179.208
Found Yes
Hash 391a280fc4654fa08b8bd322915fb0425dd69838204b51d08ea4403576761f28
SimHash 0a99ea53a5b3

Groups

*

Rule Path
Allow /
Allow /article/*
Allow /category/*
Allow /api/articles
Allow /api/categories
Disallow /admin
Disallow /api/admin/*
Disallow /api/auth/*
Disallow /api/subscribe
Disallow /api/*/view

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

slurp

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

baiduspider

Rule Path
Allow /

yandexbot

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://ctools.org/sitemap.xml
sitemap https://ctools.org/news-sitemap.xml
Back to top

Comments

  • Disallow admin areas
  • Disallow private APIs
  • SEO Enhancement
  • Sitemap location
  • News sitemap for Google News
  • Allow all major search engines
Back to top