Log In Sign Up

cstsetup.com
robots.txt

Robots Exclusion Standard data for cstsetup.com

Resource Scan

Scan Details

Site Domain cstsetup.com
Base Domain cstsetup.com
Scan Status Ok
Last Scan2025-09-17T22:20:46+00:00
Next Scan 2025-09-24T22:20:46+00:00

Last Scan

Scanned2025-09-17T22:20:46+00:00
URL https://cstsetup.com/robots.txt
Domain IPs 104.21.64.137, 172.67.151.52, 2606:4700:3031::6815:4089, 2606:4700:3037::ac43:9734
Response IP 104.21.64.137
Found Yes
Hash 08fd0c05ac07d1ae90901a4293242da0cacad6ff16734864fac1e3ec8a85bc28
SimHash 4538da40e5e7

Groups

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

baiduspider

Rule Path
Allow /

yandexbot

Rule Path
Allow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

*

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://cstsetup.com/wp-sitemap.xml
Back to top

Comments

  • Allow Google and Bing search engine crawlers
  • Allow other common search engines
  • Disallow Google's AI model training crawler
  • Disallow OpenAI's ChatGPT
  • Disallow Claude AI
  • General rule for all other bots
  • Block WP App Ninja plugin urls
  • User-agent: *
  • Disallow: /*?wpappninja*
  • Disallow: /*?*is_wppwa=*
  • Disallow: /*?*wpappninja_cache=*
  • Disallow: /*?*wpappninja_v=*
  • Disallow: /*?*replytocom=*
  • Sitemap reference for better indexing by allowed crawlers
Back to top