Log In Sign Up

starklytech.com
robots.txt

Robots Exclusion Standard data for starklytech.com

Resource Scan

Scan Details

Site Domain starklytech.com
Base Domain starklytech.com
Scan Status Ok
Last Scan2025-07-18T09:48:36+00:00
Next Scan 2025-07-25T09:48:36+00:00

Last Scan

Scanned2025-07-18T09:48:36+00:00
URL https://www.starklytech.com/robots.txt
Domain IPs 2404:6800:4003:c05::79, 74.125.68.121
Response IP 142.250.4.121
Found Yes
Hash 0500eaf3f2d5cca5216f94dab34968b2c47f80bf9ff93c5226ccf901bae0c016
SimHash 19a6ab10c532

Groups

mediapartners-google

Rule Path
Disallow

gptbot

Rule Path
Allow /

claudebot

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

ccbot

Rule Path
Allow /

google-extended

Rule Path
Allow /

meta-externalagent

Rule Path
Allow /

*

Rule Path
Disallow /search*
Disallow /20*
Allow /*.html
Back to top

Other Records

Field Value
sitemap https://www.starklytech.com/sitemap.xml
sitemap https://www.starklytech.com/sitemap-pages.xml
Back to top

Comments

  • Allow Google AdSense
  • Allow GPTBot (OpenAI's crawler)
  • Allow ClaudeBot (Anthropic's crawler)
  • Allow PerplexityBot (Perplexity.ai's crawler)
  • Allow CCBot (Common Crawl)
  • Allow Google-Extended (Google's AI crawler)
  • Allow Meta's AI crawler
  • The following rules apply to all search engines: block search and archive pages, but allow all blog posts and pages.
  • General rules for all other bots
  • Sitemaps
Back to top