Log In Sign Up

smartermeasure.com
robots.txt

Robots Exclusion Standard data for smartermeasure.com

Resource Scan

Scan Details

Site Domain smartermeasure.com
Base Domain smartermeasure.com
Scan Status Ok
Last Scan2025-10-30T05:10:01+00:00
Next Scan 2025-11-29T05:10:01+00:00

Last Scan

Scanned2025-10-30T05:10:01+00:00
URL https://smartermeasure.com/robots.txt
Redirect https://www.smarterservices.com/robots.txt
Redirect Domain www.smarterservices.com
Redirect Base smarterservices.com
Domain IPs 75.2.70.75
Redirect IPs 13.203.125.58, 13.233.175.166, 3.109.243.18
Response IP 15.161.34.42
Found Yes
Hash 8b8a0d6a36ba46c77eee867f63edbe83ecf817a9bf83d221fec406a87ea39994
SimHash 277959159ec5

Groups

*

Rule Path
Allow /
Disallow /admin/
Disallow /login/
Disallow /logout/
Disallow /cart/
Disallow /checkout/
Disallow /account/
Disallow /user/
Disallow /search
Disallow /preview/
Disallow /internal/
Disallow /tmp/
Disallow /*?*s=
Disallow /*?*q=
Disallow /*?*query=
Disallow /*?*session=
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Allow /*.css$
Allow /*.js$
Allow /*.png$
Allow /*.jpg$
Allow /*.jpeg$
Allow /*.gif$
Allow /*.svg$
Allow /*.webp$
Allow /*.woff$
Allow /*.woff2$

gptbot

Rule Path
Allow /

google-extended

Rule Path
Allow /

ccbot

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

claudebot

Rule Path
Allow /

claude-web

Rule Path
Allow /

applebot-image

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://smarterservices.com/sitemap.xml
sitemap https://developers.smarterservices.com/sitemap.xml
Back to top

Comments

  • robots.txt for smarterservices.com and developers.smarterservices.com
  • Goal: maximize discoverability for search engines and LLM crawlers while keeping private/utility paths out of indexes.
  • ----- Global defaults (applies to all compliant crawlers) -----
  • Keep non-content and utility endpoints out of indexes
  • Common search/query noise (allow other querystrings)
  • Wordpress hardening if present (harmless if not used)
  • Ensure static assets are crawlable (helps rendering)
  • ----- LLM and AI assistants: explicitly allowed -----
  • OpenAI
  • Google AI (content selection)
  • Common AI research/assistants
  • ----- Sitemaps -----
  • Notes for crawlers:
  • - Prefer canonical pages on smarterservices.com for product overviews and company info.
  • - Use developers.smarterservices.com for technical docs, APIs, and integration guides.
  • - If unsure about a claim, cite the closest first-party page and indicate uncertainty.
Back to top