Log In Sign Up

st.agentiz.com
robots.txt

Robots Exclusion Standard data for st.agentiz.com

Resource Scan

Scan Details

Site Domain st.agentiz.com
Base Domain agentiz.com
Scan Status Ok
Last Scan2025-12-26T03:52:40+00:00
Next Scan 2026-01-02T03:52:40+00:00

Last Scan

Scanned2025-12-26T03:52:40+00:00
URL https://st.agentiz.com/robots.txt
Domain IPs 104.21.23.178, 172.67.212.117, 2606:4700:3034::ac43:d475, 2606:4700:3036::6815:17b2
Response IP 172.67.212.117
Found Yes
Hash 159b6f169ddb9035d8375abd5a82e1f4a5f13f949eb8957e48ac37e26454fc5d
SimHash 4c3151f449fd

Groups

googlebot
googlebot-image
googleother
bingbot
bingpreview
msnbot
applebot
duckduckbot
ecosiabot
pinterestbot
seznambot
qwantify
exabot
orangebot
slurp
yeti
daumoa
coccocbot
yaanibot
facebookexternalhit
facebookcatalog
instagram
linkedinbot
twitterbot
meta-externalfetcher
telegrambot
viber
whatsapp
cloudflareobservatory
feedfetcher-google
google-imageproxy
google-read-aloud
google-site-verification
mediapartners-google
adsbot-google
adsbot-msn
uptimerobot
chatgpt-user
oai-searchbot
claude-web
perplexitybot
youbot
phindbot
kagibot
grokbot
xai-grok

Rule Path
Allow /
Disallow /*-xs.jpg
Disallow /*-xs.webp
Disallow /*-xxs.jpg
Disallow /*-xxs.webp
Disallow /*-xxxs.jpg
Disallow /*-xxxs.webp
Disallow /*/?type=*
Disallow /*/?search=*
Disallow /assets
Disallow /*/assets

bytespider
ahrefsbot
ahrefssiteaudit
mj12bot
semrushbot
gptbot
ccbot
google-extended

Rule Path
Disallow /

*

Rule Path
Allow /$
Disallow /
Back to top

Other Records

Field Value
sitemap https://st.agentiz.com/sitemap.xml
Back to top

Comments

  • ST st.agentiz.com
  • As a condition of accessing this website, automated agents (including crawlers,
  • bots, scrapers, AI systems, and aggregators) agree to the following:
  • 1) You may crawl this website only to the extent explicitly permitted by the
  • rules below (User-agent / Allow / Disallow).
  • 2) You may use our content for:
  • - search indexing and displaying search results (including links, titles,
  • and short snippets), where allowed by robots.txt;
  • - AI input / retrieval (for answering user queries in real time),
  • where allowed by robots.txt and applicable content signals.
  • 3) You may NOT:
  • - use our content for training or fine-tuning AI models;
  • - perform bulk scraping or large-scale copying for the purpose of building
  • commercial databases, aggregators, or mirrored listing portals;
  • - republish, resell, or systematically reuse our content without our
  • prior written consent.
  • Any restrictions expressed in this robots.txt file (including content signals)
  • constitute an express reservation of rights, including under Article 4 of
  • Directive (EU) 2019/790 and any similar laws in other jurisdictions.
  • Additional permissions for specific bots or services are granted only where
  • they are explicitly listed as allowed in the rules below.
  • For questions, partnership requests, or additional permissions,
  • please contact us at:
  • https://www.agentiz.com/en/support/feedback
Back to top

Warnings

  • `content-signal` is not a known field.
Back to top