Log In Sign Up

servicetarget.com
robots.txt

Robots Exclusion Standard data for servicetarget.com

Resource Scan

Scan Details

Site Domain servicetarget.com
Base Domain servicetarget.com
Scan Status Ok
Last Scan2025-08-17T03:25:26+00:00
Next Scan 2025-09-16T03:25:26+00:00

Last Scan

Scanned2025-08-17T03:25:26+00:00
URL https://servicetarget.com/robots.txt
Redirect https://www.servicetarget.com/robots.txt
Redirect Domain www.servicetarget.com
Redirect Base servicetarget.com
Domain IPs 75.2.70.75, 99.83.190.102
Redirect IPs 13.203.125.58, 13.233.175.166, 3.109.243.18
Response IP 35.152.117.67
Found Yes
Hash 5ffc39c0f5aa2d458278c2f889864b29211ea3ccbfe7220e90db7b727981394f
SimHash 3b1f1b03a7ab

Groups

googlebot

Rule Path
Allow /
Allow /llms.txt
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.gif$
Disallow /*.webp$
Disallow /*.svg$
Disallow /*.bmp$
Disallow /*.tiff$

bingbot

Rule Path
Allow /
Allow /llms.txt
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.png$
Disallow /*.gif$
Disallow /*.webp$
Disallow /*.svg$
Disallow /*.bmp$
Disallow /*.tiff$

duckduckbot

Rule Path
Allow /
Allow /llms.txt

chatgpt-user

Rule Path
Allow /
Allow /llms.txt

claudebot

Rule Path
Allow /
Allow /llms.txt

perplexitybot

Rule Path
Allow /
Allow /llms.txt

google-extended

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

claude-3

Rule Path
Disallow /

anthropic

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

chatgpt-crawler

Rule Path
Disallow /

gptbot-user

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

openai-searchbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

twitterbot

Rule Path
Disallow /

tiktokbot

Rule Path
Disallow /

snapchatbot

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

meta-externalfetcher

Rule Path
Disallow /

facebookexternalhit

Rule Path
Disallow /

linkedinbot

Rule Path
Disallow /

webscraper.io

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

meltwater

Rule Path
Disallow /

seekr

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

sogou

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

yandexgpt

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

youbot

Rule Path
Disallow /

youbot

Rule Path
Disallow /

amazon q

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

google-inspectiontool

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

semrushbot

Rule Path
Allow /

mozbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

googleother

Rule Path
Disallow /

timpibot

Rule Path
Disallow /

sbintuitionsbot

Rule Path
Disallow /

ai2bot

Rule Path
Disallow /

claude-user

Rule Path
Disallow /

mistralbot

Rule Path
Disallow /

neevaai

Rule Path
Disallow /

iaskspider

Rule Path
Disallow /

searchgpt

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.servicetarget.com/sitemap.xml
Back to top

Comments

  • Search engines and traffic drivers only
  • AI inference/citation systems (traffic referring only)
  • AI Training Crawlers
  • Social Media Training
  • Commercial Scrapers
  • International Search/AI (Training Focus)
  • E-commerce/Platform Crawlers
  • Archive/Research
  • SEO/Marketing Scrapers
  • Additional AI/Bot Crawlers
Back to top

Warnings

  • 2 invalid lines.
Back to top