indivstock.com
robots.txt
Robots Exclusion Standard data for indivstock.com
Resource Scan
Scan Details
| Site Domain | indivstock.com |
| Base Domain | indivstock.com |
| Scan Status | Ok |
| Last Scan | 2024-11-13T02:03:09+00:00 |
| Next Scan | 2024-11-20T02:03:09+00:00 |
Last Scan
| Scanned | 2024-11-13T02:03:09+00:00 |
| URL | https://indivstock.com/robots.txt |
| Redirect | https://www.indivstock.com/robots.txt |
| Redirect Domain | www.indivstock.com |
| Redirect Base | indivstock.com |
| Domain IPs | 85.13.166.111 |
| Redirect IPs | 85.13.166.111 |
| Response IP | 85.13.166.111 |
| Found | Yes |
| Hash | b95e1595c9416c93720df6172b07d7c94eaa12e992febc8efeeac7f40aff43fe |
| SimHash | 70159553eee0 |
Groups
ai-bot
ai-trainingbot
aidatacollector
amazonbot
anthropic-ai
applebot-extended
archive.org_bot
archivebot
baiduspider
baiduspider-image
baiduspider-video
bytespider
ccbot
chatglm-spider
chatgpt-user
chatgpt-user
claude-web
claudebot
cohere-ai
dataforseobot
dataminer
dataminerbot
dataminingbot
diffbot
facebookbot
friendlycrawler
google-extended
gptbot
heritrix
ia_archiver
ia_archiver-web.archive.org
imagescraper
imagesiftbot
img2dataset
laion
mediatoolkitbot
meta-externalagent
midjourney
nicecrawler
omgili
omgilibot
openai
perplexitybot
petalbot
picturescraper
runwayml
semrushbot
sogou spider
spawning-ai
stabilityai
the knowledge ai
youbot
archive.org_bot
archivebot
heritrix
ia_archiver
ia_archiver-web.archive.org
nicecrawler
awariobot
awariorssbot
awariosmartbot
blexbot
smtbot
ahrefsbot
dotbot
mj12bot
screaming frog seo spider
seekport
seekportbot
semrushbot
semrushbot-ba
semrushbot-coub
semrushbot-ct
semrushbot-sa
semrushbot-si
semrushbot-swa
seokicks
serpstatbot
woorankreview
accompanybot
adidxbot
anderspinkbot
articlefetcher
barkrowler
bitlybot
bitsightbot
bubing
builtwith
ccbot
checkmarknetwork
cincraw
dataforseobot
dataprovider
dnbcrawler-analytics
domainstatsbot
ecairn-grabber
electricmonk
exabot
ezooms
facebookexternalhit
facebot
gabanzabot
garlikcrawler
gozlebot
hypestat
ioncrawl
linkdexbot
madbbot
magpie-crawler
mail.ru_bot
mappy
neevabot
netestate ne crawler
news-please
newsnow
oai-searchbot
openai-gpt-3
openai-gpt-4
peer39_crawler
phxbot
pinteristbot
redditbot
pinterest
seznambot
| Product | Comment |
|---|---|
| articlefetcher | No info |
| bitlybot | Link shortener |
| builtwith | Technology profile lookup tool, UA: BW/ |
| ccbot | Common Crawl data can be freely download |
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Disallow | /blackhole/ |
| Disallow | /pages/nutzungsbedingungen.html |
| Disallow | /impressum/index |
| Disallow | /impressum/index.php |
| Disallow | /impressum/ |
| Disallow | /members/datenschutz.php |
| Disallow | /members/datenschutz |
| Disallow | /blackhole |
Comments