thailand-faq.de
robots.txt
Robots Exclusion Standard data for thailand-faq.de
Resource Scan
Scan Details
| Site Domain | thailand-faq.de |
| Base Domain | thailand-faq.de |
| Scan Status | Ok |
| Last Scan | 2024-11-18T14:44:26+00:00 |
| Next Scan | 2024-11-25T14:44:26+00:00 |
Last Scan
| Scanned | 2024-11-18T14:44:26+00:00 |
| URL | https://thailand-faq.de/robots.txt |
| Redirect | https://www.thailand-faq.de/robots.txt |
| Redirect Domain | www.thailand-faq.de |
| Redirect Base | thailand-faq.de |
| Domain IPs | 85.13.163.43 |
| Redirect IPs | 85.13.163.43 |
| Response IP | 85.13.163.43 |
| Found | Yes |
| Hash | ed60ab767eed56838087731e4545b3e7d83ed7b3f8ec2c85b81287cb34972578 |
| SimHash | 70159553eee0 |
Groups
ai-bot
ai-trainingbot
aidatacollector
amazonbot
anthropic-ai
applebot-extended
archive.org_bot
archivebot
baiduspider
baiduspider-image
baiduspider-video
bytespider
ccbot
chatglm-spider
chatgpt-user
chatgpt-user
claude-web
claudebot
cohere-ai
dataforseobot
dataminer
dataminerbot
dataminingbot
diffbot
facebookbot
friendlycrawler
google-extended
gptbot
heritrix
ia_archiver
ia_archiver-web.archive.org
imagescraper
imagesiftbot
img2dataset
laion
mediatoolkitbot
meta-externalagent
midjourney
nicecrawler
omgili
omgilibot
openai
perplexitybot
petalbot
picturescraper
runwayml
semrushbot
sogou spider
spawning-ai
stabilityai
the knowledge ai
youbot
archive.org_bot
archivebot
heritrix
ia_archiver
ia_archiver-web.archive.org
nicecrawler
awariobot
awariorssbot
awariosmartbot
blexbot
smtbot
ahrefsbot
dotbot
mj12bot
screaming frog seo spider
seekport
seekportbot
semrushbot
semrushbot-ba
semrushbot-coub
semrushbot-ct
semrushbot-sa
semrushbot-si
semrushbot-swa
seokicks
serpstatbot
woorankreview
accompanybot
adidxbot
anderspinkbot
articlefetcher
barkrowler
bitlybot
bitsightbot
bubing
builtwith
ccbot
checkmarknetwork
cincraw
dataforseobot
dataprovider
dnbcrawler-analytics
domainstatsbot
ecairn-grabber
electricmonk
exabot
ezooms
facebookexternalhit
facebot
gabanzabot
garlikcrawler
gozlebot
hypestat
ioncrawl
linkdexbot
madbbot
magpie-crawler
mail.ru_bot
mappy
neevabot
netestate ne crawler
news-please
newsnow
oai-searchbot
openai-gpt-3
openai-gpt-4
peer39_crawler
phxbot
pinteristbot
redditbot
pinterest
seznambot
| Product | Comment |
|---|---|
| articlefetcher | No info |
| bitlybot | Link shortener |
| builtwith | Technology profile lookup tool, UA: BW/ |
| ccbot | Common Crawl data can be freely download |
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Disallow | /blackhole/ |
| Disallow | /pages/nutzungsbedingungen.html |
| Disallow | /impressum/index |
| Disallow | /impressum/index.php |
| Disallow | /impressum/ |
| Disallow | /members/datenschutz.php |
| Disallow | /members/datenschutz |
| Disallow | /blackhole |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.thailand-faq.de/sitemapindex.xml |
Comments