carlroth.com
robots.txt
Robots Exclusion Standard data for carlroth.com
Resource Scan
Scan Details
| Site Domain | carlroth.com |
| Base Domain | carlroth.com |
| Scan Status | Ok |
| Last Scan | 2024-10-19T15:51:12+00:00 |
| Next Scan | 2024-11-18T15:51:12+00:00 |
Last Scan
| Scanned | 2024-10-19T15:51:12+00:00 |
| URL | https://www.carlroth.com/robots.txt |
| Domain IPs | 20.101.236.175 |
| Response IP | 20.101.236.175 |
| Found | Yes |
| Hash | 97b1c75829366e6099d1e2139e0bd42b664c3bb5fb86703466adc217c205a8c9 |
| SimHash | 6ad65551cc78 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /en/en/ |
| Disallow | /medias/sys_master |
| Disallow | /at/de/cart |
| Disallow | /at/de/checkout |
| Disallow | /at/de/my-account |
| Disallow | /at/de/search |
| Disallow | /at/en/cart |
| Disallow | /at/en/checkout |
| Disallow | /at/en/my-account |
| Disallow | /at/en/search |
| Disallow | /be/en/cart |
| Disallow | /be/en/checkout |
| Disallow | /be/en/my-account |
| Disallow | /be/en/search |
| Disallow | /be/fr/cart |
| Disallow | /be/fr/checkout |
| Disallow | /be/fr/my-account |
| Disallow | /be/fr/search |
| Disallow | /be/nl/cart |
| Disallow | /be/nl/checkout |
| Disallow | /be/nl/my-account |
| Disallow | /be/nl/search |
| Disallow | /ch/de/cart |
| Disallow | /ch/de/checkout |
| Disallow | /ch/de/my-account |
| Disallow | /ch/de/search |
| Disallow | /ch/en/cart |
| Disallow | /ch/en/checkout |
| Disallow | /ch/en/my-account |
| Disallow | /ch/en/search |
| Disallow | /ch/fr/cart |
| Disallow | /ch/fr/checkout |
| Disallow | /ch/fr/my-account |
| Disallow | /ch/fr/search |
| Disallow | /com/en/cart |
| Disallow | /com/en/checkout |
| Disallow | /com/en/my-account |
| Disallow | /com/en/search |
| Disallow | /de/de/cart |
| Disallow | /de/de/checkout |
| Disallow | /de/de/my-account |
| Disallow | /de/de/search |
| Disallow | /de/en/cart |
| Disallow | /de/en/checkout |
| Disallow | /de/en/my-account |
| Disallow | /de/en/search |
| Disallow | /en/en/search |
| Disallow | /fr/en/cart |
| Disallow | /fr/en/checkout |
| Disallow | /fr/en/my-account |
| Disallow | /fr/en/search |
| Disallow | /fr/fr/cart |
| Disallow | /fr/fr/checkout |
| Disallow | /fr/fr/my-account |
| Disallow | /fr/fr/search |
| Disallow | /nl/nl/cart |
| Disallow | /nl/nl/checkout |
| Disallow | /nl/nl/my-account |
| Disallow | /nl/nl/search |
| Disallow | /pl/en/cart |
| Disallow | /pl/en/checkout |
| Disallow | /pl/en/my-account |
| Disallow | /pl/en/search |
| Disallow | /pl/pl/cart |
| Disallow | /pl/pl/checkout |
| Disallow | /pl/pl/my-account |
| Disallow | /pl/pl/search |
| Disallow | /search |
Other Records
| Field | Value | Comment |
|---|---|---|
| crawl-delay | 10 | 10 seconds between page requests |
googlebot
| Rule | Path |
|---|---|
| Disallow | /medias/sys_master |
| Disallow | */search |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 7 |
googlebot-image
adsbot-google-mobile
adsbot-google
googlebot-news
googlebot-video
adsbot-google-mobile-apps
feedfetcher-google
google-read-aloud
duplexweb-google
google favicon
googleweblight
storebot-google
| Rule | Path |
|---|---|
| Disallow | /medias/sys_master |
| Disallow | */search |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
bingbot
duckduckbot
slurp
msnbot
| Rule | Path |
|---|---|
| Disallow | /medias/sys_master |
| Disallow | */search |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 8 |
applebot
pinterestbot
twitterbot
| Rule | Path |
|---|---|
| Disallow | /medias/sys_master |
| Disallow | */search |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 10 |
optimizer
sistrix crawler
sistrix optimizer
semrushbot
semrushbot-ba
semrushbot-bm
semrushbot-ct
semrushbot-sa
semrushbot-seoab
semrushbot-si
semrushbot-swa
semrushbot/7~bl
sistrix
| Rule | Path |
|---|---|
| Disallow | /pl/ |
| Disallow | /medias/sys_master |
| Disallow | */search |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
baiduspider
baiduspider-image
mediapartners-google
neevabot
yandex
yandexaccessibilitybot
yandexbot
yandeximages
| Rule | Path |
|---|---|
| Disallow | /pl/ |
| Disallow | /be/ |
| Disallow | /ch/ |
| Disallow | /at/ |
| Disallow | /en/ |
| Disallow | /de/en/ |
| Disallow | /fr/en/ |
| Disallow | /nl/en/ |
| Disallow | /medias/sys_master/ |
| Disallow | */search |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
alphabot
backlinkcrawler
baiduspider-video
bleriot
ccbot
cliqzbot
crawlson
dataforseobot
discordbot
exabot
huaweiwebcatbot
kraken
mail.ru_bot
megaindex.ru
megaindex.ru/2.0
page audit bot
plista-datafication
qwantify
rytebot
smtbot
searchmetricsbot
seobility
seznambot
slack-imgproxy
slackbot
slackbot-linkexpanding
sogou web spider
spinn3r
swiftbot
turnitin
tweetmemebot
velenpublicwebcrawler
xovibot
youdaobot
zoombot
auskunftbot
coccocbot-web
grapeshot
ichiro
ltx71
memorybot
metajobbot
moget
netestate ne crawler
plista
proximic
publiclibraryarchive
publiclibraryarchive.org
seoscanners.net
sogou spider
trendictionbot
turnitinbot
vebidoobot
| Rule | Path |
|---|---|
| Disallow | /pl/ |
| Disallow | /be/ |
| Disallow | /ch/ |
| Disallow | /at/ |
| Disallow | /en/ |
| Disallow | /de/en/ |
| Disallow | /fr/en/ |
| Disallow | /nl/en/ |
| Disallow | /medias/sys_master/ |
| Disallow | */search |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 30 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.carlroth.com/at/de/sitemap.xml |
| sitemap | https://www.carlroth.com/at/en/sitemap.xml |
| sitemap | https://www.carlroth.com/be/en/sitemap.xml |
| sitemap | https://www.carlroth.com/be/fr/sitemap.xml |
| sitemap | https://www.carlroth.com/be/nl/sitemap.xml |
| sitemap | https://www.carlroth.com/ch/de/sitemap.xml |
| sitemap | https://www.carlroth.com/ch/en/sitemap.xml |
| sitemap | https://www.carlroth.com/ch/fr/sitemap.xml |
| sitemap | https://www.carlroth.com/com/en/sitemap.xml |
| sitemap | https://www.carlroth.com/de/de/sitemap.xml |
| sitemap | https://www.carlroth.com/de/en/sitemap.xml |
| sitemap | https://www.carlroth.com/fr/en/sitemap.xml |
| sitemap | https://www.carlroth.com/fr/fr/sitemap.xml |
| sitemap | https://www.carlroth.com/nl/nl/sitemap.xml |
| sitemap | https://www.carlroth.com/pl/en/sitemap.xml |
| sitemap | https://www.carlroth.com/pl/pl/sitemap.xml |
Warnings
- 6 invalid lines.
- `request-rate` is not a known field.
- `visit-time` is not a known field.
Comments