carbonframe.com
robots.txt
Robots Exclusion Standard data for carbonframe.com
Resource Scan
Scan Details
| Site Domain | carbonframe.com |
| Base Domain | carbonframe.com |
| Scan Status | Ok |
| Last Scan | 2025-08-03T08:18:45+00:00 |
| Next Scan | 2025-09-02T08:18:45+00:00 |
Last Scan
| Scanned | 2025-08-03T08:18:45+00:00 |
| URL | https://carbonframe.com/robots.txt |
| Redirect | https://www.carbonframe.com/robots.txt |
| Redirect Domain | www.carbonframe.com |
| Redirect Base | carbonframe.com |
| Domain IPs | 104.21.38.175, 172.67.137.26, 2606:4700:3030::6815:26af, 2606:4700:3030::ac43:891a |
| Redirect IPs | 104.21.38.175, 172.67.137.26, 2606:4700:3030::6815:26af, 2606:4700:3030::ac43:891a |
| Response IP | 172.67.137.26 |
| Found | Yes |
| Hash | d8af8577e0604829143526e3c571bc864a6df79a43ccdfceccec4f55b29339bd |
| SimHash | 38567912e12b |
Groups
petalbot
aliyunsecbot
tiktokspider
farmerduckybot
mauibot
megaindex.ru
zoominfobot
extlinksbot
hubspot
leiki
webmeup
ahrefsbot
alexibot
surveybot
siteauditbot
semrushbot-ba
semrushbot-si
semrushbot-swa
splitsignalbot
semrushbot-ocob
semrushbot-ft
semrushbot
barkrowler
seokicks
dataforseobot
awariorssbot
awariosmartbot
dotbot
blexbot
bytedance
bytespider
haosouspider
imagesiftbot
ccbot
zoombot
mj12bot
meta-externalagent
amazonbot
skyworkspider
geedoproductsearch
geedobot
yisouspider
sosospider
keys-so-bot
sogou web spider/4.0
sogou web spider
sogou spider
sogou inst spider
sogou spider
sogou blog
sogou news spider
sogou orion spider
screaming frog seo spider
wuhensoubot
mijisoubot
gptbot
chatgpt-user
oai-searchbot
perplexitybot
perplexity-user
claudebot
yandex
rambler
baiduspider
baiduspider-image
baiduspider-video
baiduspider-news
baiduspider-mobile
| Rule | Path |
|---|---|
| Disallow | / |
bingbot
msnbot
naverbot
naversearch
pinterestbot
applebot
coccocbot
coccocbot-image
yeti
slurp
mojeekbot
seznambot
duckduckbot
startpagebot
dogedogebot
lookaobot
searxbot
searchphibot
ecosia
qwant
cari
qwantbot
| Rule | Path |
|---|---|
| Allow | /*.js$ |
| Allow | /*.css$ |
| Disallow | /*?* |
| Disallow | /tag/* |
| Disallow | /*/tag/* |
*
| Rule | Path |
|---|---|
| Allow | /*.js$ |
| Allow | /*.css$ |
| Disallow | /*?* |
| Disallow | /admin/ |
| Disallow | /index.php?route=product%2Fsearch%2Fvalidate |
| Disallow | /*?sort= |
| Disallow | /*%26sort%3D |
| Disallow | /*?limit= |
| Disallow | /*%26limit%3D |
| Disallow | /*%26ajax%3D1 |
| Disallow | /*?page=1 |
| Disallow | /*/index.php?route=product%2Fmanufacturer |
| Disallow | /*/index.php?route=product%2Fspecial&* |
| Disallow | /index.php?route=product%2Fspecial* |
| Disallow | /*/*?cireview_product_id=* |
| Disallow | /*/customers_also_viewed/ |
| Disallow | /*/product/review* |
| Disallow | /*?route=product%2Fcategory&* |
| Disallow | /*/index.php?route=product%2Fcategory |
| Disallow | /*/index.php?route=product%2Fproduct&product_id=* |
| Disallow | /index.php?route=product%2Fproduct&product_id=* |
| Disallow | /*/page-1.html |
| Disallow | /index.php?route=product%2Fsearch&search=* |
| Disallow | /*/index.php?route=product%2Fsearch&search=* |
| Disallow | /*/search/validate |
| Disallow | /*%26is_webp%3D |
| Disallow | /*?item_id= |
| Disallow | /*/tag%26product_id%3D |
| Disallow | /*/category%26product_id%3D |
Warnings
- 3 invalid lines.