comet.tech
robots.txt
Robots Exclusion Standard data for comet.tech
Resource Scan
Scan Details
| Site Domain | comet.tech |
| Base Domain | comet.tech |
| Scan Status | Ok |
| Last Scan | 2025-08-27T10:56:01+00:00 |
| Next Scan | 2025-09-26T10:56:01+00:00 |
Last Scan
| Scanned | 2025-08-27T10:56:01+00:00 |
| URL | https://comet.tech/robots.txt |
| Domain IPs | 13.69.68.41 |
| Response IP | 13.69.68.41 |
| Found | Yes |
| Hash | 35b5f4bced52cec62c6d75d0bd9cc0715b450b84f406eab3fca9d30b2a96061b |
| SimHash | 29d55918e7c8 |
Groups
*
adsbot-google-mobile
adsbot-google
adidxbot
| Rule | Path |
|---|---|
| Disallow | /Sandbox |
| Disallow | /kentico.pagebuilder |
| Disallow | /kentico/bundles |
| Disallow | /kentico.activities |
| Disallow | /kentico.resource |
| Disallow | /bundles |
| Disallow | /fonts |
| Disallow | /web.config |
| Disallow | /.git |
| Disallow | /*/form?docId=*&formId=*&formName=*&formType=* |
| Disallow | /*?*tag= |
| Disallow | /*.mdb |
| Disallow | /404 |
| Disallow | /*/404 |
mj12bot
curl/7.4.0
python-urllib/3.9
python-urllib/3.11
python-requests/2.25.1
go-http-client/1.1
okhttp/3.14.9
claudebot
mozilla/5.0 (l9scan/2.0.1343e28363e29363e23313; +https://leakix.net)
mozilla/5.0 (linux; android 11; vivo 1906; wv) applewebkit/537.36 (khtml, like gecko) version/4.0 chrome/87.0.4280.141 mobile safari/537.36 vivobrowser/8.9.0.0
mozilla/5.0 (linux; android 11; vivo 1906; wv) applewebkit/537.36 (khtml, like gecko) version/4.0 chrome/87.0.4280.141 mobile safari/537.36 vivobrowser/8.9.0.0 uni-app html5plus/1.0
mozilla/4.0 (compatible; msie 8.0; windows nt 6.1; trident/4.0)
mozilla/5.0 (compatible; mj12bot/v1.4.8; http://mj12bot.com/)
microsoft-webdav-miniredir/10.0.19045
pingdom.com_bot_version_1.4_(http://www.pingdom.com/)
turnitin (https://bit.ly/2uvnfoq)
mozilla/5.0 (linux; android 7.0;) applewebkit/537.36 (khtml, like gecko) mobile safari/537.36 (compatible; petalbot;+https://webmaster.petalsearch.com/site/petalbot)
mozilla/5.0 (compatible; semrushbot/7~bl; +http://www.semrush.com/bot.html)
claudebot
mozilla/5.0 (compatible; bytespider; spider-feedback@bytedance.com) applewebkit/537.36 (khtml, like gecko) chrome/70.0.0.0 safari/537.36
blackbox exporter/0.24.0
mozilla/5.0 (compatible; dotbot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)
java/1.8.0_161
mozilla/5.0 (compatible; ahrefsbot/7.0; +http://ahrefs.com/robot/)
go-http-client/2.0
buck/2.4.2; (+https://app.hypefactors.com/media-monitoring/about.html)
mozilla/5.0 (linux; android 5.0) applewebkit/537.36 (khtml, like gecko) mobile safari/537.36 (compatible; bytespider; spider-feedback@bytedance.com)
| Rule | Path |
|---|---|
| Disallow | / |
Other Records
| Field | Value |
|---|---|
| sitemap | https://comet.tech/sitemapindex.xml |