cleardarksky.com
robots.txt
Robots Exclusion Standard data for cleardarksky.com
Resource Scan
Scan Details
| Site Domain | cleardarksky.com |
| Base Domain | cleardarksky.com |
| Scan Status | Ok |
| Last Scan | 2024-06-08T06:31:13+00:00 |
| Next Scan | 2024-06-15T06:31:13+00:00 |
Last Scan
| Scanned | 2024-06-08T06:31:13+00:00 |
| URL | https://www.cleardarksky.com/robots.txt |
| Domain IPs | 192.252.149.19, 192.254.227.51 |
| Response IP | 192.254.227.51 |
| Found | Yes |
| Hash | a488f3b9080704543f5302de7d8ba22715cfdd44466d0d533f2bee55cac53a3c |
| SimHash | 013cca65dc91 |
Groups
googlebot
mediapartners-google
| Rule | Path |
|---|---|
| Disallow | /cgi-bin/ |
| Disallow | /so/anal/ |
| Disallow | /txtc/ |
| Disallow | /getcr.php |
| Disallow | /f.php |
| Allow | /index.html |
| Allow | /c/* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 50 |
yandexbot
| Rule | Path |
|---|---|
| Disallow | /cgi-bin/ |
| Disallow | /maps/ |
| Disallow | /so/ |
| Allow | /index.html |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 50 |
yandex
| Rule | Path |
|---|---|
| Disallow | /cgi-bin/ |
| Disallow | /maps/ |
| Disallow | /so/ |
| Allow | /index.html |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 50 |
baiduspider
| Rule | Path |
|---|---|
| Disallow | /cgi-bin/ |
| Disallow | /csk/ |
| Disallow | /c/ |
| Disallow | /maps/ |
| Disallow | /so/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 50 |
mj12bot
majestic
| Rule | Path |
|---|---|
| Disallow | /cgi-bin/ |
| Disallow | /maps/ |
| Disallow | /so/ |
| Allow | /index.html |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 50 |
the knowledge ai
| Rule | Path |
|---|---|
| Disallow | /cgi-bin/ |
| Disallow | /maps |
| Disallow | /so |
| Allow | /index.html |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 50 |
semanticscholar.org
| Rule | Path |
|---|---|
| Disallow | /cgi-bin/ |
| Disallow | /maps/ |
| Disallow | /so/ |
| Allow | /index.html |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 50 |
semrushbot
| Rule | Path |
|---|---|
| Disallow | /cgi-bin/ |
| Disallow | /maps/ |
| Disallow | /so/ |
| Allow | /index.html |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 50 |
*
| Rule | Path |
|---|---|
| Disallow | /cgi-bin/ |
| Disallow | /so/anal/ |
| Disallow | /txtc/ |
| Disallow | /getcr.php |
| Disallow | /f.php |
| Allow | /index.html |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 50 |
Warnings
- 1 invalid line.