cpantesters.org
robots.txt
Robots Exclusion Standard data for cpantesters.org
Resource Scan
Scan Details
| Site Domain | cpantesters.org |
| Base Domain | cpantesters.org |
| Scan Status | Ok |
| Last Scan | 2024-05-16T08:57:43+00:00 |
| Next Scan | 2024-06-15T08:57:43+00:00 |
Last Scan
| Scanned | 2024-05-16T08:57:43+00:00 |
| URL | https://cpantesters.org/robots.txt |
| Domain IPs | 151.101.130.217, 151.101.194.217, 151.101.2.217, 151.101.66.217 |
| Response IP | 151.101.194.217 |
| Found | Yes |
| Hash | 2ed0ced8bcec5db0faccb77779da1c978d56f3081150e12d5e084c55c99bac49 |
| SimHash | 382c5fc9d563 |
Groups
googlebot
| Rule | Path |
|---|---|
| Disallow | /private/ |
| Disallow | /*.yaml$ |
| Disallow | /*.rss$ |
| Disallow | /*.json$ |
| Disallow | /cpan/report |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 600 |
msnbot
| Rule | Path |
|---|---|
| Disallow | /private/ |
| Disallow | /*.yaml$ |
| Disallow | /*.rss$ |
| Disallow | /*.json$ |
| Disallow | /cpan/report |
dotbot
| Rule | Path |
|---|---|
| Disallow | /private/ |
| Disallow | /*.yaml$ |
| Disallow | /*.rss$ |
| Disallow | /*.json$ |
| Disallow | /cpan/report |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 600 |
*
| Rule | Path |
|---|---|
| Disallow | /private/ |
| Disallow | /*.yaml$ |
| Disallow | /*.rss$ |
| Disallow | /*.json$ |
| Disallow | /cpan/report |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 600 |
spbot
| Rule | Path |
|---|---|
| Disallow | /cpan/report |
| Disallow | /private/ |
| Disallow | /*.yaml$ |
| Disallow | /*.rss$ |
| Disallow | /*.json$ |
| Disallow | /cpan/report |
bixolabs
| Rule | Path |
|---|---|
| Disallow | /cpan/report |
| Disallow | /private/ |
| Disallow | /*.yaml$ |
| Disallow | /*.rss$ |
| Disallow | /*.json$ |
| Disallow | /cpan/report |
sogou web spider
| Rule | Path |
|---|---|
| Disallow | /cpan/report |
| Disallow | /private/ |
| Disallow | /*.yaml$ |
| Disallow | /*.rss$ |
| Disallow | /*.json$ |
| Disallow | /cpan/report |
sogou spider
| Rule | Path |
|---|---|
| Disallow | /cpan/report |
| Disallow | /private/ |
| Disallow | /*.yaml$ |
| Disallow | /*.rss$ |
| Disallow | /*.json$ |
| Disallow | /cpan/report |
ccbot
| Rule | Path |
|---|---|
| Disallow | /cpan/report |
| Disallow | /private/ |
| Disallow | /*.yaml$ |
| Disallow | /*.rss$ |
| Disallow | /*.json$ |
| Disallow | /cpan/report |
gaisbot
| Rule | Path |
|---|---|
| Disallow | /cpan/report |
| Disallow | /private/ |
| Disallow | /*.yaml$ |
| Disallow | /*.rss$ |
| Disallow | /*.json$ |
| Disallow | /cpan/report |
linkedinbot
| Rule | Path |
|---|---|
| Disallow | /cpan/report |
| Disallow | /private/ |
| Disallow | /*.yaml$ |
| Disallow | /*.rss$ |
| Disallow | /*.json$ |
| Disallow | /cpan/report |
pathtraq
| Rule | Path |
|---|---|
| Disallow | /cpan/report |
| Disallow | /private/ |
| Disallow | /*.yaml$ |
| Disallow | /*.rss$ |
| Disallow | /*.json$ |
| Disallow | /cpan/report |
msiecrawler
| Rule | Path |
|---|---|
| Disallow | /cpan/report |
| Disallow | /private/ |
| Disallow | /*.yaml$ |
| Disallow | /*.rss$ |
| Disallow | /*.json$ |
| Disallow | /cpan/report |
riddler
| Rule | Path |
|---|---|
| Disallow | /cpan/report |
| Disallow | /private/ |
| Disallow | /*.yaml$ |
| Disallow | /*.rss$ |
| Disallow | /*.json$ |
| Disallow | /cpan/report |
zeerchbot
| Rule | Path |
|---|---|
| Disallow | /cpan/report |
| Disallow | /private/ |
| Disallow | /*.yaml$ |
| Disallow | /*.rss$ |
| Disallow | /*.json$ |
| Disallow | /cpan/report |
Comments