developtask.com
robots.txt
Robots Exclusion Standard data for developtask.com
Resource Scan
Scan Details
| Site Domain | developtask.com |
| Base Domain | developtask.com |
| Scan Status | Ok |
| Last Scan | 2024-11-05T12:46:50+00:00 |
| Next Scan | 2024-11-12T12:46:50+00:00 |
Last Scan
| Scanned | 2024-11-05T12:46:50+00:00 |
| URL | https://developtask.com/robots.txt |
| Domain IPs | 74.208.234.56 |
| Response IP | 74.208.234.56 |
| Found | Yes |
| Hash | 3366840f4326d81b75b2c6ca12a3aebf609c09ea0d98efc59c50aea42ebd5b73 |
| SimHash | ca0b1614a32d |
Groups
yandex
| Rule | Path |
|---|---|
| Disallow | |
| Disallow | /sitemap/202310* |
| Disallow | /pubblica/* |
| Disallow | /site/captcha |
| Disallow | /autenticazione |
| Disallow | /site/reportbug |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
*
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | /articolo*?a= |
| Disallow | /articolo*?titolo= |
| Disallow | /en/articolo*?titolo= |
| Disallow | /jp/articolo*?titolo= |
| Disallow | /en/tag*?per-page= |
| Disallow | /ru/tag*?per-page= |
| Disallow | /enticolo* |
| Disallow | /ptticolo* |
| Disallow | /utente/* |
| Disallow | /pubblica/* |
| Disallow | /site/captcha |
| Disallow | /autenticazione |
| Disallow | /site/reportbug |
Warnings
- `clean-param` is not a known field.
Comments