dataretrieval.com
robots.txt
Robots Exclusion Standard data for dataretrieval.com
Resource Scan
Scan Details
| Site Domain | dataretrieval.com |
| Base Domain | dataretrieval.com |
| Scan Status | Ok |
| Last Scan | 2025-10-14T14:25:47+00:00 |
| Next Scan | 2025-11-13T14:25:47+00:00 |
Last Scan
| Scanned | 2025-10-14T14:25:47+00:00 |
| URL | https://dataretrieval.com/robots.txt |
| Domain IPs | 104.21.40.131, 172.67.151.242, 2606:4700:3031::ac43:97f2, 2606:4700:3037::6815:2883 |
| Response IP | 104.21.40.131 |
| Found | Yes |
| Hash | e837ec24fc387e8cc1d5b2370efc26dc6df1e0dc57406504af0bb01f5985d55b |
| SimHash | b30070730367 |
Groups
*
| Rule | Path |
|---|---|
| Allow | / |
| Disallow | */wp-admin |
| Disallow | */wp-includes |
| Disallow | */wp-login.php |
| Disallow | */wp-register.php |
| Disallow | /xmlrpc.php |
| Disallow | /comments |
| Disallow | /phpmyadmin |
| Disallow | */trackback |
| Disallow | /cgi-bin |
| Disallow | *?s= |
| Disallow | *? |
| Allow | */uploads |
| Allow | */wp-includes/css/ |
| Allow | */wp-includes/js/ |
| Allow | */wp-includes/images/ |
| Allow | */wp-*.png |
| Allow | */wp-*.jpg |
| Allow | */wp-*.jpeg |
| Allow | */wp-content/plugins/ |
| Allow | */wp-content/plugins/*.js |
| Allow | */wp-content/plugins*.gif |
| Allow | */wp-content/plugins/*.css |
| Allow | */wp-content/cache/ |
| Allow | */wp-content/themes/ |
| Allow | */wp-content/themes/*.css |
| Allow | */wp-content/themes/*.png |
| Allow | */wp-content/themes/*.js |
| Allow | */wp-content/themes/*.svg |
| Allow | */wp-content/themes/*.woff2 |
| Allow | */wp-content/uploads/*.css |
| Allow | */wp-content/uploads/*.png |
| Allow | */wp-content/uploads/*.js |
| Allow | /assets/css/ |
| Allow | /assets/js/ |
| Allow | /assets/img/ |
| Allow | /fonts/ |
| Allow | */wp-admin/admin-ajax.php |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.dataretrieval.com/dataretrwp/mpg-sitemap-us-cities.xml |