datarade.ai
robots.txt
Robots Exclusion Standard data for datarade.ai
Resource Scan
Scan Details
| Site Domain | datarade.ai |
| Base Domain | datarade.ai |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 2025-10-25T04:22:10+00:00 |
| Next Scan | 2026-01-23T04:22:10+00:00 |
Last Successful Scan
| Scanned | 2024-06-09T18:06:43+00:00 |
| URL | https://datarade.ai/robots.txt |
| Domain IPs | 172.66.40.248, 172.66.43.8, 2606:4700:3108::ac42:28f8, 2606:4700:3108::ac42:2b08 |
| Response IP | 172.66.43.8 |
| Found | Yes |
| Hash | eb4e34cb2f79ef114d136fd0fb443e4377997497724a229d5ccfb695bac53871 |
| SimHash | 431d7571b7bf |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /*? |
| Disallow | */requests/new* |
| Disallow | */requests/steps* |
| Disallow | */reviews/new* |
| Disallow | /search* |
| Disallow | /link/* |
| Disallow | /my/* |
| Disallow | /data-categories/*/providers/* |
| Allow | /data-categories/*/providers/1$ |
| Allow | /data-categories/*/providers/2$ |
| Allow | /data-categories/*/providers/3$ |
| Allow | /data-categories/*/providers/4$ |
| Allow | /data-categories/*/providers/5$ |
| Disallow | /es/data-categories/*/providers/* |
| Allow | /es/data-categories/*/providers/1$ |
| Allow | /es/data-categories/*/providers/2$ |
| Allow | /es/data-categories/*/providers/3$ |
| Allow | /es/data-categories/*/providers/4$ |
| Allow | /es/data-categories/*/providers/5$ |
| Allow | /search/products/* |
Other Records
| Field | Value |
|---|---|
| sitemap | https://datarade.ai/sitemap.xml.gz |