beautifydata.com
robots.txt
Robots Exclusion Standard data for beautifydata.com
Resource Scan
Scan Details
| Site Domain | beautifydata.com |
| Base Domain | beautifydata.com |
| Scan Status | Ok |
| Last Scan | 2025-05-31T23:15:06+00:00 |
| Next Scan | 2025-06-07T23:15:06+00:00 |
Last Scan
| Scanned | 2025-05-31T23:15:06+00:00 |
| URL | https://beautifydata.com/robots.txt |
| Domain IPs | 35.225.76.33 |
| Response IP | 35.225.76.33 |
| Found | Yes |
| Hash | 7296989a24739b66135a201a1d72e8c96a36e33b4265602be8190e57d9fd409e |
| SimHash | 641fd070e443 |
Groups
googlebot
bingbot
slurp
duckduckbot
| Rule | Path |
|---|---|
| Disallow | /privacy-policy |
| Disallow | /contact |
| Disallow | /about |
| Disallow | /terms-and-conditions |
| Disallow | /economics/united-states/qcew-employment/ |
| Disallow | /stats/us/acs-5yrs/ |
| Disallow | /health/ |
| Disallow | /us-election-results/ |
| Disallow | /us-mass-shootings/ |
| Disallow | /us-traffic-facts/ |
| Disallow | /us-prisons/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://beautifydata.com/sitemap.xml |