designcrawl.com
robots.txt
Robots Exclusion Standard data for designcrawl.com
Resource Scan
Scan Details
| Site Domain | designcrawl.com |
| Base Domain | designcrawl.com |
| Scan Status | Ok |
| Last Scan | 2026-01-27T04:13:35+00:00 |
| Next Scan | 2026-02-26T04:13:35+00:00 |
Last Scan
| Scanned | 2026-01-27T04:13:35+00:00 |
| URL | https://designcrawl.com/robots.txt |
| Redirect | https://www.designcrawl.com/robots.txt |
| Redirect Domain | www.designcrawl.com |
| Redirect Base | designcrawl.com |
| Domain IPs | 153.92.212.43, 2a02:4780:15:30aa:2c39:3ac9:e8b1:1b09, 2a02:4780:16:6ff7:2b06:240:7cd1:3b05 |
| Redirect IPs | 2a02:4780:84:62d5:a2c9:c867:8ec7:2660, 2a02:4780:84:d7f9:2ca6:4008:50c9:afbd, 84.32.84.105, 84.32.84.228 |
| Response IP | 77.37.115.190 |
| Found | Yes |
| Hash | 991239a1d7f997518d809f410f6314919944123480e7373f4674f113f4b2a832 |
| SimHash | 11488e14d731 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /mint/ |
| Disallow | /labs/ |
| Disallow | /*/wp-* |
| Disallow | /*/feed/* |
| Disallow | /*/*?s=* |
| Disallow | /*/*.js$ |
| Disallow | /*/*.inc$ |
| Disallow | /transfer/ |
| Disallow | /*/cgi-bin/* |
| Disallow | /*/blackhole/* |
| Disallow | /*/trackback/* |
| Disallow | /*/xmlrpc.php |
| Allow | /*/20*/wp-* |
| Allow | /press/feed/$ |
| Allow | /press/tag/feed/$ |
| Allow | /*/wp-content/online/* |
Other Records
| Field | Value |
|---|---|
| sitemap | http://designcrawl.com/sitemap.xml |