craw.in
robots.txt
Robots Exclusion Standard data for craw.in
Resource Scan
Scan Details
| Site Domain | craw.in |
| Base Domain | craw.in |
| Scan Status | Ok |
| Last Scan | 2026-01-03T06:00:51+00:00 |
| Next Scan | 2026-02-02T06:00:51+00:00 |
Last Scan
| Scanned | 2026-01-03T06:00:51+00:00 |
| URL | https://craw.in/robots.txt |
| Domain IPs | 72.61.234.8 |
| Response IP | 72.61.234.8 |
| Found | Yes |
| Hash | db590258ec142eb1ddebdb755b9e5bf2ff8f15ac91b23cdeb436d018f3feac80 |
| SimHash | 61a573d066ad |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /wp-admin/ |
| Allow | /wp-admin/admin-ajax.php |
googlebot-image
| Rule | Path |
|---|---|
| Allow | /images/ |
| Disallow | /wp-content/uploads/wpo-plugins-tables-list.json |
| Disallow | /*?utm_* |
| Disallow | /wp-includes/ |
| Disallow | /cgi-bin/ |
| Disallow | /cart/* |
| Disallow | /cart |
| Disallow | /index/hidden |
| Disallow | /index/hidden/* |
| Disallow | /projects/melvin/ |
| Disallow | /projects/ |
| Disallow | /maintenance.php |
| Disallow | /wp-content/* |
| Disallow | /checkout/* |
| Disallow | /secure/* |
| Disallow | /shop.php* |
| Disallow | /comments/feed |
| Disallow | /archives |
| Disallow | /tutor-certificate/ |
| Disallow | /*tutor-certificate/ |
| Disallow | /dashboard |
| Disallow | /blog/tag/* |
| Disallow | /blog/tag |
| Disallow | /tag/* |
| Disallow | /tag |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.craw.in/sitemap.xml |
| sitemap | https://www.craw.in/sitemap-products.xml |
| sitemap | https://www.craw.in/sitemap-courses.xml |
| sitemap | https://www.craw.in/sitemap-pages.xml |
| sitemap | https://www.craw.in/sitemap-blogs.xml |
Comments