reviewgist.com
robots.txt
Robots Exclusion Standard data for reviewgist.com
Resource Scan
Scan Details
| Site Domain | reviewgist.com |
| Base Domain | reviewgist.com |
| Scan Status | Ok |
| Last Scan | 2025-10-17T19:10:22+00:00 |
| Next Scan | 2025-11-16T19:10:22+00:00 |
Last Scan
| Scanned | 2025-10-17T19:10:22+00:00 |
| URL | https://www.reviewgist.com/robots.txt |
| Domain IPs | 104.21.6.188, 172.67.135.46, 2606:4700:3035::ac43:872e, 2606:4700:3036::6815:6bc |
| Response IP | 104.21.6.188 |
| Found | Yes |
| Hash | cddd64504a1677e864f846ae47110ecfbc0f255ac320b99faf376576ea0c2459 |
| SimHash | 1e5add4195d2 |
Groups
googlebot
| Rule | Path |
|---|---|
| Disallow | /cachemgr |
| Disallow | /jsp/tools |
| Disallow | /jsp/* |
| Disallow | *-reviews? |
| Disallow | *listConfigs.jsp* |
| Disallow | *page%3D* |
| Disallow | *start%3D* |
| Disallow | /best*%3D* |
| Disallow | /latest*%3D* |
| Disallow | /easiest*%3D* |
| Disallow | /rdnf?id* |
| Disallow | /redirectorNoFollow?id* |
| Disallow | *-vs-* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 0.2 |
*
| Rule | Path |
|---|---|
| Disallow | /rdnf?id* |
| Disallow | /redirectorNoFollow?id* |
| Disallow | /jsp/* |
| Disallow | *page%3D* |
| Disallow | *start%3D* |
| Disallow | /best*%3D* |
| Disallow | /latest*%3D* |
| Disallow | /easiest*%3D* |
| Disallow | *listConfigs.jsp* |
| Disallow | *-reviews? |
| Disallow | /cachemgr |
| Disallow | /jsp/tools |
| Disallow | *reviewComparator.jsp* |
| Disallow | *comparator.jsp* |
| Disallow | *reviewComparatorEmbed.jsp* |
| Disallow | *order.jsp* |
| Disallow | *widget.jsp* |
| Disallow | /widgets |
| Disallow | /*?section=* |
| Disallow | /*user_review-* |
| Disallow | /*description-* |
| Disallow | /*compareList* |
| Disallow | *-vs-* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 10 |
Warnings
- 2 invalid lines.
- `noindex` is not a known field.
Comments