4.img-dpreview.com
robots.txt
Robots Exclusion Standard data for 4.img-dpreview.com
Resource Scan
Scan Details
Site Domain | 4.img-dpreview.com |
Base Domain | img-dpreview.com |
Scan Status | Failed |
Failure Stage | Fetching resource. |
Failure Reason | Server returned a client error. |
Last Scan | 2024-08-07T22:21:47+00:00 |
Next Scan | 2024-10-06T22:21:47+00:00 |
Last Successful Scan
Scanned | 2024-05-17T22:19:40+00:00 |
URL | https://4.img-dpreview.com/robots.txt |
Domain IPs | 104.22.34.206, 104.22.35.206, 172.67.30.66 |
Response IP | 104.22.35.206 |
Found | Yes |
Hash | abf51f4946eefba904ea2b357951d6c7d2c1685b1793ce08c55d5f877a6e0754 |
SimHash | 6c36d550ae50 |
Groups
*
Rule | Path |
---|---|
Disallow | /api |
Disallow | /forums/reply |
Disallow | /forums/redirect-last-post/ |
Disallow | /forums/redirect-unread/ |
Disallow | /forums/redirect-post/ |
Disallow | /reviews_data |
Disallow | /search |
Disallow | /articles/redirect/ |
Disallow | /set-location |
Disallow | /members/messages/ |
Disallow | /tag/ |
Disallow | /preview/ |
googlebot
Rule | Path |
---|---|
Allow | /ads.txt |
Disallow | /ad |
Disallow | /api |
Disallow | /forums/reply |
Disallow | /forums/redirect-last-post/ |
Disallow | /forums/redirect-unread/ |
Disallow | /forums/redirect-post/ |
Disallow | /reviews_data |
Disallow | /search |
Disallow | /articles/redirect/ |
Disallow | /set-location |
Disallow | /members/messages/ |
Disallow | /tag/ |
Disallow | /preview/ |
Disallow | /*%3D$ |
bingbot
msnbot
Rule | Path |
---|---|
Disallow | /api |
Disallow | /forums/reply |
Disallow | /forums/redirect-last-post/ |
Disallow | /forums/redirect-unread/ |
Disallow | /forums/redirect-post/ |
Disallow | /reviews_data |
Disallow | /search |
Disallow | /articles/redirect/ |
Disallow | /set-location |
Disallow | /members/messages/ |
Disallow | /tag/ |
Disallow | /preview/ |
Other Records
Field | Value |
---|---|
crawl-delay | 1 |
ahrefsbot
Rule | Path |
---|---|
Disallow | /api |
Disallow | /forums/reply |
Disallow | /forums/redirect-last-post/ |
Disallow | /forums/redirect-unread/ |
Disallow | /forums/redirect-post/ |
Disallow | /reviews_data |
Disallow | /search |
Disallow | /articles/redirect/ |
Disallow | /set-location |
Disallow | /members/messages/ |
Disallow | /tag/ |
Disallow | /preview/ |
Disallow | /galleries |
Other Records
Field | Value |
---|---|
crawl-delay | 2 |
baiduspider
sogou
dotbot
Rule | Path |
---|---|
Disallow | /api |
Disallow | /forums/reply |
Disallow | /forums/redirect-last-post/ |
Disallow | /forums/redirect-unread/ |
Disallow | /forums/redirect-post/ |
Disallow | /reviews_data |
Disallow | /search |
Disallow | /articles/redirect/ |
Disallow | /set-location |
Disallow | /members/messages/ |
Disallow | /tag/ |
Disallow | /preview/ |
Disallow | /galleries |
googleother
Rule | Path |
---|---|
Disallow | /api |
Disallow | /forums/reply |
Disallow | /forums/redirect-last-post/ |
Disallow | /forums/redirect-unread/ |
Disallow | /forums/redirect-post/ |
Disallow | /reviews_data |
Disallow | /search |
Disallow | /articles/redirect/ |
Disallow | /set-location |
Disallow | /members/messages/ |
Disallow | /tag/ |
Disallow | /preview/ |
Disallow | /*%3D$ |
Other Records
Field | Value |
---|---|
crawl-delay | 10 |
magpie-crawler
Rule | Path |
---|---|
Disallow | /members/ |
Disallow | /api |
Disallow | /forums/reply |
Disallow | /forums/redirect-last-post/ |
Disallow | /forums/redirect-unread/ |
Disallow | /forums/redirect-post/ |
Disallow | /reviews_data |
Disallow | /search |
Disallow | /articles/redirect/ |
Disallow | /set-location |
Disallow | /members/messages/ |
Disallow | /tag/ |
Disallow | /preview/ |
Disallow | /galleries |
ias_crawler
Rule | Path |
---|---|
Disallow | /api |
Disallow | /forums/reply |
Disallow | /forums/redirect-last-post/ |
Disallow | /forums/redirect-unread/ |
Disallow | /forums/redirect-post/ |
Disallow | /reviews_data |
Disallow | /search |
Disallow | /articles/redirect/ |
Disallow | /set-location |
Disallow | /members/messages/ |
Disallow | /tag/ |
Disallow | /preview/ |
Other Records
Field | Value |
---|---|
sitemap | https://www.dpreview.com/sitemaps |
sitemap | https://www.dpreview.com/sitemaps/news |
Warnings
- 2 invalid lines.