hackernews.com
robots.txt
Robots Exclusion Standard data for hackernews.com
Resource Scan
Scan Details
| Site Domain | hackernews.com |
| Base Domain | hackernews.com |
| Scan Status | Ok |
| Last Scan | 2024-11-04T11:57:56+00:00 |
| Next Scan | 2024-12-04T11:57:56+00:00 |
Last Scan
| Scanned | 2024-11-04T11:57:56+00:00 |
| URL | https://hackernews.com/robots.txt |
| Redirect | https://news.ycombinator.com/robots.txt |
| Redirect Domain | news.ycombinator.com |
| Redirect Base | ycombinator.com |
| Domain IPs | 18.155.68.123, 18.155.68.28, 18.155.68.33, 18.155.68.57, 2600:9000:23d2:1c00:c:bfb4:2d00:93a1, 2600:9000:23d2:4200:c:bfb4:2d00:93a1, 2600:9000:23d2:5e00:c:bfb4:2d00:93a1, 2600:9000:23d2:7600:c:bfb4:2d00:93a1, 2600:9000:23d2:9000:c:bfb4:2d00:93a1, 2600:9000:23d2:b200:c:bfb4:2d00:93a1, 2600:9000:23d2:cc00:c:bfb4:2d00:93a1, 2600:9000:23d2:e000:c:bfb4:2d00:93a1 |
| Redirect IPs | 209.216.230.207, 2606:7100:1:67::26 |
| Response IP | 209.216.230.207 |
| Found | Yes |
| Hash | 01ad7f8437bb233fee86db4bde6e95d3d3144b912e1307f88918db4f75092da9 |
| SimHash | 6011f2816b90 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /collapse? |
| Disallow | /context? |
| Disallow | /flag? |
| Disallow | /login |
| Disallow | /logout |
| Disallow | /r? |
| Disallow | /reply? |
| Disallow | /submitlink? |
| Disallow | /vote? |
| Disallow | /x? |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 30 |