news.ycombinator.com
robots.txt
Robots Exclusion Standard data for news.ycombinator.com
Resource Scan
Scan Details
| Site Domain | news.ycombinator.com |
| Base Domain | ycombinator.com |
| Scan Status | Ok |
| Last Scan | 2024-09-20T00:22:47+00:00 |
| Next Scan | 2024-10-20T00:22:47+00:00 |
Last Scan
| Scanned | 2024-09-20T00:22:47+00:00 |
| URL | https://news.ycombinator.com/robots.txt |
| Domain IPs | 209.216.230.207, 2606:7100:1:67::26 |
| Response IP | 209.216.230.207 |
| Found | Yes |
| Hash | 01ad7f8437bb233fee86db4bde6e95d3d3144b912e1307f88918db4f75092da9 |
| SimHash | 6011f2816b90 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /collapse? |
| Disallow | /context? |
| Disallow | /flag? |
| Disallow | /login |
| Disallow | /logout |
| Disallow | /r? |
| Disallow | /reply? |
| Disallow | /submitlink? |
| Disallow | /vote? |
| Disallow | /x? |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 30 |