theweb.com
robots.txt
Robots Exclusion Standard data for theweb.com
Resource Scan
Scan Details
| Site Domain | theweb.com |
| Base Domain | theweb.com |
| Scan Status | Failed |
| Failure Reason | Scan timed out. |
| Last Scan | 2024-09-06T15:50:54+00:00 |
| Next Scan | 2024-12-05T15:50:54+00:00 |
Last Successful Scan
| Scanned | 2022-11-13T04:31:28+00:00 |
| URL | https://www.theweb.com/robots.txt |
| Response IP | 13.224.250.14, 13.224.250.2, 13.224.250.57, 13.224.250.54 |
| Found | Yes |
| Hash | c36038f68d31ed8131910e30c74bf61cafed0157138de412d1697facdbca96dd |
| SimHash | 2400d0c05592 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /api |
| Disallow | /click_ad |
| Disallow | /clk_ad |
| Disallow | /wr_clk |
Comments