spidervpn.org
robots.txt
Robots Exclusion Standard data for spidervpn.org
Resource Scan
Scan Details
| Site Domain | spidervpn.org |
| Base Domain | spidervpn.org |
| Scan Status | Ok |
| Last Scan | 2025-10-24T14:05:56+00:00 |
| Next Scan | 2025-11-23T14:05:56+00:00 |
Last Scan
| Scanned | 2025-10-24T14:05:56+00:00 |
| URL | https://spidervpn.org/robots.txt |
| Domain IPs | 104.21.59.164, 172.67.180.237, 2606:4700:3030::ac43:b4ed, 2606:4700:3035::6815:3ba4 |
| Response IP | 104.21.59.164 |
| Found | Yes |
| Hash | ddfc4a6bf73c29bc9ce8ec80a7f8fff4255a33f47d442ab59417a1a370ae762e |
| SimHash | 446de95db277 |
Groups
*
| Rule | Path | Comment |
|---|---|---|
| Disallow | /wp-admin/ | block admin UI |
| Allow | /wp-admin/admin-ajax.php | keep AJAX endpoints public |
| Disallow | /wp-login.php | hide login page |
| Disallow | /spdervpnapi/ | prevent crawling of JSON/API routes |
| Disallow | /clientarea/admin/ | block admin area |
| Disallow | /clientarea/includes/ | block source includes |
| Disallow | /clientarea/templates_c/ | block compiled templates |
| Disallow | /clientarea/pipe/ | block AJAX pipe |
| Disallow | /clientarea/downloads/ | protect file downloads |
| Disallow | /clientarea/css/ | hide styling assets |
| Disallow | /clientarea/js/ | hide JS assets |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 10 |
Other Records
| Field | Value | Comment |
|---|---|---|
| sitemap | https://example.com/sitemap.xml | point to your main sitemap |
Comments