istarthub.net
robots.txt

Robots Exclusion Standard data for istarthub.net

Resource Scan

Scan Details

Site Domain istarthub.net
Base Domain istarthub.net
Scan Status Ok
Last Scan2025-12-05T17:16:36+00:00
Next Scan 2025-12-12T17:16:36+00:00

Last Scan

Scanned2025-12-05T17:16:36+00:00
URL https://istarthub.net/robots.txt
Domain IPs 104.21.16.134, 172.67.212.116, 2606:4700:3031::ac43:d474, 2606:4700:3033::6815:1086
Response IP 172.67.212.116
Found Yes
Hash 04d7cd5720a7e1f248af411421aef796f72809db7cb0b39350c3027e6e25d4f6
SimHash 69019a0307b0

Groups

scrapy

Rule Path
Allow /

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php

Other Records

Field Value
sitemap https://istarthub.net/sitemap_index.xml

Warnings

  • `host` is not a known field.