spnet.com.au
robots.txt
Robots Exclusion Standard data for spnet.com.au
Resource Scan
Scan Details
| Site Domain | spnet.com.au |
| Base Domain | spnet.com.au |
| Scan Status | Ok |
| Last Scan | 2026-04-02T08:52:51+00:00 |
| Next Scan | 2026-05-02T08:52:51+00:00 |
Last Scan
| Scanned | 2026-04-02T08:52:51+00:00 |
| URL | https://spnet.com.au/robots.txt |
| Redirect | https://www.spnet.com.au/robots.txt |
| Redirect Domain | www.spnet.com.au |
| Redirect Base | spnet.com.au |
| Domain IPs | 54.79.237.151 |
| Redirect IPs | 54.79.237.151 |
| Response IP | 54.79.237.151 |
| Found | Yes |
| Hash | 01c0c426d2baa7cb6af1082cddedb8d6ae1c9fef52907a6ab02968bff37371e9 |
| SimHash | f215d002c2a1 |
Groups
mj12bot
yandex
yandexbot
seekportbot
barkrowler
blexbot
petalbot
geedoproductsearch
ahrefssiteaudit
facebookexternalhit
facebookcatalog
meta-externalfetcher
meta-webindexer
oai-searchbot
applebot
dataforseobot
pinterestbot
claude-searchbot
No rules defined. All paths allowed.
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
*
| Rule | Path |
|---|---|
| Disallow | /wp-admin/ |
| Disallow | /*yith_wcan%3D |
| Disallow | /*wc-ajax%3D |
| Disallow | /*?s= |
| Disallow | /*%26s%3D |
| Disallow | /*?orderby= |
| Disallow | /*%26orderby%3D |
| Disallow | /*?min_price= |
| Disallow | /*%26min_price%3D |
| Disallow | /*?max_price= |
| Disallow | /*%26max_price%3D |
| Disallow | /*?product_brand= |
| Disallow | /*%26product_brand%3D |
| Disallow | /*?product_filter_by= |
| Disallow | /*%26product_filter_by%3D |
| Disallow | /*?stock_status= |
| Disallow | /*%26stock_status%3D |
| Disallow | /*?filter_ |
| Disallow | /*%26filter_ |
| Disallow | /*?query_type_ |
| Disallow | /*%26query_type_ |
| Disallow | /*?add-to-cart |
| Disallow | /*%26add-to-cart |
| Disallow | /*?add_to_wishlist |
| Disallow | /*%26add_to_wishlist |
| Allow | /wp-admin/admin-ajax.php |