booktopia.com.au
robots.txt
Robots Exclusion Standard data for booktopia.com.au
Resource Scan
Scan Details
| Site Domain | booktopia.com.au |
| Base Domain | booktopia.com.au |
| Scan Status | Ok |
| Last Scan | 2024-06-20T16:36:50+00:00 |
| Next Scan | 2024-07-20T16:36:50+00:00 |
Last Scan
| Scanned | 2024-06-20T16:36:50+00:00 |
| URL | https://booktopia.com.au/robots.txt |
| Redirect | https://www.booktopia.com.au/robots.txt |
| Redirect Domain | www.booktopia.com.au |
| Redirect Base | booktopia.com.au |
| Domain IPs | 13.33.88.18, 13.33.88.27, 13.33.88.51, 13.33.88.91 |
| Redirect IPs | 108.157.254.11, 108.157.254.110, 108.157.254.34, 108.157.254.4 |
| Response IP | 108.157.254.4 |
| Found | Yes |
| Hash | 7bab159b68d17069d34392e033b1d4ee73433cf72ae059372daeefc213e661f9 |
| SimHash | e32c5c8d9353 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /reviews/pwr/ |
| Disallow | /*productUrl |
| Disallow | /*sorter%3D* |
| Disallow | /*priceFilters%3D* |
| Disallow | /*formatFilters%3D* |
| Disallow | /*ageFilter%3D* |
| Disallow | /*isbn%3D* |
| Disallow | /*pn%3D* |
| Disallow | /*dym%3D* |
| Disallow | /*deviceType%3D* |
| Disallow | /*pID%3D* |
| Disallow | /*gclid%3D* |
| Disallow | /*source%3D* |
| Disallow | /*utm_medium%3D* |
| Disallow | /*utm_source%3D* |
| Disallow | /*utm_campaign%3D* |
| Disallow | /*attachment_id%3D* |
| Disallow | /*clickid%3D* |
| Disallow | /*gradeFilters%3D* |
| Disallow | /*bk_source_id%3D* |
| Disallow | /*bk_source%3D* |
| Disallow | /*PageSpeed%3D* |
| Disallow | /*src%3D* |
| Disallow | /*reviewRatingFilter%3D* |
| Disallow | /*sc_src%3D* |
| Disallow | /*sc_customer%3D* |
| Disallow | /*sc_lid%3D* |
| Disallow | /*sc_llid%3D* |
| Disallow | /*sc_uid%3D* |
| Disallow | /*clear%3D* |
| Disallow | /*like%3D* |
| Disallow | /*SUBID1%3D* |
| Disallow | /*SUBID2%3D* |
| Disallow | /*promoCode%3D* |
| Disallow | /*binding%3D* |
| Disallow | /*WT.dcsvid%3D* |
| Disallow | /*WT.mc_id%3D* |
| Disallow | /*utm_content%3D* |
| Disallow | /*zsrc%3D* |
| Disallow | /*submit.y%3D* |
| Disallow | /*submit.x%3D* |
| Disallow | /search.ep* |
| Disallow | /*/_next/static/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 10 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.booktopia.com.au/sitemap.xml.gz |
| sitemap | https://www.booktopia.com.au/buy/sitemap.xml |