polyguru.au
robots.txt

Robots Exclusion Standard data for polyguru.au

Resource Scan

Scan Details

Site Domain polyguru.au
Base Domain polyguru.au
Scan Status Ok
Last Scan2025-11-18T00:15:12+00:00
Next Scan 2025-11-25T00:15:12+00:00

Last Scan

Scanned2025-11-18T00:15:12+00:00
URL https://www.polyguru.au/robots.txt
Domain IPs 151.101.1.124, 151.101.129.124, 151.101.193.124, 151.101.65.124
Response IP 151.101.129.124
Found Yes
Hash a4b82a848277b19294a5ad029a19d6e4ae46f7916f86e0c2514c697a20f84b15
SimHash 8010dc12c7a0

Groups

*

Rule Path
Disallow /app/
Disallow /bin/
Disallow /dev/
Disallow /lib/
Disallow /phpserver/
Disallow /pub/
Disallow /setup/
Disallow /update/
Disallow /var/
Disallow /vendor/
Disallow /generated/
Disallow /admin/
Disallow /customer/account/login/
Disallow /customer/account/logout/
Disallow /customer/account/create/
Disallow /customer/account/forgotpassword/
Disallow /checkout/
Disallow /onestepcheckout/
Disallow /cart/
Disallow /search/
Disallow /catalogsearch/
Disallow /catalogsearch/
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /catalog/product/view/
Allow /pub/static/
Allow /static/
Allow /media/

bytespider

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

Comments

  • Disallow admin and login URLs
  • Disallow search results and filtered URLs
  • Allow CSS, JS, and images
  • Crawl-delay directive to prevent server overload (optional)