weberperformance.com.au
robots.txt

Robots Exclusion Standard data for weberperformance.com.au

Resource Scan

Scan Details

Site Domain weberperformance.com.au
Base Domain weberperformance.com.au
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-03-16T22:32:25+00:00
Next Scan 2025-05-15T22:32:25+00:00

Last Successful Scan

Scanned2024-01-29T15:33:04+00:00
URL https://weberperformance.com.au/robots.txt
Domain IPs 104.21.48.230, 172.67.188.85, 2606:4700:3030::ac43:bc55, 2606:4700:3031::6815:30e6
Response IP 104.21.48.230
Found Yes
Hash b192b4efcef1ecfeea8d33839eb1a6df74535287c1993497bd24bf8eb1716514
SimHash 285e49074adc

Groups

*

Rule Path
Disallow /admin/
Disallow /cache/
Disallow /cert/
Disallow /debug/
Disallow /download/
Disallow /ext/
Disallow /includes/
Disallow /pear/
Disallow /pub/
Disallow /temp/
Disallow /templates/
Disallow /tmp/
Disallow /account.php
Disallow /account_edit.php
Disallow /account_history.php
Disallow /account_history_info.php
Disallow /account_password.php
Disallow /address_book.php
Disallow /address_book_process.php
Disallow /checkout_confirmation.php
Disallow /checkout_payment.php
Disallow /checkout_payment_address.php
Disallow /checkout_process.php
Disallow /checkout_shipping.php
Disallow /checkout_shipping_address.php
Disallow /checkout_success.php
Disallow /cookie_usage.php
Disallow /create_account.php
Disallow /create_account_success.php
Disallow /ipn.php
Disallow /login.php
Disallow /logoff.php
Disallow /password_forgotten.php
Disallow /paypal_notify.php
Disallow /redirect.php
Disallow /shipping_estimator_popup.php
Disallow /ssl_check.php
Disallow /vvc_display.php
Disallow /wpcallback.php

Comments

  • CRELoaded Generated Robots.txt
  • Robot Exclusion File -- robots.txt
  • Author: CRELoaded Team
  • Last Updated : September 11th, 2007
  • To prevent only a select bots from any scans
  • User-agent: crawlers.looksmart.com
  • Disallow: /
  • To prenet any images from being scanned
  • Disallow: *.gif
  • Disallow: *.jpg
  • Disallow: *.png
  • To block main page due to size from bots
  • Disallow: /index.php
  • To shut down site completely from bots use
  • Disallow: /
  • Block out things that are secure or login oriented
  • These are based on not using any rewrite rules
  • IF YOU DO NOT WISH TO HAVE THE GOOGLE IMAGE BOT SCAN YOUR DOMAIN FOR IMAGES
  • THEN YOU CAN INCLUDE THE FOLLOWING IN YOUR ROBOTS FILE.
  • I FOUND THAT MY BANDWIDTH USAGE DROPPED BY A MASSIVE AMOUNT AFTER I GOT RID
  • OF THE GOOGLE IMAGE BOT. ALL I HAD WAS IMAGE HUNTERS STEALING PRODUCT SHOTS
  • AND NOT EVEN BROWSING THE SITE.
  • User-agent: Googlebot-Image
  • Disallow: /