artglasssupplies.com
robots.txt

Robots Exclusion Standard data for artglasssupplies.com

Resource Scan

Scan Details

Site Domain artglasssupplies.com
Base Domain artglasssupplies.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-03-16T04:25:42+00:00
Next Scan 2024-06-14T04:25:42+00:00

Last Successful Scan

Scanned2023-10-26T04:05:57+00:00
URL https://artglasssupplies.com/robots.txt
Domain IPs 172.66.40.232, 172.66.43.24, 2606:4700:3108::ac42:28e8, 2606:4700:3108::ac42:2b18
Response IP 172.66.40.232
Found Yes
Hash 9aa857583d1b8e445d528dd4e02981a2842fe69eddec39e6cf4331dee249d893
SimHash 284e4116cef8

Groups

*

Rule Path
Disallow /admin/
Disallow /manager/
Disallow /sites/
Disallow /cache/
Disallow /cert/
Disallow /debug/
Disallow /download/
Disallow /ext/
Disallow /pear/
Disallow /pub/
Disallow /temp/
Disallow /tmp/
Disallow /account.php
Disallow /account_edit.php
Disallow /account_history.php
Disallow /account_history_info.php
Disallow /account_password.php
Disallow /address_book.php
Disallow /address_book_process.php
Disallow /checkout_confirmation.php
Disallow /checkout_payment.php
Disallow /checkout_payment_address.php
Disallow /checkout_process.php
Disallow /checkout_shipping.php
Disallow /checkout_shipping_address.php
Disallow /checkout_success.php
Disallow /cookie_usage.php
Disallow /create_account.php
Disallow /create_account_success.php
Disallow /ipn.php
Disallow /login.php
Disallow /logoff.php
Disallow /password_forgotten.php
Disallow /paypal_notify.php
Disallow /redirect.php
Disallow /shipping_estimator_popup.php
Disallow /ssl_check.php
Disallow /vvc_display.php
Disallow /wpcallback.php
Disallow /affiliate_validproducts.php

Other Records

Field Value
sitemap https://www.artglasssupplies.com/sitemap.xml

Comments

  • To prevent only a select bots from any scans
  • User-agent: crawlers.looksmart.com
  • Disallow: /
  • To prenet any images from being scanned
  • Disallow: *.gif
  • Disallow: *.png
  • To block main page due to size from bots
  • Disallow: /index.php
  • To shut down site completely from bots use
  • Disallow: /
  • Block out things that are secure or login oriented
  • These are based on not using any rewrite rules
  • THEN YOU CAN INCLUDE THE FOLLOWING IN YOUR ROBOTS FILE.
  • I FOUND THAT MY BANDWIDTH USAGE DROPPED BY A MASSIVE AMOUNT AFTER I GOT RID
  • AND NOT EVEN BROWSING THE SITE.