cleanitsupply.com
robots.txt
Robots Exclusion Standard data for cleanitsupply.com
Resource Scan
Scan Details
| Site Domain | cleanitsupply.com |
| Base Domain | cleanitsupply.com |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 2024-04-16T22:18:23+00:00 |
| Next Scan | 2024-07-15T22:18:23+00:00 |
Last Successful Scan
| Scanned | 2023-08-28T17:52:19+00:00 |
| URL | https://cleanitsupply.com/robots.txt |
| Redirect | https://www.cleanitsupply.com/robots.txt |
| Redirect Domain | www.cleanitsupply.com |
| Redirect Base | cleanitsupply.com |
| Domain IPs | 151.139.128.10 |
| Redirect IPs | 151.139.128.10 |
| Response IP | 151.139.128.10 |
| Found | Yes |
| Hash | 1d5a6913e0d2933d8ef7737b864d905d4bd1afa7943ade149aa02dbcebbc6242 |
| SimHash | 80d4831ecd7d |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /ASPDNSFCommon/ |
| Disallow | /ASPDNSFEncrypt/ |
| Disallow | /ASPDNSFGateways/ |
| Disallow | /ASPDNSFPatterns/ |
| Disallow | /ASPDNSFQuickBooks/ |
| Disallow | /bin/ |
| Disallow | /categorydescriptions/ |
| Disallow | /db/ |
| Disallow | /download/ |
| Disallow | /emailtemplates/ |
| Disallow | /orderdownloads/ |
| Disallow | /productdescriptions/ |
| Disallow | /productspecs/ |
| Disallow | /receipts/ |
| Disallow | /remotetracer/ |
| Disallow | /sectiondescriptions/ |
| Disallow | /shipped/ |
| Disallow | /skins/ |
| Disallow | /subscription/ |
| Disallow | /URLRewriter/ |
| Disallow | /XMLImporter/ |
| Disallow | /addtocart.aspx |
| Disallow | /about/meet-rich-hebert.aspx |
| Disallow | /cardinalauth.aspx |
| Disallow | /cardinalform.aspx |
| Disallow | /cardinal_process.aspx |
| Disallow | /checkoutanon.aspx |
| Disallow | /checkoutgiftcard.aspx |
| Disallow | /checkoutpayment.aspx |
| Disallow | /checkoutreview.aspx |
| Disallow | /checkoutshipping.aspx |
| Disallow | /checkoutshippingmult.aspx |
| Disallow | /checkoutshippingmult2.aspx |
| Disallow | /clearcookies.aspx |
| Disallow | /clickout.aspx |
| Disallow | /createaccount.aspx |
| Disallow | /disclaimer.aspx |
| Disallow | /dumpuser.aspx |
| Disallow | /dyop_addtocart.aspx |
| Disallow | /dyop_delete.aspx |
| Disallow | /dyop_quan.aspx |
| Disallow | /account.aspx |
| Disallow | /editaddress.aspx |
| Disallow | /emailproduct.aspx |
| Disallow | /gcCallback.aspx |
| Disallow | /giftregistrysearch.aspx |
| Disallow | /JpegImage.aspx |
| Disallow | /lat_account.aspx |
| Disallow | /lat_driver.aspx |
| Disallow | /lat_getlinking.aspx |
| Disallow | /lat_signin.aspx |
| Disallow | /lat_signout.aspx |
| Disallow | /lat_signup.aspx |
| Disallow | /notification.aspx |
| Disallow | /nxfeed.aspx |
| Disallow | /orderconfirmation.aspx |
| Disallow | /paypalcancel.aspx |
| Disallow | /paypalexpressok.aspx |
| Disallow | /paypalnotification.aspx |
| Disallow | /paypalok.aspx |
| Disallow | /pollvote.aspx |
| Disallow | /popup.aspx |
| Disallow | /ratecomment.aspx |
| Disallow | /rateit.aspx |
| Disallow | /receipt.aspx |
| Disallow | /remove.aspx |
| Disallow | /reorder.aspx |
| Disallow | /rorentity.aspx |
| Disallow | /rorindex.aspx |
| Disallow | /rortopics.aspx |
| Disallow | /rssfeed.aspx |
| Disallow | /sb.aspx |
| Disallow | /search.aspx |
| Disallow | /searchadv.aspx |
| Disallow | /searchnx.aspx |
| Disallow | /secureauth.aspx |
| Disallow | /secureform.aspx |
| Disallow | /secureprocess.aspx |
| Disallow | /selectaddress.aspx |
| Disallow | /sendform.aspx |
| Disallow | /setcurrency.aspx |
| Disallow | /setlocale.aspx |
| Disallow | /setvatsetting.aspx |
| Disallow | /shoppingcart.aspx |
| Disallow | /shoppingcart_change.aspx |
| Disallow | /smartcheckout2.aspx |
| Disallow | /smartcheckout3.aspx |
| Disallow | /signin.aspx |
| Disallow | /signout.aspx |
| Disallow | /stoneedge.aspx |
| Disallow | /tableorder_process.aspx |
| Disallow | /twocheckout_return.aspx |
| Disallow | /Version.aspx |
| Disallow | /wishlist.aspx |
| Disallow | /wolthuis.aspx |
| Disallow | /worldpayreturn.aspx |
| Disallow | /c-2299-discontinued-products.aspx |
| Disallow | /c-2299/discontinued-products.aspx |
| Disallow | /popShip.aspx |
| Disallow | /*.pdf$ |
| Disallow | /buy/ |
| Disallow | /ScriptResource.axd |
| Disallow | /WebResource.axd |
| Disallow | /wCheckoutBongo.aspx |
| Disallow | /wCustomerWelcome0.aspx |
| Disallow | /wCustomerWelcome1.aspx |
| Disallow | /wCustomerWelcome2.aspx |
| Disallow | /wEmailFriend.aspx |
| Disallow | /wEstimateShipping.aspx |
| Disallow | /wQuestion.aspx |
| Disallow | /wQuote.aspx |
| Disallow | /wTracking.aspx |
| Disallow | /wTracking2.aspx |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.cleanitsupply.com/sitemapxml.aspx |