am-autoparts.com
robots.txt

Robots Exclusion Standard data for am-autoparts.com

Resource Scan

Scan Details

Site Domain am-autoparts.com
Base Domain am-autoparts.com
Scan Status Ok
Last Scan2024-09-15T11:22:46+00:00
Next Scan 2024-10-15T11:22:46+00:00

Last Scan

Scanned2024-09-15T11:22:46+00:00
URL https://am-autoparts.com/robots.txt
Redirect https://www.am-autoparts.com/robots.txt
Redirect Domain www.am-autoparts.com
Redirect Base am-autoparts.com
Domain IPs 13.107.246.59
Redirect IPs 13.107.246.59, 2620:1ec:bdf::59
Response IP 13.107.246.59
Found Yes
Hash f46d368b8684f594dbdaee48125b8700f32eb35ce47ad18fa356bb93c76233bb
SimHash 6c50d013e685

Groups

*

Rule Path
Disallow /cart.html
Disallow /register.html
Disallow /retrieve_password.html
Disallow /checkout/
Disallow /checkout.html
Disallow /checkout/shippingOptions.html
Disallow /account/password.html
Disallow /transfercart.html
Disallow /search.html
Disallow /my_account.html
Disallow /cart.html
Disallow /vehicles/state/update
Disallow /vehicles/makes.json
Disallow /vehicles/models.json
Disallow /vehicles/years.json

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

scrapy

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.am-autoparts.com/sitemap.xml.gz

Comments

  • "Any empty [Disallow] value, indicates that all URLs can be retrieved.
  • At least one Disallow field needs to be present in a record . "