westinautomotive.com
robots.txt

Robots Exclusion Standard data for westinautomotive.com

Resource Scan

Scan Details

Site Domain westinautomotive.com
Base Domain westinautomotive.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-10-11T00:47:01+00:00
Next Scan 2025-01-09T00:47:01+00:00

Last Successful Scan

Scanned2023-11-24T00:45:34+00:00
URL https://westinautomotive.com/robots.txt
Redirect https://www.westinautomotive.com/robots.txt
Redirect Domain www.westinautomotive.com
Redirect Base westinautomotive.com
Domain IPs 151.101.130.132, 151.101.194.132, 151.101.2.132, 151.101.66.132
Redirect IPs 151.101.130.132, 151.101.194.132, 151.101.2.132, 151.101.66.132
Response IP 199.232.46.132
Found Yes
Hash 10e0d0fff5cadeab8b4411ea114be04cce3d12195aa220125860b4cd4a39354e
SimHash 621d9881d093

Groups

*

Rule Path
Disallow

mediapartners-google

Rule Path
Disallow

Other Records

Field Value
crawl-delay 15

googlebot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 15

adsbot-google

Rule Path
Disallow

Other Records

Field Value
crawl-delay 15

googlebot-image

Rule Path
Disallow

Other Records

Field Value
crawl-delay 15

googlebot-mobile

Rule Path
Disallow

Other Records

Field Value
crawl-delay 15

msnbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 15

bingbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 15

slurp

Rule Path
Disallow

Other Records

Field Value
crawl-delay 15

yahoo! slurp

Rule Path
Disallow

Other Records

Field Value
crawl-delay 15

ahrefsbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 15

mj12bot

Rule Path
Disallow /

Other Records

Field Value
sitemap http://www.westinautomotive.com/sitemap.xml