Log In Sign Up

autowares.com
robots.txt

Robots Exclusion Standard data for autowares.com

Resource Scan

Scan Details

Site Domain autowares.com
Base Domain autowares.com
Scan Status Ok
Last Scan2025-12-27T16:24:30+00:00
Next Scan 2026-01-26T16:24:30+00:00

Last Scan

Scanned2025-12-27T16:24:30+00:00
URL https://autowares.com/robots.txt
Domain IPs 63.239.186.59
Response IP 63.239.186.59
Found Yes
Hash ba41518575f806de16ea719f5e5c494a830c81d39cda39061c9bd14999104da4
SimHash 795ebc04a8ae

Groups

googlebot

Rule Path
Disallow /indeed-jobs

googlebot-image

Rule Path
Disallow /indeed-jobs

bingbot

Rule Path
Disallow /indeed-jobs

Other Records

Field Value
crawl-delay 60

slurp

Rule Path
Disallow /indeed-jobs
Disallow /doc
Disallow /img

Other Records

Field Value
crawl-delay 60

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

ahrefssiteaudit

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

uptimerobot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

awariobot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

wpbot

Rule Path
Disallow /

sidetrade

Rule Path
Disallow /

geedoproductsearch

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

facebookexternalhit

Rule Path
Disallow /

facebookcatalog

Rule Path
Disallow /

meta-externalfetcher

Rule Path
Disallow /

*

Rule Path
Disallow /indeed-jobs
Disallow /doc
Disallow /img
Disallow /baldwin
Disallow /standard-ignition

Other Records

Field Value
crawl-delay 90
Back to top