21centuryautogroup.com
robots.txt

Robots Exclusion Standard data for 21centuryautogroup.com

Resource Scan

Scan Details

Site Domain 21centuryautogroup.com
Base Domain 21centuryautogroup.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-03-23T10:11:21+00:00
Next Scan 2024-06-21T10:11:21+00:00

Last Successful Scan

Scanned2022-09-14T23:56:48+00:00
URL http://21centuryautogroup.com/robots.txt
Response IP 199.184.144.99
Found Yes
Hash f33b437f6e37ef07c91b9c5bcb2b4f48bed368f5716758ff1716248630faffd4
SimHash 6d5418b2c6b0

Groups

googlebot
googlebot-mobile
slurp
msnbot
bingbot
adidxbot
teoma
adsbot-google
mediapartners-google
motominerbot

Rule Path
Disallow /*.pdf
Disallow /admin/
Disallow /secure/
Disallow /tcd/
Disallow /staff.cfm

Other Records

Field Value
crawl-delay 5

*

Rule Path
Disallow /

Other Records

Field Value
sitemap /sitemap.cfm

Warnings

  • `request-rate` is not a known field.