caranddriver.com
robots.txt

Robots Exclusion Standard data for caranddriver.com

Resource Scan

Scan Details

Site Domain caranddriver.com
Base Domain caranddriver.com
Scan Status Ok
Last Scan2024-11-02T09:49:27+00:00
Next Scan 2024-11-09T09:49:27+00:00

Last Scan

Scanned2024-11-02T09:49:27+00:00
URL https://caranddriver.com/robots.txt
Domain IPs 151.101.0.155, 151.101.128.155, 151.101.192.155, 151.101.64.155
Response IP 151.101.192.155
Found Yes
Hash 0fbd88cf35d668ad091734ff5edeeac30be678c934246f9c72120df05827839d
SimHash 1922176405d3

Groups

mediapartners-google

Rule Path
Disallow

facebot

Rule Path
Disallow

twitterbot

Rule Path
Disallow

ccbot

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /au/
Disallow /cn/
Disallow /dk/
Disallow /en/
Disallow /es/
Disallow /in/
Disallow /it/
Disallow /jp/
Disallow /ng/
Disallow /nl/
Disallow /no/
Disallow /se/
Disallow /tw/
Disallow /ua/
Disallow /uk/

chatgpt-user

Rule Path
Disallow /au/
Disallow /cn/
Disallow /dk/
Disallow /en/
Disallow /es/
Disallow /in/
Disallow /it/
Disallow /jp/
Disallow /ng/
Disallow /nl/
Disallow /no/
Disallow /se/
Disallow /tw/
Disallow /ua/
Disallow /uk/

gptbot

Rule Path
Disallow /au/
Disallow /cn/
Disallow /dk/
Disallow /en/
Disallow /es/
Disallow /in/
Disallow /it/
Disallow /jp/
Disallow /ng/
Disallow /nl/
Disallow /no/
Disallow /se/
Disallow /tw/
Disallow /ua/
Disallow /uk/

*

Rule Path
Disallow /_p/
Disallow /afflink/
Disallow /ajax/
Disallow /auth/
Disallow /au/ajax/
Disallow /cn/ajax/
Disallow /dk/ajax/
Disallow /en/ajax/
Disallow /es/ajax/
Disallow /in/ajax/
Disallow /it/ajax/
Disallow /jp/ajax/
Disallow /ng/ajax/
Disallow /nl/ajax/
Disallow /no/ajax/
Disallow /se/ajax/
Disallow /tw/ajax/
Disallow /ua/ajax/
Disallow /uk/ajax/
Disallow /landing-feed/
Disallow /oauth/
Disallow /preview/
Disallow /au/preview/
Disallow /cn/preview/
Disallow /dk/preview/
Disallow /en/preview/
Disallow /es/preview/
Disallow /in/preview/
Disallow /it/preview/
Disallow /jp/preview/
Disallow /ng/preview/
Disallow /nl/preview/
Disallow /no/preview/
Disallow /se/preview/
Disallow /tw/preview/
Disallow /ua/preview/
Disallow /uk/preview/
Disallow /search
Disallow /transporter/
Disallow /vehicles/
Allow /ajax/content-product/
Allow /au/ajax/content-product/
Allow /cn/ajax/content-product/
Allow /dk/ajax/content-product/
Allow /en/ajax/content-product/
Allow /es/ajax/content-product/
Allow /in/ajax/content-product/
Allow /it/ajax/content-product/
Allow /jp/ajax/content-product/
Allow /ng/ajax/content-product/
Allow /nl/ajax/content-product/
Allow /no/ajax/content-product/
Allow /se/ajax/content-product/
Allow /tw/ajax/content-product/
Allow /ua/ajax/content-product/
Allow /uk/ajax/content-product/
Allow /ajax/contentmedia/
Allow /au/ajax/contentmedia/
Allow /cn/ajax/contentmedia/
Allow /dk/ajax/contentmedia/
Allow /en/ajax/contentmedia/
Allow /es/ajax/contentmedia/
Allow /in/ajax/contentmedia/
Allow /it/ajax/contentmedia/
Allow /jp/ajax/contentmedia/
Allow /ng/ajax/contentmedia/
Allow /nl/ajax/contentmedia/
Allow /no/ajax/contentmedia/
Allow /se/ajax/contentmedia/
Allow /tw/ajax/contentmedia/
Allow /ua/ajax/contentmedia/
Allow /uk/ajax/contentmedia/

Other Records

Field Value
sitemap https://www.caranddriver.com/sitemap_index.xml
sitemap https://www.caranddriver.com/sitemap_google_news.xml