Log In Sign Up

carfolio.com
robots.txt

Robots Exclusion Standard data for carfolio.com

Resource Scan

Scan Details

Site Domain carfolio.com
Base Domain carfolio.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-10-23T11:53:37+00:00
Next Scan 2025-01-21T11:53:37+00:00

Last Successful Scan

Scanned2023-04-03T07:52:34+00:00
URL https://carfolio.com/robots.txt
Redirect https://www.carfolio.com/robots.txt
Redirect Domain www.carfolio.com
Redirect Base carfolio.com
Domain IPs 104.21.12.65, 172.67.193.183
Redirect IPs 104.21.12.65, 172.67.193.183
Response IP 104.21.12.65
Found Yes
Hash 7e10c7411d2cd18765ec1a397b6f00093a950fddb91f2bc4c9ddce994a24ea15
SimHash c23790012c7d

Groups

bingpreview

Rule Path
Disallow /

paracrawl

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

grabber

Rule Path
Disallow /

riddler

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

java/1.6.0_20

Rule Path
Disallow /

www.integromedb.org/crawler

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sogou inst spider

Rule Path
Disallow /

sogou spider2

Rule Path
Disallow /

sogou blog

Rule Path
Disallow /

sogou news spider

Rule Path
Disallow /

sogou orion spider

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

pangusospider

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

import.io

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

httrack

Rule Path
Disallow /

import.io

Rule Path
Disallow /

sitesucker

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

httrack

Rule Path
Disallow /

wget

Rule Path
Disallow /

curl

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

grub-client

Rule Path
Disallow /specifications/

mj12bot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

accelobot

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

googlebot

Rule Path
Disallow /stable/

*

Rule Path
Disallow /rt3/
Disallow /classifieds/
Disallow /mail/
Disallow /diary/
Disallow /specifications/models/nl/
Disallow /specifications/models/de/
Disallow /specifications/models/fr/
Disallow /links/
Disallow /ebay/
Disallow /login/
Disallow /logout/
Disallow /register
Disallow /activate/
Disallow /privacy/
Disallow /scale/
Disallow /newlook/
Disallow /news/
Disallow /ads/
Disallow /downloads/
Disallow /test/
Disallow /pass
Disallow /press/
Disallow /specificatons/
Disallow /specifications/models/comp/
Disallow /6?*
Disallow /%E2%80%A6*
Disallow *tytu*
Disallow /newlook.dhtml
Disallow /*%26amp%3Bamp%3B
Disallow /specifications/models/car/?car=97251%2Ffi-fi%2F
Disallow /specifications/models/car/?car=193237%3Fpagewanted=all
Disallow /specifications/models/car/?car=92895%3Fpagewanted=all
Disallow /specifications/models/car/?car=995%E2%80%A6
Disallow /specifications/models/car/?car=5236%3Fiframe=true&width=100%&height=100%
Disallow /specifications/models/car/?car=175758%EF%BF%BD%DE%B8%3AWl%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD%C2%ADj%EF%BF%BD%EF%BF%BD%EF%BF%BD%E0%A8%9E%EF%BF%BD%EF%BF%BD%C2%B6&%C3%A5z%C3%8A%5Er%C3%8B%29y%C8%9F%EF%BF%BD%C6%AD%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD%C3%A7%21
Disallow /specifications/models/car/fr/?car=24288%3Fiframe=true&width=100%&height=100%
Disallow /specifications/models/car/fr/?car=26499%3Fiframe=true&width=100%&height=100%
Disallow /specifications/models/car/?car=120%E2%80%A6
Disallow /specifications/models/car/?car=775%E2%80%A6
Disallow /specifications/models/car/?car=107426%2Ffi-fi%2F
Disallow /specifications/models/car/?car=803%E2%80%A6
Disallow /specifications/models/car/?car=91751%2Ffi-fi%2F
Disallow /specifications/models/car/?car=104%E2%80%A6
Disallow /specifications/models/car/?car=168586%2Fno-no%2F
Disallow /specifications/models/car/?car=114708%7Cdata
Disallow /specifications/models/car/?car=dtcenuxbwij
Disallow /specifications/models/car/?car=107%E2%80%A6
Disallow /specifications/models/car/?car=52789%2F
Disallow /specifications/models/car/?car=%3C%5BFAVL%5Dsid%5B%2FFVAL%5D%3E
Disallow /specifications/models/car/?car=130%E2%80%A6
Disallow /specifications/models/car/?car=112816.
Disallow /specifications/models/car/?car=864%E2%80%A6

Other Records

Field Value
crawl-delay 30
Back to top

Comments

  • Disallow: /articles/
  • Disallow: /tools/
  • Disallow: /stable/
  • Disallow: /search/
  • Google
  • Google is a stupid bot. Really stupid.
  • Not google
  • Allow: /
Back to top