carjudge.com
robots.txt

Robots Exclusion Standard data for carjudge.com

Resource Scan

Scan Details

Site Domain carjudge.com
Base Domain carjudge.com
Scan Status Ok
Last Scan2025-09-15T23:44:35+00:00
Next Scan 2025-10-15T23:44:35+00:00

Last Scan

Scanned2025-09-15T23:44:35+00:00
URL https://carjudge.com/robots.txt
Redirect https://dwdextkgwmnwh.cloudfront.net/cb4ffa1a-c81a-43e2-9903-1384147a2bf0/robots.txt
Redirect Domain dwdextkgwmnwh.cloudfront.net
Redirect Base dwdextkgwmnwh.cloudfront.net
Domain IPs 13.49.131.183, 13.53.211.199
Redirect IPs 108.156.139.120, 108.156.139.152, 108.156.139.78, 108.156.139.92, 2600:9000:2755:2600:d:c80:a040:21, 2600:9000:2755:2800:d:c80:a040:21, 2600:9000:2755:3400:d:c80:a040:21, 2600:9000:2755:4a00:d:c80:a040:21, 2600:9000:2755:5000:d:c80:a040:21, 2600:9000:2755:b800:d:c80:a040:21, 2600:9000:2755:da00:d:c80:a040:21, 2600:9000:2755:fa00:d:c80:a040:21
Response IP 108.156.139.152
Found Yes
Hash 7da8b31ab5a71730e1f053576452d4cc2f47dc3167437c87f27b6d3e2fcace6a
SimHash 401ccec3c798

Groups

*

Rule Path
Disallow /reports/view/
Disallow /private/
Disallow /tmp/

googlebot

Rule Path
Disallow

bingbot

Rule Path
Disallow

duckduckbot

Rule Path
Disallow

applebot

Rule Path
Disallow

slurp

Product Comment
slurp Yahoo's bot
Rule Path
Disallow

sogou

Rule Path
Disallow

exabot

Rule Path
Disallow

ia_archiver

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

waybackmachine

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

Comments

  • === Default rules for all bots ===
  • === Allow major search engines ===
  • === Block archivers and aggressive crawlers ===