Log In Sign Up

crdai.com
robots.txt

Robots Exclusion Standard data for crdai.com

Resource Scan

Scan Details

Site Domain crdai.com
Base Domain crdai.com
Scan Status Ok
Last Scan2025-07-29T17:21:19+00:00
Next Scan 2025-08-28T17:21:19+00:00

Last Scan

Scanned2025-07-29T17:21:19+00:00
URL http://crdai.com/robots.txt
Domain IPs 38.11.170.44
Response IP 38.11.170.44
Found Yes
Hash 661c86d34014caf52bd0225c229d5b795cad7b3331d380caee1073ee50c952c1
SimHash 5935b16bd12a

Groups

semrushbot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

acoonbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

alexa

Rule Path
Disallow /

toolbar

Rule Path
Disallow /

apachebench

Rule Path
Disallow /

applebot

Rule Path
Disallow /

asktbfxtv

Rule Path
Disallow /

chinasospider

Rule Path
Disallow /

compspybot

Rule Path
Disallow /

coolpadwebkit

Rule Path
Disallow /

crawldaddy

Rule Path
Disallow /

curl

Rule Path
Disallow /

digext

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

ec2linkfinder

Rule Path
Disallow /

edisterbot

Rule Path
Disallow /

elefent

Rule Path
Disallow /

exabot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

feeddemon

Rule Path
Disallow /

feedly

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

httpclient

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

indy

Rule Path
Disallow /

library

Rule Path
Disallow /

jaunty

Rule Path
Disallow /

java

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

jorgee

Rule Path
Disallow /

lightdeckreports

Rule Path
Disallow /

bot

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

microsoft

Rule Path
Disallow /

url

Rule Path
Disallow /

control

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

msnbot-media

Rule Path
Disallow /

obot

Rule Path
Disallow /

perl

Rule Path
Disallow /

psbot

Rule Path
Disallow /

purebot

Rule Path
Disallow /

python

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

spbot

Rule Path
Disallow /

spiderman

Rule Path
Disallow /

swebot

Rule Path
Disallow /

swiftbot

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

turnitinbot-agent

Rule Path
Disallow /

universalfeedparser

Rule Path
Disallow /

wangidspider

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

webdup

Rule Path
Disallow /

wget

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

wsanalyzer

Rule Path
Disallow /

xbfmozilla

Rule Path
Disallow /

xenu

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yottaa

Rule Path
Disallow /

yunguance

Rule Path
Disallow /

yyspider

Rule Path
Disallow /

zmeu

Rule Path
Disallow /

googlebot

Rule Path
Disallow /

yahoo! slurp china

Rule Path
Disallow /

yahoo! slurp

Rule Path
Disallow /

msnbot

Rule Path
Disallow /

msnbot-media

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

ezooms.bot

Rule Path
Disallow /
Back to top