designcrowd.com
robots.txt

Robots Exclusion Standard data for designcrowd.com

Resource Scan

Scan Details

Site Domain designcrowd.com
Base Domain designcrowd.com
Scan Status Ok
Last Scan2024-05-27T22:15:57+00:00
Next Scan 2024-06-10T22:15:57+00:00

Last Scan

Scanned2024-05-27T22:15:57+00:00
URL https://designcrowd.com/robots.txt
Redirect https://www.designcrowd.com:443/robots.txt
Redirect Domain www.designcrowd.com
Redirect Base designcrowd.com
Domain IPs 13.227.254.19, 13.227.254.24, 13.227.254.47, 13.227.254.61
Redirect IPs 13.227.254.19, 13.227.254.24, 13.227.254.47, 13.227.254.61
Response IP 13.227.254.19
Found Yes
Hash c5278b8c506093d198152506e1c099a83c8107a95893158d1a5d4786df355209
SimHash 4290fb6303c1

Groups

trovitbot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

psbot

Rule Path
Disallow /

seoengbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

careerbot

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

spbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

compspybot

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

zumbot

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

obot

Rule Path
Disallow /

changedetection

Rule Path
Disallow /

slurp

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

proximic

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

blekkobot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

adbeat_bot

Rule Path
Disallow /

peerindex

Rule Path
Disallow /

genieo

Rule Path
Disallow /

www.integromedb.org/crawler

Rule Path
Disallow /

queryseekerspider

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

kraken/0.1

Rule Path
Disallow /

steeler

Rule Path
Disallow /

flipboard

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

coccoc

Rule Path
Disallow /

bdcbot

Rule Path
Disallow /

zenbot

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

memorybot

Rule Path
Disallow /

sputnikbot

Rule Path
Disallow /

jooblebot

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

crazywebcrawler

Rule Path
Disallow /

backlinkcrawler

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

riddler

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

netseer

Rule Path
Disallow /

aboundexbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

daumoa

Rule Path
Disallow /

surdotlybot

Rule Path
Disallow /

znajdzfoto

Rule Path
Disallow /

jamesbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

griffinbot

Rule Path
Disallow /

deusu

Rule Path
Disallow /

fatbot

Rule Path
Disallow /

icc-crawler

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

femtosearchbot

Rule Path
Disallow /

buckyohare

Rule Path
Disallow /

semanticscholarbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

bubing

Rule Path
Disallow /

alphaseobot

Rule Path
Disallow /

alphaseobot-sa

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

linguee

Rule Path
Disallow /

applebot

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

lcc

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

linespider

Rule Path
Disallow /

adsbot

Rule Path
Disallow /

neevabot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

*

Rule Path
Disallow /cgi-bin/
Disallow /admin/
Disallow /*.axd
Disallow /WebResource.axd
Disallow /ScriptResource.axd
Disallow /search.aspx*
Disallow /report-issue-re-design.aspx?*
Disallow /manage-project/
Disallow /manage-account/
Disallow /designer-account/
Disallow /manage-profile/
Disallow /customer-handover/
Disallow /designer-handover/
Disallow /submit-design/
Disallow /login.aspx?returnUrl*
Disallow /login?returnUrl*
Disallow /s/
Disallow /poll/
Disallow /manage-poll/
Disallow /vote/

Other Records

Field Value
sitemap https://www.designcrowd.com/sitemap-index.xml

Comments

  • robots.txt

Warnings

  • 4 invalid lines.