don-men.com
robots.txt

Robots Exclusion Standard data for don-men.com

Archived Snapshots

Resource Scan

Scan Details

Site Domain	don-men.com
Base Domain	don-men.com
Scan Status	Ok
Last Scan	2024-09-23T13:51:13+00:00
Next Scan	2024-10-23T13:51:13+00:00

Last Scan

Scanned	2024-09-23T13:51:13+00:00
URL	https://don-men.com/robots.txt
Redirect	https://www.don-men.com/robots.txt
Redirect Domain	www.don-men.com
Redirect Base	don-men.com
Domain IPs	185.92.193.187
Redirect IPs	185.92.193.187
Response IP	185.92.193.187
Found	Yes
Hash	4e29a9a0fef713d91225abb6e928e0ae265ca4453b2a608cf292fd0f43596dd4
SimHash	6350685862a0

Groups

*

Rule	Path
Allow	/
Disallow	/index.php?route=extension%2Fmodule%2Fgmt_datalayer%2F*

Rule

Path

Allow

Disallow

/index.php?route=extension%2Fmodule%2Fgmt_datalayer%2F*

webprosbot

Rule	Path
Disallow	/

Rule

Path

Disallow

gptbot

Rule	Path
Disallow	/

Rule

Path

Disallow

claude-web

Rule	Path
Disallow	/

Rule

Path

Disallow

imagesiftbot

Rule	Path
Disallow	/

Rule

Path

Disallow

timpibot

Rule	Path
Disallow	/

Rule

Path

Disallow

sidetrade indexer bot

Rule	Path
Disallow	/

Rule

Path

Disallow

sidetrade indexer bot

Rule	Path
Disallow	/

Rule

Path

Disallow

cohere-ai

Rule	Path
Disallow	/

Rule

Path

Disallow

perplexitybot

Rule	Path
Disallow	/

Rule

Path

Disallow

diffbot

Rule	Path
Disallow	/

Rule

Path

Disallow

omgili

Rule	Path
Disallow	/

Rule

Path

Disallow

omgilibot

Rule	Path
Disallow	/

Rule

Path

Disallow

www.aihitdata.com

Rule	Path
Disallow	/

Rule

Path

Disallow

anthropicbot

Rule	Path
Disallow	/

Rule

Path

Disallow

bingbot

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	5

Field

Value

crawl-delay

pinterestbot

Rule	Path
Disallow	/

Rule

Path

Disallow

claude

Rule	Path
Disallow	/

Rule

Path

Disallow

claudebot

Rule	Path
Disallow	/

Rule

Path

Disallow

netestate ne crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

chatgpt-user

Rule	Path
Disallow	/

Rule

Path

Disallow

webprosbot/2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

crawler_eb_germany_2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

anthropic

Rule	Path
Disallow	/

Rule

Path

Disallow

anthropic-ai

Rule	Path
Disallow	/

Rule

Path

Disallow

spawning

Rule	Path
Disallow	/

Rule

Path

Disallow

yandex

Rule	Path
Disallow	/

Rule

Path

Disallow

bytespider

Rule	Path
Disallow	/

Rule

Path

Disallow

sogou web spider

Rule	Path
Disallow	/

Rule

Path

Disallow

sogou inst spider

Rule	Path
Disallow	/

Rule

Path

Disallow

coccocbot-web

Rule	Path
Disallow	/

Rule

Path

Disallow

yandexbot

Rule	Path
Disallow	/

Rule

Path

Disallow

yandexcalendar

Rule	Path
Disallow	/

Rule

Path

Disallow

yandexmobilebot

Rule	Path
Disallow	/

Rule

Path

Disallow

yandexadnet

Rule	Path
Disallow	/

Rule

Path

Disallow

yandexblogs

Rule	Path
Disallow	/

Rule

Path

Disallow

yandexdirect

Rule	Path
Disallow	/

Rule

Path

Disallow

yandexfordomain

Rule	Path
Disallow	/

Rule

Path

Disallow

yandeximages

Rule	Path
Disallow	/

Rule

Path

Disallow

yandeximageresizer

Rule	Path
Disallow	/

Rule

Path

Disallow

yandexmarket

Rule	Path
Disallow	/

Rule

Path

Disallow

yandexnews

Rule	Path
Disallow	/

Rule

Path

Disallow

yandexontodb

Rule	Path
Disallow	/

Rule

Path

Disallow

yandexpagechecker

Rule	Path
Disallow	/

Rule

Path

Disallow

yandexsitelinks

Rule	Path
Disallow	/

Rule

Path

Disallow

yandexspravbot

Rule	Path
Disallow	/

Rule

Path

Disallow

yandexturbo

Rule	Path
Disallow	/

Rule

Path

Disallow

yandexvertis

Rule	Path
Disallow	/

Rule

Path

Disallow

yandexverticals

Rule

Path

Disallow

yandexvideo

Rule

Path

Disallow

yandexwebmaster

Rule

Path

Disallow

crmnlcrawlagent

Rule

Path

Disallow

cliqzbot

Rule

Path

Disallow

ccbot

Rule

Path

Disallow

haosouspider

Rule

Path

Disallow

dotbot

Rule

Path

Disallow

barkrowler

Rule

Path

Disallow

spawning-ai

Rule

Path

Disallow

grapeshot

Rule

Path

Disallow

proximic

Rule

Path

Disallow

linkpadbot

Rule

Path

Disallow

sunrise

Rule

Path

Disallow

butterfly

Rule

Path

Disallow

linguee bot

Rule

Path

Disallow

twengabot-2.0

Rule

Path

Disallow

turnitinbot

Rule

Path

Disallow

blexbot

Rule

Path

Disallow

amznkassocbot/4.0

Rule

Path

Disallow

nerdybot

Rule

Path

Disallow

trovitbot

Rule

Path

Disallow

shopwiki

Rule

Path

Disallow

xovibot

Rule

Path

Disallow

smtbot

Rule

Path

Disallow

meanpathbot

Rule

Path

Disallow

mixrankbot

Rule

Path

Disallow

easouspider

Rule

Path

Disallow

riddler

Rule

Path

Disallow

seznambot

Rule

Path

Disallow

swiftbot

Rule

Path

Disallow

qwantify

Rule

Path

Disallow

psbot

Rule

Path

Disallow

zoominfobot

Rule

Path

Disallow

seekportbot

Rule

Path

Disallow

mediatoolkitbot

Rule

Path

Disallow

netseer

Rule

Path

Disallow

backlinkcrawler

Rule

Path

Disallow

surdotlybot

Rule

Path

Disallow

megaindex.ru/2.0

Rule

Path

Disallow

megaindex.ru

Rule

Path

Disallow

yacybot

Rule

Path

Disallow

queryseekerspider

Rule

Path

Disallow

wbsearchbot

Rule

Path

Disallow

deusu

Rule

Path

Disallow

tdjbot

Rule

Path

Disallow

sogou

Rule

Path

Disallow

uptimerobot

Rule

Path

Disallow

yeti

Rule

Path

Disallow

velenpublicwebcrawler

Rule

Path

Disallow

serpstatbot

Rule

Path

Disallow

blexbot

Rule

Path

Disallow

baiduspider

Rule

Path

Disallow

baiduspider-video

Rule

Path

Disallow

baiduspider-image

Rule

Path

Disallow

baiduspider-news

Rule

Path

Disallow

baiduspider-favo

Rule

Path

Disallow

baiduspider-ads

Rule

Path

Disallow

baiduspider-cpro

Rule

Path

Disallow

adreview/1.0

Rule

Path

Disallow

aihitbot/

Rule

Path

Disallow

aiva/1.0

Rule

Path

Disallow

awariorssbot
awariosmartbot
awariobot

Rule

Path

Disallow

dataforseobot

Rule

Path

Disallow

daum

Rule

Path

Disallow

seznambot

Rule

Path

Disallow

mail.ru_bot

Rule

Path

Disallow

mail.ru

Rule

Path

Disallow

yak

Rule

Path

Disallow

mj12bot

Rule

Path

Disallow

archive.org_bot

Rule

Path

Disallow

ioncrawl

Rule

Path

Disallow

velenpublicwebcrawler

Rule

Path

Disallow

netcraftsurveyagent

Rule

Path

Disallow

checkmarknetwork/1.0 (+https://www.checkmarknetwork.com/spider.html)

Rule

Path

Disallow

pagepeeker

Rule

Path

Disallow

python-urllib/2.6

Rule

Path

Disallow

python-requests/2.31.0

Rule

Path

Disallow

python-requests

Rule

Path

Disallow

python-requests/2.28.1

Rule

Path

Disallow

python/3.9 aiohttp/3.8.3

Rule

Path

Disallow

python-requests/2.27.1

Rule

Path

Disallow

python-requests/2.21.0

Rule

Path

Disallow

python-requests/2.20.1

Rule

Path

Disallow

python-urllib/3.8

Rule

Path

Disallow

python-urllib/3.9

Rule

Path

Disallow

python-requests/2.17.3

Rule

Path

Disallow

python-requests/2.28.1

Rule

Path

Disallow

python-requests/2.26.0

Rule

Path

Disallow

linguee

Rule

Path

Disallow

moget

Rule

Path

Disallow

yodaobot

Rule

Path

Disallow

obot

Rule

Path

Disallow

addthis

Rule

Path

Disallow

wotbox

Rule

Path

Disallow

embedly

Rule

Path

Disallow

paperlibot

Rule

Path

Disallow

genieo

Rule

Path

Disallow

showyoubot

Rule

Path

Disallow

ichiro

Rule

Path

Disallow

tweetmemebot

Rule

Path

Disallow

bdcbot

Rule

Path

Disallow

bdcbot/1.0

Rule

Path

Disallow

rogerbot

Rule

Path

Disallow

spbot

Rule

Path

Disallow

linkdexbot

Rule

Path

Disallow

exabot

Rule

Path

Disallow

surveybot

Rule

Path

Disallow

sistrix

Rule

Path

Disallow

python/3.8 aiohttp/3.8.4

Rule

Path

Disallow

python-requests/1.2.3 cpython/2.7.3 linux/3.3.8-gcg-201308121035

Rule

Path

Disallow

python-requests/1.2.3 cpython/2.7.2+ linux/3.0.0-16-virtual

Rule

Path

Disallow

python-requests/1.2.3 cpython/2.7.4 linux/3.8.11-ec2

Rule

Path

Disallow

python-urllib/2.7

Rule

Path

Disallow

python-httplib2/0.7.7 (gzip)

Rule

Path

Disallow

python-requests/1.1.0 cpython/2.6.8 linux/3.4.48-45.46.amzn1.x86_64

Rule

Path

Disallow

Other Records

Field

Value

sitemap

https://www.don-men.com/sitemap.xml

sitemap

https://www.don-men.com/sitemaptags.xml

sitemap

https://www.don-men.com/sitemapsearch.xml

Warnings

14 invalid lines.

don-men.comrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

*

webprosbot

gptbot

claude-web

imagesiftbot

timpibot

sidetrade indexer bot

sidetrade indexer bot

cohere-ai

perplexitybot

diffbot

omgili

omgilibot

www.aihitdata.com

anthropicbot

bingbot

Other Records

pinterestbot

claude

claudebot

netestate ne crawler

chatgpt-user

webprosbot/2.0

crawler_eb_germany_2.0

anthropic

anthropic-ai

spawning

yandex

bytespider

sogou web spider

sogou inst spider

coccocbot-web

yandexbot

yandexcalendar

yandexmobilebot

yandexadnet

yandexblogs

yandexdirect

yandexfordomain

yandeximages

yandeximageresizer

yandexmarket

yandexnews

yandexontodb

yandexpagechecker

yandexsitelinks

yandexspravbot

yandexturbo

yandexvertis

yandexverticals

yandexvideo

yandexwebmaster

crmnlcrawlagent

cliqzbot

ccbot

haosouspider

dotbot

barkrowler

spawning-ai

grapeshot

proximic

linkpadbot

sunrise

butterfly

linguee bot

twengabot-2.0

turnitinbot

blexbot

amznkassocbot/4.0

nerdybot

trovitbot

shopwiki

xovibot

smtbot

meanpathbot

don-men.com
robots.txt

awariorssbot
awariosmartbot
awariobot