pronettoyeur.com
robots.txt

Robots Exclusion Standard data for pronettoyeur.com

Archived Snapshots

Resource Scan

Scan Details

Site Domain	pronettoyeur.com
Base Domain	pronettoyeur.com
Scan Status	Ok
Last Scan	2024-11-13T16:17:52+00:00
Next Scan	2024-11-20T16:17:52+00:00

Last Scan

Scanned	2024-11-13T16:17:52+00:00
URL	https://pronettoyeur.com/robots.txt
Domain IPs	91.134.190.187
Response IP	91.134.190.187
Found	Yes
Hash	56d5e8bdaa5fe75e202301dec7d9b525c61c4f7f3b38af3281190c1d9a241f52
SimHash	e0269da805f2

Groups

*

Rule	Path
Disallow	/wp-admin/
Allow	/wp-admin/admin-ajax.php
Disallow	/wp-login.php
Disallow	*/trackback
Disallow	/*/feed
Disallow	/*/comments
Disallow	/cgi-bin
Disallow	/*.php$
Disallow	/*.inc$
Disallow	/*.gz
Disallow	/*.cgi
Disallow	/view
Disallow	/documents/
Disallow	/*.pdf

Rule

Path

Disallow

/wp-admin/

Allow

/wp-admin/admin-ajax.php

Disallow

/wp-login.php

Disallow

*/trackback

Disallow

/*/feed

Disallow

/*/comments

Disallow

/cgi-bin

Disallow

/*.php$

Disallow

/*.inc$

Disallow

/*.gz

Disallow

/*.cgi

Disallow

/view

Disallow

/documents/

Disallow

/*.pdf

dotbot

Rule	Path
Disallow	/

Rule

Path

Disallow

giftghostbot

Rule	Path
Disallow	/

Rule

Path

Disallow

seznam

Rule	Path
Disallow	/

Rule

Path

Disallow

paperlibot

Rule	Path
Disallow	/

Rule

Path

Disallow

genieo

Rule	Path
Disallow	/

Rule

Path

Disallow

dataprovider/6.101

Rule	Path
Disallow	/

Rule

Path

Disallow

dataprovidersiteexplorer

Rule	Path
Disallow	/

Rule

Path

Disallow

dazoobot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

diffbot

Rule	Path
Disallow	/

Rule

Path

Disallow

domainstatsbot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

dotbot/1.1

Rule	Path
Disallow	/

Rule

Path

Disallow

dubaiindex

Rule	Path
Disallow	/

Rule

Path

Disallow

ecommercebot

Rule	Path
Disallow	/

Rule

Path

Disallow

expertsearchspider

Rule	Path
Disallow	/

Rule

Path

Disallow

feedbin

Rule	Path
Disallow	/

Rule

Path

Disallow

fetch/2.0a

Rule	Path
Disallow	/

Rule

Path

Disallow

ffbot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

focusbot/1.1

Rule	Path
Disallow	/

Rule

Path

Disallow

huaweisymantecspider

Rule	Path
Disallow	/

Rule

Path

Disallow

huaweisymantecspider/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

jobdiggerspider

Rule	Path
Disallow	/

Rule

Path

Disallow

lemurwebcrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

lipperheylinkexplorer

Rule	Path
Disallow	/

Rule

Path

Disallow

lssrocketcrawler/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

lyt.srv1.5

Rule	Path
Disallow	/

Rule

Path

Disallow

miadev/0.0.1

Rule	Path
Disallow	/

Rule

Path

Disallow

najdi.si/3.1

Rule	Path
Disallow	/

Rule

Path

Disallow

bountiibot

Rule	Path
Disallow	/

Rule

Path

Disallow

experibot_v1

Rule	Path
Disallow	/

Rule

Path

Disallow

bixocrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

bixocrawler testcrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

crawler4j

Rule	Path
Disallow	/

Rule

Path

Disallow

crowsnest/0.5

Rule	Path
Disallow	/

Rule

Path

Disallow

cukbot

Rule	Path
Disallow	/

Rule

Path

Disallow

dataprovider/6.92

Rule	Path
Disallow	/

Rule

Path

Disallow

dblbot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

diffbot/0.1

Rule	Path
Disallow	/

Rule

Path

Disallow

digg deeper/v1

Rule	Path
Disallow	/

Rule

Path

Disallow

discobot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

discobot/1.1

Rule	Path
Disallow	/

Rule

Path

Disallow

discobot/2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

discoverybot/2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

dlvr.it/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

domainstatsbot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

drupact/0.7

Rule	Path
Disallow	/

Rule

Path

Disallow

ezooms/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

fastbot crawler beta 2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

fastbot crawler beta 4.0

Rule

Path

Disallow

feedly social

Rule

Path

Disallow

feedly/1.0

Rule

Path

Disallow

feedlybot/1.0

Rule

Path

Disallow

feedspot

Rule

Path

Disallow

feedspotbot/1.0

Rule

Path

Disallow

clickagy intelligence bot v2

Rule

Path

Disallow

classbot

Rule

Path

Disallow

cispa vulnerability notification

Rule

Path

Disallow

cirrusexplorer/1.1

Rule

Path

Disallow

checksem/nutch-1.10

Rule

Path

Disallow

catchbot/5.0

Rule

Path

Disallow

catchbot/3.0

Rule

Path

Disallow

catchbot/2.0

Rule

Path

Disallow

catchbot/1.0

Rule

Path

Disallow

camontspider/1.0

Rule

Path

Disallow

buzzbot/1.0

Rule

Path

Disallow

buzzbot

Rule

Path

Disallow

businessseek.biz_spider

Rule

Path

Disallow

bubing

Rule

Path

Disallow

fyberspider/1.3

Rule

Path

Disallow

findlinks/1.1.6-beta5

Rule

Path

Disallow

g2reader-bot/1.0

Rule

Path

Disallow

findlinks/1.1.6-beta6

Rule

Path

Disallow

findlinks/2.0

Rule

Path

Disallow

findlinks/2.0.1

Rule

Path

Disallow

findlinks/2.0.2

Rule

Path

Disallow

findlinks/2.0.4

Rule

Path

Disallow

findlinks/2.0.5

Rule

Path

Disallow

findlinks/2.0.9

Rule

Path

Disallow

findlinks/2.1

Rule

Path

Disallow

findlinks/2.1.5

Rule

Path

Disallow

findlinks/2.1.3

Rule

Path

Disallow

findlinks/2.2

Rule

Path

Disallow

findlinks/2.5

Rule

Path

Disallow

findlinks/2.6

Rule

Path

Disallow

ffbot/1.0

Rule

Path

Disallow

findlinks/1.0

Rule

Path

Disallow

findlinks/1.1.3-beta8

Rule

Path

Disallow

findlinks/1.1.3-beta9

Rule

Path

Disallow

findlinks/1.1.4-beta7

Rule

Path

Disallow

findlinks/1.1.6-beta1

Rule

Path

Disallow

findlinks/1.1.6-beta1 yacy

Rule

Path

Disallow

findlinks/1.1.6-beta2

Rule

Path

Disallow

findlinks/1.1.6-beta3

Rule

Path

Disallow

findlinks/1.1.6-beta4

Rule

Path

Disallow

bixo

Rule

Path

Disallow

bixolabs/1.0

Rule

Path

Disallow

crawlera/1.10.2

Rule

Path

Disallow

dataprovider site explorer

Rule

Path

Disallow

Allow

/*css?*

Allow

/*js?*

Other Records

Field

Value

sitemap

https://pronettoyeur.com/sitemap_index.xml

Comments

Block Bad Bots

Warnings

8 invalid lines.

pronettoyeur.comrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

*

dotbot

giftghostbot

seznam

paperlibot

genieo

dataprovider/6.101

dataprovidersiteexplorer

dazoobot/1.0

diffbot

domainstatsbot/1.0

dotbot/1.1

dubaiindex

ecommercebot

expertsearchspider

feedbin

fetch/2.0a

ffbot/1.0

focusbot/1.1

huaweisymantecspider

huaweisymantecspider/1.0

jobdiggerspider

lemurwebcrawler

lipperheylinkexplorer

lssrocketcrawler/1.0

lyt.srv1.5

miadev/0.0.1

najdi.si/3.1

bountiibot

experibot_v1

bixocrawler

bixocrawler testcrawler

crawler4j

crowsnest/0.5

cukbot

dataprovider/6.92

dblbot/1.0

diffbot/0.1

digg deeper/v1

discobot/1.0

discobot/1.1

discobot/2.0

discoverybot/2.0

dlvr.it/1.0

domainstatsbot/1.0

drupact/0.7

ezooms/1.0

fastbot crawler beta 2.0

fastbot crawler beta 4.0

feedly social

feedly/1.0

feedlybot/1.0

feedspot

feedspotbot/1.0

clickagy intelligence bot v2

classbot

cispa vulnerability notification

cirrusexplorer/1.1

checksem/nutch-1.10

catchbot/5.0

catchbot/3.0

catchbot/2.0

catchbot/1.0

camontspider/1.0

buzzbot/1.0

buzzbot

businessseek.biz_spider

bubing

fyberspider/1.3

findlinks/1.1.6-beta5

g2reader-bot/1.0

findlinks/1.1.6-beta6

findlinks/2.0

findlinks/2.0.1

findlinks/2.0.2

pronettoyeur.com
robots.txt