madlaine.fr
robots.txt

Robots Exclusion Standard data for madlaine.fr

Archived Snapshots

Resource Scan

Scan Details

Site Domain	madlaine.fr
Base Domain	madlaine.fr
Scan Status	Ok
Last Scan	2024-09-23T23:54:46+00:00
Next Scan	2024-10-23T23:54:46+00:00

Last Scan

Scanned	2024-09-23T23:54:46+00:00
URL	https://madlaine.fr/robots.txt
Redirect	https://www.madlaine.fr/robots.txt
Redirect Domain	www.madlaine.fr
Redirect Base	madlaine.fr
Domain IPs	46.101.141.162
Redirect IPs	46.101.141.162
Response IP	46.101.141.162
Found	Yes
Hash	60081fd42fcdec9db2f953741f5d6cfc7c5dbe294f97e90a823427a9e566c513
SimHash	74945bcbc778

Groups

*

Rule	Path
Allow	/misc/*.css$
Allow	/misc/*.css?
Allow	/misc/*.js$
Allow	/misc/*.js?
Allow	/misc/*.gif
Allow	/misc/*.jpg
Allow	/misc/*.jpeg
Allow	/misc/*.png
Allow	/modules/*.css$
Allow	/modules/*.css?
Allow	/modules/*.js$
Allow	/modules/*.js?
Allow	/modules/*.gif
Allow	/modules/*.jpg
Allow	/modules/*.jpeg
Allow	/modules/*.png
Allow	/profiles/*.css$
Allow	/profiles/*.css?
Allow	/profiles/*.js$
Allow	/profiles/*.js?
Allow	/profiles/*.gif
Allow	/profiles/*.jpg
Allow	/profiles/*.jpeg
Allow	/profiles/*.png
Allow	/themes/*.css$
Allow	/themes/*.css?
Allow	/themes/*.js$
Allow	/themes/*.js?
Allow	/themes/*.gif
Allow	/themes/*.jpg
Allow	/themes/*.jpeg
Allow	/themes/*.png
Disallow	/includes/
Disallow	/misc/
Disallow	/modules/
Disallow	/profiles/
Disallow	/scripts/
Disallow	/themes/
Disallow	/CHANGELOG.txt
Disallow	/cron.php
Disallow	/INSTALL.mysql.txt
Disallow	/INSTALL.pgsql.txt
Disallow	/INSTALL.sqlite.txt
Disallow	/install.php
Disallow	/INSTALL.txt
Disallow	/LICENSE.txt
Disallow	/MAINTAINERS.txt
Disallow	/update.php
Disallow	/UPGRADE.txt
Disallow	/xmlrpc.php
Disallow	/admin/
Disallow	/comment/reply/
Disallow	/filter/tips/
Disallow	/node/add/
Disallow	/search/
Disallow	/user/register/
Disallow	/user/password/
Disallow	/user/login/
Disallow	/user/logout/
Disallow	/promo/
Disallow	/nouveau/
Disallow	/?q=admin%2F
Disallow	/?q=comment%2Freply%2F
Disallow	/?q=filter%2Ftips%2F
Disallow	/?q=node%2Fadd%2F
Disallow	/?q=search%2F
Disallow	/?q=user%2Fpassword%2F
Disallow	/?q=user%2Fregister%2F
Disallow	/?q=user%2Flogin%2F
Disallow	/?q=user%2Flogout%2F
Disallow	/*modal

Rule

Path

Allow

/misc/*.css$

Allow

/misc/*.css?

Allow

/misc/*.js$

Allow

/misc/*.js?

Allow

/misc/*.gif

Allow

/misc/*.jpg

Allow

/misc/*.jpeg

Allow

/misc/*.png

Allow

/modules/*.css$

Allow

/modules/*.css?

Allow

/modules/*.js$

Allow

/modules/*.js?

Allow

/modules/*.gif

Allow

/modules/*.jpg

Allow

/modules/*.jpeg

Allow

/modules/*.png

Allow

/profiles/*.css$

Allow

/profiles/*.css?

Allow

/profiles/*.js$

Allow

/profiles/*.js?

Allow

/profiles/*.gif

Allow

/profiles/*.jpg

Allow

/profiles/*.jpeg

Allow

/profiles/*.png

Allow

/themes/*.css$

Allow

/themes/*.css?

Allow

/themes/*.js$

Allow

/themes/*.js?

Allow

/themes/*.gif

Allow

/themes/*.jpg

Allow

/themes/*.jpeg

Allow

/themes/*.png

Disallow

/includes/

Disallow

/misc/

Disallow

/modules/

Disallow

/profiles/

Disallow

/scripts/

Disallow

/themes/

Disallow

/CHANGELOG.txt

Disallow

/cron.php

Disallow

/INSTALL.mysql.txt

Disallow

/INSTALL.pgsql.txt

Disallow

/INSTALL.sqlite.txt

Disallow

/install.php

Disallow

/INSTALL.txt

Disallow

/LICENSE.txt

Disallow

/MAINTAINERS.txt

Disallow

/update.php

Disallow

/UPGRADE.txt

Disallow

/xmlrpc.php

Disallow

/admin/

Disallow

/comment/reply/

Disallow

/filter/tips/

Disallow

/node/add/

Disallow

/search/

Disallow

/user/register/

Disallow

/user/password/

Disallow

/user/login/

Disallow

/user/logout/

Disallow

/promo/

Disallow

/nouveau/

Disallow

/?q=admin%2F

Disallow

/?q=comment%2Freply%2F

Disallow

/?q=filter%2Ftips%2F

Disallow

/?q=node%2Fadd%2F

Disallow

/?q=search%2F

Disallow

/?q=user%2Fpassword%2F

Disallow

/?q=user%2Fregister%2F

Disallow

/?q=user%2Flogin%2F

Disallow

/?q=user%2Flogout%2F

Disallow

/*modal

Other Records

Field	Value
crawl-delay	10

Field

Value

crawl-delay

atomic_email_hunter/4.0

Rule	Path
Disallow	/

Rule

Path

Disallow

atspider/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

autoemailspider

Rule	Path
Disallow	/

Rule

Path

Disallow

bwh3_user_agent

Rule	Path
Disallow	/

Rule

Path

Disallow

china local browse 2.6

Rule	Path
Disallow	/

Rule

Path

Disallow

contactbot/0.2

Rule	Path
Disallow	/

Rule

Path

Disallow

contentsmartz

Rule	Path
Disallow	/

Rule

Path

Disallow

datacha0s/2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

dbrowse 1.4b

Rule	Path
Disallow	/

Rule

Path

Disallow

dbrowse 1.4d

Rule	Path
Disallow	/

Rule

Path

Disallow

demo bot dot 16b

Rule	Path
Disallow	/

Rule

Path

Disallow

demo bot z 16b

Rule	Path
Disallow	/

Rule

Path

Disallow

dsurf15a 01

Rule	Path
Disallow	/

Rule

Path

Disallow

dsurf15a 71

Rule	Path
Disallow	/

Rule

Path

Disallow

dsurf15a 81

Rule	Path
Disallow	/

Rule

Path

Disallow

dsurf15a va

Rule	Path
Disallow	/

Rule

Path

Disallow

ebrowse 1.4b

Rule	Path
Disallow	/

Rule

Path

Disallow

educate search vxb

Rule	Path
Disallow	/

Rule

Path

Disallow

emailsiphon

Rule	Path
Disallow	/

Rule

Path

Disallow

emailspider

Rule	Path
Disallow	/

Rule

Path

Disallow

emailwolf 1.00

Rule	Path
Disallow	/

Rule

Path

Disallow

esurf15a 15

Rule	Path
Disallow	/

Rule

Path

Disallow

extractorpro

Rule	Path
Disallow	/

Rule

Path

Disallow

franklin locator 1.8

Rule	Path
Disallow	/

Rule

Path

Disallow

fsurf15a 01

Rule	Path
Disallow	/

Rule

Path

Disallow

full web bot 0416b

Rule	Path
Disallow	/

Rule

Path

Disallow

full web bot 0516b

Rule	Path
Disallow	/

Rule

Path

Disallow

full web bot 2816b

Rule	Path
Disallow	/

Rule

Path

Disallow

guestbook auto submitter

Rule	Path
Disallow	/

Rule

Path

Disallow

industry program 1.0.x

Rule	Path
Disallow	/

Rule

Path

Disallow

infoconveracrawler/0.8 ( http://www.authoritativeweb.com/crawl)

Rule	Path
Disallow	/

Rule

Path

Disallow

isc systems irc search 2.1

Rule	Path
Disallow	/

Rule

Path

Disallow

iupui research bot v 1.9a

Rule	Path
Disallow	/

Rule

Path

Disallow

larbin-experimental (efp@gmx.net)

Rule	Path
Disallow	/

Rule

Path

Disallow

letscrawl.com/1.0 +http://letscrawl.com/

Rule	Path
Disallow	/

Rule

Path

Disallow

lincoln state web browser

Rule	Path
Disallow	/

Rule

Path

Disallow

lmqueuebot/0.2

Rule	Path
Disallow	/

Rule

Path

Disallow

lwp::simple/5.803

Rule	Path
Disallow	/

Rule

Path

Disallow

mac finder 1.0.xx

Rule	Path
Disallow	/

Rule

Path

Disallow

mfc foundation class library 4.0

Rule	Path
Disallow	/

Rule

Path

Disallow

microsoft url control â 6.00.8xxx

Rule	Path
Disallow	/

Rule

Path

Disallow

missauga locate 1.0.0

Rule	Path
Disallow	/

Rule

Path

Disallow

missigua locator 1.9

Rule	Path
Disallow	/

Rule

Path

Disallow

missouri college browse

Rule	Path
Disallow	/

Rule

Path

Disallow

mizzu labs 2.2

Rule	Path
Disallow	/

Rule

Path

Disallow

mo college 1.9

Rule	Path
Disallow	/

Rule

Path

Disallow

mozilla/2.0 (compatible; newt activex; win32)

Rule

Path

Disallow

mozilla/3.0 (compatible)

Rule

Path

Disallow

mozilla/3.0 (compatible; indy library)

Rule

Path

Disallow

mozilla/3.0 (compatible; scan4mail (advanced version) http://www.peterspages.net/?scan4mail)

Rule

Path

Disallow

mozilla/4.0 (compatible; advanced email extractor v2.xx)

Rule

Path

Disallow

mozilla/4.0 (compatible; iplexx spider/1.0 http://www.iplexx.at)

Rule

Path

Disallow

mozilla/4.0 (compatible; msie 5.0; windows nt; digext; dts agent

Rule

Path

Disallow

mozilla/4.0 (compatible; msie 5.5; windows nt 5.0) fetch api request

Rule

Path

Disallow

mozilla/4.0 efp@gmx.net

Rule

Path

Disallow

mozilla/4.08 [en] (win98; u ;nav)

Rule

Path

Disallow

mozilla/5.0 (version: xxxx type:xx)

Rule

Path

Disallow

mvaclient

Rule

Path

Disallow

nameofagent (cms spider)

Rule

Path

Disallow

nasa search 1.0

Rule

Path

Disallow

nsauditor/1.x

Rule

Path

Disallow

pbrowse 1.4b

Rule

Path

Disallow

peval 1.4b

Rule

Path

Disallow

poirot

Rule

Path

Disallow

port huron labs

Rule

Path

Disallow

production bot 0116b

Rule

Path

Disallow

production bot 2016b

Rule

Path

Disallow

production bot dot 3016b

Rule

Path

Disallow

program shareware 1.0.2

Rule

Path

Disallow

psurf15a 11

Rule

Path

Disallow

psurf15a 51

Rule

Path

Disallow

psurf15a va

Rule

Path

Disallow

psycheclone

Rule

Path

Disallow

rsurf15a 41

Rule

Path

Disallow

rsurf15a 51

Rule

Path

Disallow

rsurf15a 81

Rule

Path

Disallow

searchbot admin@google.com

Rule

Path

Disallow

shablastbot 1.0

Rule

Path

Disallow

snap.com beta crawler v0

Rule

Path

Disallow

snapbot/1.0

Rule

Path

Disallow

snapbot/1.0 (snap shots, +http://www.snap.com)

Rule

Path

Disallow

sogou develop spider

Rule

Path

Disallow

sogou orion spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm

Product

Comment

sogou orion spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm

07)

Rule

Path

Disallow

sogou spider

Rule

Path

Disallow

sogou web spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm

Product

Comment

sogou web spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm

07)

Rule

Path

Disallow

sohu agent

Rule

Path

Disallow

ssurf15a 11

Rule

Path

Disallow

tsurf15a 11

Rule

Path

Disallow

under the rainbow 2.2

Rule

Path

Disallow

user-agent: mozilla/4.0 (compatible; msie 6.0; windows nt 5.1)

Rule

Path

Disallow

vadixbot

Rule

Path

Disallow

webvulncrawl.unknown/1.0 libwww-perl/5.803

Rule

Path

Disallow

wells search ii

Rule

Path

Disallow

wep search 00

Rule

Path

Disallow

atomic_email_hunter/4.0

Rule

Path

Disallow

atspider/1.0

Rule

Path

Disallow

autoemailspider

Rule

Path

Disallow

bwh3_user_agent

Rule

Path

Disallow

china local browse 2.6

Rule

Path

Disallow

contactbot/0.2

Rule

Path

Disallow

contentsmartz

Rule

Path

Disallow

datacha0s/2.0

Rule

Path

Disallow

dbrowse 1.4b

Rule

Path

Disallow

dbrowse 1.4d

Rule

Path

Disallow

demo bot dot 16b

Rule

Path

Disallow

demo bot z 16b

Rule

Path

Disallow

dsurf15a 01

Rule

Path

Disallow

dsurf15a 71

Rule

Path

Disallow

dsurf15a 81

Rule

Path

Disallow

dsurf15a va

Rule

Path

Disallow

ebrowse 1.4b

Rule

Path

Disallow

educate search vxb

Rule

Path

Disallow

emailsiphon

Rule

Path

Disallow

emailspider

Rule

Path

Disallow

emailwolf 1.00

Rule

Path

Disallow

esurf15a 15

Rule

Path

Disallow

extractorpro

Rule

Path

Disallow

franklin locator 1.8

Rule

Path

Disallow

fsurf15a 01

Rule

Path

Disallow

full web bot 0416b

Rule

Path

Disallow

full web bot 0516b

Rule

Path

Disallow

full web bot 2816b

Rule

Path

Disallow

guestbook auto submitter

Rule

Path

Disallow

industry program 1.0.x

Rule

Path

Disallow

infoconveracrawler/0.8 ( http://www.authoritativeweb.com/crawl)

Rule

Path

Disallow

isc systems irc search 2.1

Rule

Path

Disallow

iupui research bot v 1.9a

Rule

Path

Disallow

larbin-experimental (efp@gmx.net)

Rule

Path

Disallow

letscrawl.com/1.0 +http://letscrawl.com/

Rule

Path

Disallow

lincoln state web browser

Rule

Path

Disallow

lmqueuebot/0.2

Rule

Path

Disallow

lwp::simple/5.803

Rule

Path

Disallow

mac finder 1.0.xx

Rule

Path

Disallow

mfc foundation class library 4.0

Rule

Path

Disallow

microsoft url control â 6.00.8xxx

Rule

Path

Disallow

missauga locate 1.0.0

Rule

Path

Disallow

missigua locator 1.9

Rule

Path

Disallow

missouri college browse

Rule

Path

Disallow

mizzu labs 2.2

Rule

Path

Disallow

mo college 1.9

Rule

Path

Disallow

mozilla/2.0 (compatible; newt activex; win32)

Rule

Path

Disallow

mozilla/3.0 (com

Rule

Path

Disallow

mozilla/3.0 (compatible; indy library)

Rule

Path

Disallow

mozilla/3.0 (compatible; scan4mail (advanced version) http://www.peterspages.net/?scan4mail)

Rule

Path

Disallow

mozilla/4.0 (compatible; advanced email extractor v2.xx)

Rule

Path

Disallow

mozilla/4.0 (compatible; iplexx spider/1.0 http://www.iplexx.at)

Rule

Path

Disallow

mozilla/4.0 (compatible; msie 5.0; windows nt; digext; dts agent

Rule

Path

Disallow

mozilla/4.0 (compatible; msie 5.5; windows nt 5.0) fetch api request

Rule

Path

Disallow

mozilla/4.0 efp@gmx.net

Rule

Path

Disallow

mozilla/4.08 [en] (win98; u ;nav)

Rule

Path

Disallow

mozilla/5.0 (version: xxxx type:xx)

Rule

Path

Disallow

mvaclient

Rule

Path

Disallow

nameofagent (cms spider)

Rule

Path

Disallow

nasa search 1.0

Rule

Path

Disallow

nsauditor/1.x

Rule

Path

Disallow

pbrowse 1.4b

Rule

Path

Disallow

peval 1.4b

Rule

Path

Disallow

poirot

Rule

Path

Disallow

port huron labs

Rule

Path

Disallow

production bot 0116b

Rule

Path

Disallow

production bot 2016b

Rule

Path

Disallow

production bot dot 3016b

Rule

Path

Disallow

program shareware 1.0.2

Rule

Path

Disallow

psurf15a 11

Rule

Path

Disallow

psurf15a 51

Rule

Path

Disallow

psurf15a va

Rule

Path

Disallow

psycheclone

Rule

Path

Disallow

rsurf15a 41

Rule

Path

Disallow

rsurf15a 51

Rule

Path

Disallow

rsurf15a 81

Rule

Path

Disallow

searchbot admin@google.com

Rule

Path

Disallow

shablastbot 1.0

Rule

Path

Disallow

snap.com beta crawler v0

Rule

Path

Disallow

snapbot/1.0

Rule

Path

Disallow

snapbot/1.0 (snap shots, +http://www.snap.com)

Rule

Path

Disallow

sogou develop spider

Rule

Path

Disallow

sogou orion spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm

Product

Comment

sogou orion spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm

07)

Rule

Path

Disallow

sogou spider

Rule

Path

Disallow

sogou web spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm

Product

Comment

sogou web spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm

07)

Rule

Path

Disallow

sohu agent

Rule

Path

Disallow

ssurf15a 11

Rule

Path

Disallow

tsurf15a 11

Rule

Path

Disallow

under the rainbow 2.2

Rule

Path

Disallow

user-agent: mozilla/4.0 (compatible; msie 6.0; windows nt 5.1)

Rule

Path

Disallow

vadixbot

Rule

Path

Disallow

webvulncrawl.unknown/1.0 libwww-perl/5.803

Rule

Path

Disallow

wells search ii

Rule

Path

Disallow

wep search 00

Rule

Path

Disallow

semrushbot

Rule

Path

Disallow

facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)

Rule

Path

Disallow

facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

Rule

Path

Disallow

facebookexternalhit/1.1

Rule

Path

Disallow

msiecrawler

Rule

Path

Disallow

offline explorer

Rule

Path

Disallow

sitecheck.internetseer.com

Rule

Path

Disallow

teleport

Rule

Path

Disallow

teleportpro

Rule

Path

Disallow

webcopier

Rule

Path

Disallow

webstripper

Rule

Path

Disallow

zealbot

Rule

Path

Disallow

asknread.com

Rule

Path

Disallow

ellisphere

Rule

Path

Disallow

spotter

Rule

Path

Disallow

omgilibot

Rule

Path

Disallow

omgili

Rule

Path

Disallow

ccbot

Rule

Path

Disallow

perplexitybot

Rule

Path

Disallow

bytespider

Rule

Path

Disallow

diffbot

Rule

Path

Disallow

facebookbot

Rule

Path

Disallow

youbot

Rule

Path

Disallow

anthropic-ai

Rule

Path

Disallow

claude-web

Rule

Path

Disallow

claudebot

Rule

Path

Disallow

cohere-ai

Rule

Path

Disallow

Other Records

Field

Value

sitemap

https://www.madlaine.fr/sitemap.xml

Comments

robots.txt
This file is to prevent the crawling and indexing of certain parts
of your site by web crawlers and spiders run by sites like Yahoo!
and Google. By telling these "robots" where not to go on your site,
you save bandwidth and server resources.
This file will be ignored unless it is at the root of your host:
Used: http://example.com/robots.txt
Ignored: http://example.com/site/robots.txt
For more information about the robots.txt standard, see:
http://www.robotstxt.org/robotstxt.html
CSS, JS, Images
Directories
Files
Paths (clean URLs)
Paths (no clean URLs)
modal pages
robotstxt_antibadbot begin
robotstxt_antibadbot end

Warnings

5 invalid lines.

madlaine.frrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

*

Other Records

atomic_email_hunter/4.0

atspider/1.0

autoemailspider

bwh3_user_agent

china local browse 2.6

contactbot/0.2

contentsmartz

datacha0s/2.0

dbrowse 1.4b

dbrowse 1.4d

demo bot dot 16b

demo bot z 16b

dsurf15a 01

dsurf15a 71

dsurf15a 81

dsurf15a va

ebrowse 1.4b

educate search vxb

emailsiphon

emailspider

emailwolf 1.00

esurf15a 15

extractorpro

franklin locator 1.8

fsurf15a 01

full web bot 0416b

full web bot 0516b

full web bot 2816b

guestbook auto submitter

industry program 1.0.x

infoconveracrawler/0.8 ( http://www.authoritativeweb.com/crawl)

isc systems irc search 2.1

iupui research bot v 1.9a

larbin-experimental (efp@gmx.net)

letscrawl.com/1.0 +http://letscrawl.com/

lincoln state web browser

lmqueuebot/0.2

lwp::simple/5.803

mac finder 1.0.xx

mfc foundation class library 4.0

microsoft url control â 6.00.8xxx

missauga locate 1.0.0

missigua locator 1.9

missouri college browse

mizzu labs 2.2

mo college 1.9

mozilla/2.0 (compatible; newt activex; win32)

mozilla/3.0 (compatible)

mozilla/3.0 (compatible; indy library)

mozilla/3.0 (compatible; scan4mail (advanced version) http://www.peterspages.net/?scan4mail)

mozilla/4.0 (compatible; advanced email extractor v2.xx)

mozilla/4.0 (compatible; iplexx spider/1.0 http://www.iplexx.at)

mozilla/4.0 (compatible; msie 5.0; windows nt; digext; dts agent

mozilla/4.0 (compatible; msie 5.5; windows nt 5.0) fetch api request

mozilla/4.0 efp@gmx.net

mozilla/4.08 [en] (win98; u ;nav)

mozilla/5.0 (version: xxxx type:xx)

mvaclient

nameofagent (cms spider)

nasa search 1.0

nsauditor/1.x

pbrowse 1.4b

peval 1.4b

poirot

port huron labs

production bot 0116b

production bot 2016b

production bot dot 3016b

program shareware 1.0.2

psurf15a 11

psurf15a 51

psurf15a va

psycheclone

madlaine.fr
robots.txt

microsoft url control â 6.00.8xxx

microsoft url control â 6.00.8xxx