Log In Sign Up

marx-brothers.org
robots.txt

Robots Exclusion Standard data for marx-brothers.org

Resource Scan

Scan Details

Site Domain marx-brothers.org
Base Domain marx-brothers.org
Scan Status Ok
Last Scan2024-10-28T04:33:20+00:00
Next Scan 2024-11-27T04:33:20+00:00

Last Scan

Scanned2024-10-28T04:33:20+00:00
URL https://marx-brothers.org/robots.txt
Domain IPs 2001:8d8:100f:f000::2cc, 217.160.0.193
Response IP 217.160.0.193
Found Yes
Hash 46fca5cc2c9946f86a65c56fde67815dbd3c51e1f943511fd337031effcd585c
SimHash 49b7c2c096a3

Groups

*

Rule Path
Disallow /forbidden/
Disallow /falle.htm
Disallow /watching/movie_makers.htm
Disallow /watching/movie_makers.htm?*
Disallow /de/market_feedback.htm
Disallow /market_feedback.htm
Disallow /whyaduck/crosswords/
Disallow /whyaduck/hangman/
Disallow /whyaduck/forum/
Disallow /whyaduck/cgi-bin/
Disallow /whyaduck/concentration/
Disallow /crosswords/
Disallow /concentration/
Disallow /survey/
Disallow /hangman/
Disallow /cgi-bin/
Disallow /schreiben.htm
Disallow /living/
Disallow /pages/external/
Disallow /info/watching/
Disallow /living/
Disallow /watching/people/
Disallow /*?external_page
Disallow /*?shop_country
Disallow /*?movie_id
Disallow /audio/sounds/*.mp3
Disallow /biography/*.kmz
Disallow /degrees/
Disallow /music/
Allow /music/sheetmusic.htm
Disallow /watching/people/
Disallow /topics1.htm
Disallow /uk/
Disallow /geocities/

sitebot

Rule Path
Disallow /
Disallow /biography/harposp.htm
Allow /index.htm
Allow /biography/
Allow /watching/

hailoobot

Rule Path
Disallow /
Disallow /biography/harposp.htm
Allow /index.htm
Allow /biography/
Allow /watching/

baiduspider

Rule Path
Disallow /
Allow /index.htm

spbot

Rule Path
Disallow /
Allow /index.htm

mp3bot

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /
Allow /content.htm
Allow /index.htm

purebot

Rule Path
Disallow /
Allow /index.htm

mlbot

Rule Path
Disallow /

gingercrawler

Rule Path
Disallow /

yanga

Rule Path
Disallow /

busiversebot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

sapphirewebcrawler

Rule Path
Disallow /

gurujibot

Rule Path
Disallow /

googlebot

Rule Path
Disallow /*?css
Disallow /*%26css
Disallow /forbidden/
Disallow /falle.htm
Disallow /watching/movie_makers.htm
Disallow /watching/movie_makers.htm?*
Disallow /de/market_feedback.htm
Disallow /market_feedback.htm
Disallow /whyaduck/crosswords/
Disallow /whyaduck/hangman/
Disallow /whyaduck/cgi-bin/
Disallow /whyaduck/concentration/
Disallow /whyaduck/forum/
Disallow /crosswords/
Disallow /pages/external/
Disallow /concentration/
Disallow /survey/
Disallow /hangman/
Disallow /cgi-bin/
Disallow /schreiben.htm
Disallow /living/
Disallow /info/watching/
Disallow /watching/people/
Disallow /*?external_page
Disallow /*?shop_country
Disallow /*?movie_id
Disallow /audio/sounds/*.mp3
Disallow /biography/*.kmz
Disallow /music/
Disallow /uk/
Disallow /geocities/
Allow /music/sheetmusic.htm

bingbot

Rule Path
Disallow /*?css
Disallow /*%26css
Disallow /forbidden/
Disallow /falle.htm
Disallow /watching/movie_makers.htm
Disallow /watching/movie_makers.htm?*
Disallow /de/market_feedback.htm
Disallow /market_feedback.htm
Disallow /whyaduck/crosswords/
Disallow /whyaduck/hangman/
Disallow /whyaduck/cgi-bin/
Disallow /whyaduck/concentration/
Disallow /whyaduck/forum/
Disallow /crosswords/
Disallow /pages/external/
Disallow /concentration/
Disallow /survey/
Disallow /hangman/
Disallow /cgi-bin/
Disallow /schreiben.htm
Disallow /living/
Disallow /info/watching/
Disallow /watching/people/
Disallow /*?external_page
Disallow /*?shop_country
Disallow /*?movie_id
Disallow /audio/sounds/*.mp3
Disallow /biography/*.kmz
Disallow /music/
Disallow /uk/
Disallow /geocities/
Allow /music/sheetmusic.htm

catchbot

Rule Path
Disallow /

acontbot

Rule Path
Disallow /

gsa-crawler

Rule Path
Disallow /

aisearchbot

Rule Path
Disallow /

slurp

Rule Path
Disallow /forbidden/
Disallow /falle.htm
Disallow /watching/movie_makers.htm
Disallow /watching/movie_makers.htm?*
Disallow /de/market_feedback.htm
Disallow /market_feedback.htm
Disallow /whyaduck/crosswords/
Disallow /whyaduck/sounds/
Disallow /whyaduck/forum/
Allow /whyaduck/sounds/midi.htm
Allow /whyaduck/sounds/wav.htm
Allow /whyaduck/sounds/streaming.htm
Allow /whyaduck/sounds/video.htm
Disallow /whyaduck/hangman/
Disallow /whyaduck/cgi-bin/
Disallow /whyaduck/concentration/
Disallow /crosswords/
Disallow /concentration/
Disallow /hangman/
Disallow /cgi-bin/
Disallow /whyaduck/crosswords/*
Disallow /whyaduck/hangman/*
Disallow /whyaduck/cgi-bin/*
Disallow /whyaduck/concentration/*
Disallow /pages/external/
Disallow /crosswords/*
Disallow /concentration/*
Disallow /hangman/*
Disallow /cgi-bin/*
Disallow /schreiben.htm
Disallow /living/
Disallow /info/watching/
Disallow /living/
Disallow /watching/movie_detail.htm?*$
Disallow /watching/movie_detail.htm?*
Disallow /*?external_page
Disallow /*?shop_country
Disallow /*?movie_id
Disallow /music/imageset.html
Disallow /music/
Allow /music/sheetmusic.htm
Disallow /listening/show_music.htm?*
Disallow /watching/people/
Disallow /uk/
Disallow /geocities/

Other Records

Field Value
crawl-delay 10

yeti

Rule Path
Disallow /
Allow /index.htm

tagoobot

Rule Path
Disallow /
Allow /index.htm

naverbot

Rule Path
Disallow /
Allow /index.htm

oozbot

Rule Path
Disallow /
Allow /index.htm

iccrawler - icjobs

Rule Path
Disallow /

balihoo

Rule Path
Disallow /

synoobot

Rule Path
Disallow /

converacrawler

Rule Path
Disallow /

nextgensearchbot

Rule Path
Disallow /

favorstarbot/1.0

Rule Path
Disallow /

snapbot

Rule Path
Disallow /

shablast

Rule Path
Disallow /

webalta

Rule Path
Disallow /

yandex

Rule Path
Disallow /whyaduck

voyager

Rule Path
Disallow /

yodaobot

Rule Path
Disallow /

depspid

Rule Path
Disallow /

distribujob

Rule Path
Disallow /

ivia

Rule Path
Disallow /

datafountains

Rule Path
Disallow /

jambot

Rule Path
Disallow /

yawhen

Rule Path
Disallow /

omniexplorer_bot/6.9.3

Rule Path
Disallow /

page_verifier

Rule Path
Disallow /

webaroobot

Rule Path
Disallow /

searchdaimon.com-dc

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

shim-crawler

Rule Path
Disallow /

sbider

Rule Path
Disallow /

becomebot

Rule Path
Disallow /

galaxybot

Rule Path
Disallow /

semanticdiscovery

Rule Path
Disallow /

activetouristbot

Rule Path
Disallow /

amzn_assoc

Rule Path
Disallow /watching/
Disallow /living/
Disallow /de/living/
Disallow /acting/
Disallow /memo/
Disallow /watching/movie_makers.htm
Disallow /watching/movie_detail.htm
Disallow /de/market_feedback.htm
Disallow /market_feedback.htm
Disallow /de/compose_mail.htm
Disallow /pages/external/
Disallow /compose_mail.htm
Disallow /content.htm
Disallow /links.htm
Disallow /market.htm
Disallow /de/links.htm
Disallow /de/market.htm
Disallow /music/
Disallow /watching/people/
Disallow /geocities/

quepasacreep

Rule Path
Disallow /living/
Disallow /de/
Disallow /acting/
Disallow /memo/
Disallow /watching/
Disallow /reading/
Disallow /listening/
Disallow /market_feedback.htm
Disallow /de/compose_mail.htm
Disallow /compose_mail.htm
Disallow /pages/external/
Disallow /content.htm
Disallow /links.htm
Disallow /market.htm
Disallow /de/links.htm
Disallow /de/market.htm
Disallow /music/
Disallow /watching/people/
Disallow /geocities/
Disallow /whyaduck/

naverbot

Rule Path
Disallow /de/
Disallow /acting/
Disallow /memo/
Disallow /watching/
Disallow /pages/external/
Disallow /reading/
Disallow /listening/
Disallow /market_feedback.htm
Disallow /music/
Disallow /watching/people/
Disallow /geocities/

converacrawler

Rule Path
Disallow /de/
Disallow /acting/
Disallow /memo/
Disallow /watching/
Disallow /reading/
Disallow /listening/
Disallow /market_feedback.htm
Disallow /pages/external/
Disallow /music/
Disallow /watching/people/
Disallow /geocities/

iccrawler

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

http://www.uni-koblenz.de/~flocke/robot-info.txt

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap http://www.marx-brothers.org/sitemap.txt
Back to top

Warnings

  • 4 invalid lines.
Back to top