Log In Sign Up

fsmrd.fm
robots.txt

Robots Exclusion Standard data for fsmrd.fm

Resource Scan

Scan Details

Site Domain fsmrd.fm
Base Domain fsmrd.fm
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-10-02T02:35:14+00:00
Next Scan 2024-12-31T02:35:14+00:00

Last Successful Scan

Scanned2023-05-14T17:45:11+00:00
URL https://fsmrd.fm/robots.txt
Domain IPs 132.148.212.196
Response IP 132.148.212.196
Found Yes
Hash ae9189d84fc9dea310507a0e090e427629dd24ca6104f8071e9be49c222f5368
SimHash 6cca106ac3a3

Groups

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

baiduspider

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

baidu

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

jooblebot

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

orangebot

Rule Path
Disallow /

seznam

Rule Path
Disallow /

sogou

Rule Path
Disallow /

spbot

Rule Path
Disallow /

vebidoobot

Rule Path
Disallow /

vegi bot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yodao

Rule Path
Disallow /

youdao

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

adbeat_bot

Rule Path
Disallow /

agentlinkspammer

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

ahrefsbot/4.0

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

aihitbot/1.0

Rule Path
Disallow /

aihitbot/1.1

Rule Path
Disallow /

acoon

Rule Path
Disallow /

arachmo

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider+

Rule Path
Disallow /

baiduspider+(+http://www.baidu.com/search/spider.htm)

Rule Path
Disallow /

baiduspider/2.0;+http://www.baidu.com/search/spider.html

Rule Path
Disallow /

baiduspider/2.0

Rule Path
Disallow /

mozilla/5.0(compatible; baiduspider/2.0; +http://www.baidu.com/search/spider.html)

Rule Path
Disallow /

careerbot

Rule Path
Disallow /

comodospider/nutch-1.2

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

exabot/3.0

Rule Path
Disallow /

exalead

Rule Path
Disallow /

exalead crawler

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

ezooms/1.0

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mozilla/5.0 (compatible; ezooms/1.0; ezooms.bot[at]gmail[dot]com)

Rule Path
Disallow /

findlinks/2.6

Rule Path
Disallow /

java/1.6.0_04

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

kaloogabot

Rule Path
Disallow /

mail.ru_bot/2.0

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

mail.ru_bot/2.0; +http://go.mail.ru/help/robots

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mj12bot/v1.4.3

Rule Path
Disallow /

moget

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

ichiro 3.0

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

netcraftsurveyagent/1.0

Rule Path
Disallow /

openwebindex/nutch-1.6

Rule Path
Disallow /

openwebindex

Rule Path
Disallow /

panoptastudybot

Rule Path
Disallow /

checks.panopta.com

Rule Path
Disallow /

psbot

Rule Path
Disallow /

plukkie

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

seznambot/1.0

Rule Path
Disallow /

seznambot/1.1

Rule Path
Disallow /

seznambot/3.0

Rule Path
Disallow /

sistrixcrawler

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

sistrix crawler

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sosospider+

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

sosospider/2.0

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

wada.vn

Rule Path
Disallow /

wada.vn vietnamese search

Rule Path
Disallow /

wada.vn vietnamese search/2.1

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yandex/1.01.001

Rule Path
Disallow /

yandexbot/3.0-mirrordetector

Rule Path
Disallow /

yandeximages/3.0

Rule Path
Disallow /

yandexsomething/1.

Rule Path
Disallow /

yandex.com

Rule Path
Disallow /

yandexbot/3.0

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

youdaobot/1.0

Rule Path
Disallow /

youdaobot/1.0

Rule Path
Disallow /

zao

Rule Path
Disallow /

netcraftsurveyagent

Rule Path
Disallow /

duckduckbot/1.1

Rule Path
Disallow /

duckduckbot/1.0

Rule Path
Disallow /
Back to top

Comments

  • AgentLinkSpammer
  • AhrefsBot ads
  • aiHitBot Ukraine or Russia
  • Acoon Germany
  • Arachmo Japan
  • Baiduspider China and Japan
  • careerbot Germany
  • COMODOSpider/Nutch-1.2 United Kingdom
  • EasouSpider - China
  • Exabot/3.0 - France proxy scraper
  • Exalead proxy scraper France
  • Ezooms and dotbot
  • findlinks/2.6 Germany http://wortschatz.uni-leipzig.de/findlinks
  • Java/1.6.0_04
  • JikeSpider China
  • KaloogaBot Netherlands contextual advertising
  • Mail.RU_Bot/2.0 Russia
  • Mail.RU Russia
  • Mail.Ru Russia
  • MJ12bot United Kingdom
  • MJ12bot/v1.4.3 United Kingdon
  • Ichiro Japan
  • Ichiro 3.0 Japan
  • NetcraftSurveyAgent/1.0
  • OpenWebIndex/Nutch-1.6 Germany
  • panoptaStudyBot checks.panopta.com monitor
  • panoptaStudyBot checks.panopta.com monitor
  • picsearch Sweden searches for pictures
  • plukkie Dutch (botje.nl)/Belgium (botje.be)/France (botje.fr)/United Kingdom (botje.co.uk) search engine
  • SeznamBot Czech Republic
  • SeznamBot/3.0
  • SistrixCrawler Germany DE
  • Sogou
  • Sosospider - China http://help.soso.com/webspider.htm
  • Sosospider - China
  • Sosospider/2.0 - China may not obey robots.txt
  • 360Spider China
  • SurveyBot
  • Wada.vn Vietnamese Search/2.1
  • Yandex
  • YisouSpider China
  • YoudaoBot/1.0 China
  • YoudaoBot China
  • Zao - Japan
Back to top

Warnings

  • 8 invalid lines.
Back to top