Log In Sign Up

haborok.net
robots.txt

Robots Exclusion Standard data for haborok.net

Resource Scan

Scan Details

Site Domain haborok.net
Base Domain haborok.net
Scan Status Ok
Last Scan2024-09-24T22:50:32+00:00
Next Scan 2024-10-24T22:50:32+00:00

Last Scan

Scanned2024-09-24T22:50:32+00:00
URL https://haborok.net/robots.txt
Domain IPs 46.161.52.243
Response IP 46.161.52.243
Found Yes
Hash aae7c1a5dc3c5c64f9360e27867dd616da5061c8ab41d276634c48cf701dd19e
SimHash 5270a410b02e

Groups

*

Rule Path
Disallow /coins/*
Disallow /archive/*
Disallow /new/*
Disallow /collector/*
Disallow /bonistika/*
Disallow /cleaning/*
Disallow /accessories/*
Disallow /medals/*
Disallow /meteorites/*
Disallow /library/*
Disallow /detectors/*
Disallow /maps/*
Disallow /equipment/*
Disallow /coils/*
Disallow /gps/*
Disallow /magnets/*
Disallow /rent/*
Disallow /manuals/*
Disallow /camo/*
Disallow /knives/*
Disallow /lighters/*
Disallow /dosimeter/*
Disallow /power_supply/*
Disallow /touristware/*
Disallow /coins/*
Disallow /forum/admin/
Disallow /forum/cache/
Disallow /forum/converge_local/
Disallow /forum/hooks/
Disallow /forum/ips_kernel/
Disallow /forum/retail/
Disallow /forum/public/js/
Disallow /forum/public/style_captcha/
Disallow /forum/public/style_css/
Disallow /forum/index.php?app=core&module=task*
Disallow /forum/index.php?app=core&module=usercp&tab=forums&area=forumsubs*
Disallow /forum/index.php?app=core&module=usercp&tab=forums&area=watch&watch=topic*
Disallow /forum/index.php?app=forums&module=extras&section=forward*
Disallow /forum/index.php?app=forums&module=extras&section=rating*
Disallow /forum/index.php?app=members&module=messaging*
Disallow /forum/index.php?app=members&module=online*
Disallow /index.php?app=members&module=online*
Disallow /forum/index.php?app=members&module=chat*
Disallow /forum/index.php?app=members&module=search*
Disallow /forum/index.php?app=members&module=search&do=active*
Disallow /forum/index.php?&unlockUserAgent=1
Disallow /*app%3Dcore%26module%3Dusercp
Disallow /*app%3Dmembers%26module%3Dmessaging
Disallow /*%26p%3D
Disallow /*%26pid%3D
Disallow /*%26hl%3D
Disallow /*%26start%3D
Disallow /*view__getnewpost$
Disallow /*view__getlastpost$
Disallow /*view__old$
Disallow /*view__new$
Disallow /*view__getfirst$
Disallow /*view__getprevious$
Disallow /*view__getnext$
Disallow /*view__getlast$
Disallow /*%26view%3Dgetnewpost$
Disallow /*%26view%3Dgetlastpost$
Disallow /*%26view%3Dold$
Disallow /*%26view%3Dnew$
Disallow /*%26view%3Dgetfirst$
Disallow /*%26view%3Dgetprevious$
Disallow /*%26view%3Dgetnext$
Disallow /*%26view%3Dgetlast$
Disallow /index.php?app=core&module=attach*
Disallow /app%3Dcalendar/*
Disallow /calendar/*
Disallow /index.php?app=forums&module=extras&section=stats
Disallow /index.php?app=core&module=help*
Disallow /index.php?app=core&module=search&do=new_posts*
Disallow /index.php?app=gallery*
Disallow /index.php?app=core&module=global&section=register*
Disallow /index.php?app=members&module=profile&section=friends*
Disallow /index.php?app=forums&module=extras&section=stats*
Disallow /index.php?app=forums*
Disallow /shoutbox/*
Disallow /index.php?app=shoutbox*
Disallow /index.php?showtopic=*
Disallow /index.php?app=core&module=global&section=login*
Disallow /index.php?app=forums&module=forums&section=findpost*
Disallow /user/*/page__s__*
Disallow /forum/*/?s=*
Disallow /index.php?app=members&section=view&module=list*
Disallow /index.php?showall=0&sort_key=members_display_name*
Disallow /index.php?app=forums&module=extras&section=stats&do=leaders*
Disallow /index.php?app=forums&module=post&section=post&do=reply_post*
Disallow /index.php?app=core&module=search*
Disallow /index/page__s__*
Disallow /index.php?app=forums&module=extras&section=stats&do=who*
Disallow /index.php?act=calendar*
Disallow /index.php?app=forums&module=forums&section=findpost*
Disallow /public/min/index.php*
Disallow /topic/*/page__view*
Disallow /topic/*/page__s__*
Disallow /forum/*/page__s__*
Disallow /public/style_emoticons/default
Disallow /index.php?s=*&app=core&module=task*
Disallow /index.php?app=calendar&module=calendar*
Disallow /forum/*/page__prune*
Disallow /*__s__*
Disallow /index.php?s=*
Disallow /index.php?showuser=*
Disallow /topic/*/page__pid__*
Disallow /topic/*/page__gopid__*
Disallow /topic/*/page__pid__*?s=*
Disallow /topic/*/page__st__*?s=*
Disallow /topic/*/page__p__*
Disallow /index.php/?*
Disallow /user/*/?s=*
Disallow /topic/*/?s=*
Disallow /forum/*/?s=*
Disallow /index.php?showforum=*
Disallow /index.php?PHPSESSID=*

mj12bot

Rule Path
Disallow /

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

sistrix

Rule Path
Disallow /

sistrix crawler

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

jobs.de-robot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

unisterbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

seodiver

Rule Path
Disallow /

spbot

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

backlinkcrawler

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

obot

Rule Path
Disallow /

fr-crawler

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

cloudservermarketspider

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

careerbot

Rule Path
Disallow /

lipperhey-kaus-australis

Rule Path
Disallow /

seoscanners.net

Rule Path
Disallow /

metajobbot

Rule Path
Disallow /

spiderbot

Rule Path
Disallow /

linkstats

Rule Path
Disallow /

jobboersebot

Rule Path
Disallow /

iccrawler

Rule Path
Disallow /

plista

Rule Path
Disallow /

domain re-animator bot

Rule Path
Disallow /

lipperhey-kaus-australis

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

coccoc

Rule Path
Disallow /

um-ic

Rule Path
Disallow /

mindupbot

Rule Path
Disallow /

sg-orbiter

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

kraken

Rule Path
Disallow /

plukkie

Rule Path
Disallow /

safednsbot

Rule Path
Disallow /

haosouspider

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

openhosebot

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

thumbsniper

Rule Path
Disallow /

r6_commentreader

Rule Path
Disallow /

implisensebot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

adscanner

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

python/3.5 aiohttp

Rule Path
Disallow /

toweya.com

Rule Path
Disallow /

netestate

Rule Path
Disallow /

bubing

Rule Path
Disallow /

linguee

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

sentibot

Rule Path
Disallow /

sentibot

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

domaincrawler

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

indeedbot

Rule Path
Disallow /

garlikcrawler

Rule Path
Disallow /

gosign-security-crawler

Rule Path
Disallow /

siteliner

Rule Path
Disallow /

sabsimbot

Rule Path
Disallow /

ltx71

Rule Path
Disallow /
Back to top

Comments

  • www.robotstxt.org/
  • www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449
  • Slow down bots
  • Disallow: Sistrix
  • Disallow: Sistrix
  • Disallow: Sistrix
  • Disallow: SEOkicks-Robot
  • Disallow: jobs.de-Robot
  • Backlink Analysis
  • Bot der Leipziger Unister Holding GmbH
  • http://www.opensiteexplorer.org/dotbot
  • http://www.searchmetrics.com
  • http://www.majestic12.co.uk/projects/dsearch/mj12bot.php
  • http://www.domaintools.com/webmasters/surveybot.php
  • http://www.seodiver.com/bot
  • http://openlinkprofiler.org/bot
  • http://www.wotbox.com/bot/
  • http://www.meanpath.com/meanpathbot.html
  • http://www.backlinktest.com/crawler.html
  • http://www.brandwatch.com/magpie-crawler/
  • http://filterdb.iss.net/crawler/
  • http://webmeup-crawler.com
  • https://megaindex.com/crawler
  • http://www.cloudservermarket.com
  • http://www.trendiction.de/de/publisher/bot
  • http://www.exalead.com
  • http://www.career-x.de/bot.html
  • https://www.lipperhey.com/en/about/
  • https://www.lipperhey.com/en/about/
  • https://turnitin.com/robot/crawlerinfo.html
  • http://help.coccoc.com/
  • ubermetrics-technologies.com
  • datenbutler.de
  • http://searchgears.de/uber-uns/crawling-faq.html
  • http://commoncrawl.org/faq/
  • https://www.qwant.com/
  • http://linkfluence.net/
  • http://www.botje.com/plukkie.htm
  • https://www.safedns.com/searchbot
  • http://www.haosou.com/help/help_3_2.html
  • http://www.haosou.com/help/help_3_2.html
  • http://www.moz.com/dp/rogerbot
  • http://www.openhose.org/bot.html
  • http://www.screamingfrog.co.uk/seo-spider/
  • http://thumbsniper.com
  • http://www.radian6.com/crawler
  • http://cliqz.com/company/cliqzbot
  • https://www.aihitdata.com/about
  • http://www.trendiction.com/en/publisher/bot
  • http://seocompany.store
  • https://github.com/yasserg/crawler4j/
  • http://warebay.com/bot.html
  • http://www.website-datenbank.de/
  • http://law.di.unimi.it/BUbiNG.html
  • http://www.linguee.com/bot; bot@linguee.com
  • https://www.semrush.com/bot/
  • www.sentibot.eu
  • http://velen.io
  • https://moz.com/help/guides/moz-procedures/what-is-rogerbot
  • http://www.garlik.com
  • https://www.gosign.de/typo3-extension/typo3-sicherheitsmonitor/
  • http://www.siteliner.com/bot
  • https://sabsim.com
  • http://ltx71.com/
  • END
Back to top

Warnings

  • 2 invalid lines.
  • `host` is not a known field.
Back to top