hatarszel.hu
robots.txt

Robots Exclusion Standard data for hatarszel.hu

Archived Snapshots

Resource Scan

Scan Details

Site Domain	hatarszel.hu
Base Domain	hatarszel.hu
Scan Status	Failed
Failure Stage	Fetching resource.
Failure Reason	Couldn't connect to server.
Last Scan	2024-08-08T08:59:50+00:00
Next Scan	2024-11-06T08:59:50+00:00

Last Successful Scan

Scanned	2023-10-14T06:11:02+00:00
URL	https://hatarszel.hu/robots.txt
Redirect	https://www.hatarszel.hu/robots.txt
Redirect Domain	www.hatarszel.hu
Redirect Base	hatarszel.hu
Domain IPs	185.51.188.57
Redirect IPs	185.51.188.57
Response IP	185.51.188.57
Found	Yes
Hash	f12b8b5739b54f42267cfebf9b00c6d711b9f7b7d7ab9c0706126254de9cf773
SimHash	b3f6b1438766

Groups

*

Rule	Path
Allow	/core/*.css$
Allow	/core/*.css?
Allow	/core/*.js$
Allow	/core/*.js?
Allow	/core/*.gif
Allow	/core/*.jpg
Allow	/core/*.jpeg
Allow	/core/*.png
Allow	/core/*.svg
Allow	/profiles/*.css$
Allow	/profiles/*.css?
Allow	/profiles/*.js$
Allow	/profiles/*.js?
Allow	/profiles/*.gif
Allow	/profiles/*.jpg
Allow	/profiles/*.jpeg
Allow	/profiles/*.png
Allow	/profiles/*.svg
Disallow	/core/
Disallow	/profiles/
Disallow	/README.txt
Disallow	/web.config
Disallow	/admin/
Disallow	/comment/reply/
Disallow	/filter/tips
Disallow	/node/add/
Disallow	/search/
Disallow	/user/register
Disallow	/user/password
Disallow	/user/login
Disallow	/user/logout
Disallow	/media/oembed
Disallow	/*/media/oembed
Disallow	/index.php/admin/
Disallow	/index.php/comment/reply/
Disallow	/index.php/filter/tips
Disallow	/index.php/node/add/
Disallow	/index.php/search/
Disallow	/index.php/user/password
Disallow	/index.php/user/register
Disallow	/index.php/user/login
Disallow	/index.php/user/logout
Disallow	/index.php/media/oembed
Disallow	/index.php/*/media/oembed

Rule

Path

Allow

/core/*.css$

Allow

/core/*.css?

Allow

/core/*.js$

Allow

/core/*.js?

Allow

/core/*.gif

Allow

/core/*.jpg

Allow

/core/*.jpeg

Allow

/core/*.png

Allow

/core/*.svg

Allow

/profiles/*.css$

Allow

/profiles/*.css?

Allow

/profiles/*.js$

Allow

/profiles/*.js?

Allow

/profiles/*.gif

Allow

/profiles/*.jpg

Allow

/profiles/*.jpeg

Allow

/profiles/*.png

Allow

/profiles/*.svg

Disallow

/core/

Disallow

/profiles/

Disallow

/README.txt

Disallow

/web.config

Disallow

/admin/

Disallow

/comment/reply/

Disallow

/filter/tips

Disallow

/node/add/

Disallow

/search/

Disallow

/user/register

Disallow

/user/password

Disallow

/user/login

Disallow

/user/logout

Disallow

/media/oembed

Disallow

/*/media/oembed

Disallow

/index.php/admin/

Disallow

/index.php/comment/reply/

Disallow

/index.php/filter/tips

Disallow

/index.php/node/add/

Disallow

/index.php/search/

Disallow

/index.php/user/password

Disallow

/index.php/user/register

Disallow

/index.php/user/login

Disallow

/index.php/user/logout

Disallow

/index.php/media/oembed

Disallow

/index.php/*/media/oembed

trendictionbot

Rule	Path
Disallow	/

Rule

Path

Disallow

sistrix

Rule	Path
Disallow	/

Rule

Path

Disallow

sistrix crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

sistrix

Rule	Path
Disallow	/

Rule

Path

Disallow

seokicks-robot

Rule	Path
Disallow	/

Rule

Path

Disallow

jobs.de-robot

Rule	Path
Disallow	/

Rule

Path

Disallow

ahrefsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

unisterbot

Rule	Path
Disallow	/

Rule

Path

Disallow

dotbot

Rule	Path
Disallow	/

Rule

Path

Disallow

dotbot

Rule	Path
Disallow	/

Rule

Path

Disallow

searchmetricsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

mj12bot

Rule	Path
Disallow	/

Rule

Path

Disallow

surveybot

Rule	Path
Disallow	/

Rule

Path

Disallow

seodiver

Rule	Path
Disallow	/

Rule

Path

Disallow

spbot

Rule	Path
Disallow	/

Rule

Path

Disallow

wotbox

Rule	Path
Disallow	/

Rule

Path

Disallow

meanpathbot

Rule	Path
Disallow	/

Rule

Path

Disallow

backlinkcrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

magpie-crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

obot

Rule	Path
Disallow	/

Rule

Path

Disallow

fr-crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

blexbot

Rule	Path
Disallow	/

Rule

Path

Disallow

megaindex.ru

Rule	Path
Disallow	/

Rule

Path

Disallow

megaindex.com

Rule	Path
Disallow	/

Rule

Path

Disallow

cloudservermarketspider

Rule	Path
Disallow	/

Rule

Path

Disallow

trendictionbot

Rule	Path
Disallow	/

Rule

Path

Disallow

exabot

Rule	Path
Disallow	/

Rule

Path

Disallow

careerbot

Rule	Path
Disallow	/

Rule

Path

Disallow

lipperhey-kaus-australis

Rule	Path
Disallow	/

Rule

Path

Disallow

seoscanners.net

Rule	Path
Disallow	/

Rule

Path

Disallow

metajobbot

Rule	Path
Disallow	/

Rule

Path

Disallow

spiderbot

Rule	Path
Disallow	/

Rule

Path

Disallow

linkstats

Rule	Path
Disallow	/

Rule

Path

Disallow

jobboersebot

Rule	Path
Disallow	/

Rule

Path

Disallow

iccrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

plista

Rule	Path
Disallow	/

Rule

Path

Disallow

domain re-animator bot

Rule	Path
Disallow	/

Rule

Path

Disallow

lipperhey-kaus-australis

Rule	Path
Disallow	/

Rule

Path

Disallow

turnitinbot

Rule	Path
Disallow	/

Rule

Path

Disallow

coccoc

Rule	Path
Disallow	/

Rule

Path

Disallow

um-ic

Rule	Path
Disallow	/

Rule

Path

Disallow

mindupbot

Rule	Path
Disallow	/

Rule

Path

Disallow

sg-orbiter

Rule	Path
Disallow	/

Rule

Path

Disallow

ccbot

Rule	Path
Disallow	/

Rule

Path

Disallow

qwantify

Rule	Path
Disallow	/

Rule

Path

Disallow

kraken

Rule	Path
Disallow	/

Rule

Path

Disallow

plukkie

Rule	Path
Disallow	/

Rule

Path

Disallow

safednsbot

Rule

Path

Disallow

haosouspider

Rule

Path

Disallow

rogerbot

Rule

Path

Disallow

openhosebot

Rule

Path

Disallow

screaming frog seo spider

Rule

Path

Disallow

thumbsniper

Rule

Path

Disallow

r6_commentreader

Rule

Path

Disallow

implisensebot

Rule

Path

Disallow

cliqzbot

Rule

Path

Disallow

aihitbot

Rule

Path

Disallow

adscanner

Rule

Path

Disallow

crawler4j

Rule

Path

Disallow

wbsearchbot

Rule

Path

Disallow

python/3.5 aiohttp

Rule

Path

Disallow

toweya.com

Rule

Path

Disallow

netestate

Rule

Path

Disallow

bubing

Rule

Path

Disallow

linguee

Rule

Path

Disallow

semrushbot

Rule

Path

Disallow

semrushbot-sa

Rule

Path

Disallow

sentibot

Rule

Path

Disallow

sentibot

Rule

Path

Disallow

velenpublicwebcrawler

Rule

Path

Disallow

domaincrawler

Rule

Path

Disallow

rogerbot

Rule

Path

Disallow

indeedbot

Rule

Path

Disallow

garlikcrawler

Rule

Path

Disallow

gosign-security-crawler

Rule

Path

Disallow

siteliner

Rule

Path

Disallow

sabsimbot

Rule

Path

Disallow

ltx71

Rule

Path

Disallow

rogerbot

Rule

Path

Disallow

ms search 4.0 robot

Rule

Path

Disallow

mozilla/4.0 (compatible; msie 4.01; windows nt; ms search 4.0 robot) microsoft

Rule

Path

Disallow

proximic

Rule

Path

Disallow

admantx

Rule

Path

Disallow

admantx platform semantic analyzer - admantx inc. - www.admantx.com - support@admantx.com

Rule

Path

Disallow

useragent:admantx platform semantic analyzer us - turn - admantx inc. - www.admantx.com - support@admantx.com

Rule

Path

Disallow

maxpoint bot

Rule

Path

Disallow

maxpointcrawler

Rule

Path

Disallow

clickagy intelligence bot

Rule

Path

Disallow

clickagy intelligence bot v2

Rule

Path

Disallow

yandex

Rule

Path

Disallow

omgili/0.5 +http://omgili.com

Rule

Path

Disallow

omgili

Rule

Path

Disallow

ia_archiver

Rule

Path

Disallow

libwww-perl

Rule

Path

Disallow

libwwwperl

Rule

Path

Disallow

attach

Rule

Path

Disallow

aspseek

Rule

Path

Disallow

appie

Rule

Path

Disallow

abachobot

Rule

Path

Disallow

autoemailspider

Rule

Path

Disallow

anarchie

Rule

Path

Disallow

antibot

Rule

Path

Disallow

asterias

Rule

Path

Disallow

b2w

Rule

Path

Disallow

backweb

Rule

Path

Disallow

backdoorbot

Rule

Path

Disallow

bandit

Rule

Path

Disallow

batchftp

Rule

Path

Disallow

black\ hole

Rule

Path

Disallow

baidu

Rule

Path

Disallow

blowfish

Rule

Path

Disallow

builtbottough

Rule

Path

Disallow

bot\ mailto

Rule

Path

Disallow

botalot

Rule

Path

Disallow

buddy

Rule

Path

Disallow

bullseye

Rule

Path

Disallow

bumblebee

Rule

Path

Disallow

bunnyslippers

Rule

Path

Disallow

clariabot

Rule

Path

Disallow

curl

Rule

Path

Disallow

clshttp

Rule

Path

Disallow

chinaclaw

Rule

Path

Disallow

cheesebot

Rule

Path

Disallow

cherrypicker

Rule

Path

Disallow

crescent

Rule

Path

Disallow

cherrypickerse

Rule

Path

Disallow

cherrypickerelite

Rule

Path

Disallow

collector

Rule

Path

Disallow

coast\ webmaster

Rule

Path

Disallow

cosmos

Rule

Path

Disallow

copyrightcheck

Rule

Path

Disallow

coldfusion

Rule

Path

Disallow

copier

Rule

Path

Disallow

crescent

Rule

Path

Disallow

da

Rule

Path

Disallow

dts\ agent

Rule

Path

Disallow

disco\ pump

Rule

Path

Disallow

dittospyder

Rule

Path

Disallow

diamond

Rule

Path

Disallow

download\ demon

Rule

Path

Disallow

download\ wonder

Rule

Path

Disallow

downloader

Rule

Path

Disallow

dloader

Rule

Path

Disallow

drip

Rule

Path

Disallow

ecatch

Rule

Path

Disallow

eirgrabber

Rule

Path

Disallow

express\ webpictures

Rule

Path

Disallow

extreme\ picture\ finder

Rule

Path

Disallow

emailcollector

Rule

Path

Disallow

emailsiphon

Rule

Path

Disallow

emailwolf

Rule

Path

Disallow

easydl

Rule

Path

Disallow

eirgrabber

Rule

Path

Disallow

erocrawler

Rule

Path

Disallow

extractorpro

Rule

Path

Disallow

eyenetie

Rule

Path

Disallow

fast\ webcrawler

Rule

Path

Disallow

filehound

Rule

Path

Disallow

fetch\ api\ request

Rule

Path

Disallow

flashget

Rule

Path

Disallow

flickbot

Rule

Path

Disallow

frontpage

Rule

Path

Disallow

freefind.com

Rule

Path

Disallow

getright

Rule

Path

Disallow

getsmart

Rule

Path

Disallow

generic

Rule

Path

Disallow

go!zilla

Rule

Path

Disallow

go-ahead-got-it

Rule

Path

Disallow

gotit

Rule

Path

Disallow

grabber

Rule

Path

Disallow

grabnet

Rule

Path

Disallow

grafula

Rule

Path

Disallow

gulliver

Rule

Path

Disallow

harvest

Rule

Path

Disallow

hmview

Rule

Path

Disallow

heretrix

Rule

Path

Disallow

hitboxdoctor

Rule

Path

Disallow

httpapp

Rule

Path

Disallow

httrack

Rule

Path

Disallow

httptrack

Rule

Path

Disallow

httpviewer

Rule

Path

Disallow

httplib

Rule

Path

Disallow

httpfetcher

Rule

Path

Disallow

httpscraper

Rule

Path

Disallow

hloader

Rule

Path

Disallow

humanlinks

Rule

Path

Disallow

ia_archiver

Rule

Path

Disallow

interget

Rule

Path

Disallow

internet\ ninja

Rule

Path

Disallow

infonavirobot

Rule

Path

Disallow

internetseer.com

Rule

Path

Disallow

iria

Rule

Path

Disallow

irlbot

Rule

Path

Disallow

jetcar

Rule

Path

Disallow

joc

Rule

Path

Disallow

joc\ web\ spider

Rule

Path

Disallow

jobo

Rule

Path

Disallow

java

Rule

Path

Disallow

justview

Rule

Path

Disallow

jonzilla

Rule

Path

Disallow

jennybot

Rule

Path

Disallow

kenjin\ spider

Rule

Path

Disallow

keyword\ density

Rule

Path

Disallow

larbin

Rule

Path

Disallow

leechftp

Rule

Path

Disallow

lachesis

Rule

Path

Disallow

lexibot

Rule

Path

Disallow

libweb

Rule

Path

Disallow

libby_

Rule

Path

Disallow

linkscan

Rule

Path

Disallow

linkwalker

Rule

Path

Disallow

linkextractorpro

Rule

Path

Disallow

lftp

Rule

Path

Disallow

likse

Rule

Path

Disallow

link

Rule

Path

Disallow

lwp-trivial

Rule

Path

Disallow

lwp\ request

Rule

Path

Disallow

magnet

Rule

Path

Disallow

mag-net

Rule

Path

Disallow

mass\ downloader

Rule

Path

Disallow

miixpc

Rule

Path

Disallow

microsoft\ url\ control

Rule

Path

Disallow

msfrontpage

Rule

Path

Disallow

msiecrawler

Rule

Path

Disallow

microsofturl

Rule

Path

Disallow

missigua

Rule

Path

Disallow

mewsoft\ search\ engine

Rule

Path

Disallow

moget

Rule

Path

Disallow

mata\ hari

Rule

Path

Disallow

memo

Rule

Path

Disallow

metacarta

Rule

Path

Disallow

mercator

Rule

Path

Disallow

midown\ tool

Rule

Path

Disallow

mfc_tear_sample

Rule

Path

Disallow

mirror

Rule

Path

Disallow

miixpc

Rule

Path

Disallow

mister\ pix

Rule

Path

Disallow

nationaldirectory\ webspider

Rule

Path

Disallow

nicerspro

Rule

Path

Disallow

nikto

Rule

Path

Disallow

navroad

Rule

Path

Disallow

nearsite

Rule

Path

Disallow

netants

Rule

Path

Disallow

netspider

Rule

Path

Disallow

nicerspro

Rule

Path

Disallow

netresearchserver

Rule

Path

Disallow

netmechanic

Rule

Path

Disallow

net\ vampire

Rule

Path

Disallow

net\ probe

Rule

Path

Disallow

netzip

Rule

Path

Disallow

nexuscache

Rule

Path

Disallow

ninja

Rule

Path

Disallow

npbot

Rule

Path

Disallow

our\ agent

Rule

Path

Disallow

onestop

Rule

Path

Disallow

obot

Rule

Path

Disallow

octopus

Rule

Path

Disallow

offline\ explorer

Rule

Path

Disallow

openfind

Rule

Path

Disallow

openfind\ data\ gatherer

Rule

Path

Disallow

orangebot

Rule

Path

Disallow

pagegrabber
papa\ foto

Rule

Path

Disallow

php\ version

Rule

Path

Disallow

php

Rule

Path

Disallow

phpot

Rule

Path

Disallow

perl

Rule

Path

Disallow

pcbrowser

Rule

Path

Disallow

pavuk

Rule

Path

Disallow

pockey

Rule

Path

Disallow

ping

Rule

Path

Disallow

pingalink\ monitoring\ services

Rule

Path

Disallow

prowebwalker

Rule

Path

Disallow

propowerbot

Rule

Path

Disallow

pump

Rule

Path

Disallow

pompos

Rule

Path

Disallow

psbot

Rule

Path

Disallow

python\ urllib

Rule

Path

Disallow

python-urllib

Rule

Path

Disallow

queryn

Rule

Path

Disallow

realdownload

Rule

Path

Disallow

reaper

Rule

Path

Disallow

recorder

Rule

Path

Disallow

repomonkey

Rule

Path

Disallow

psycheclone

Rule

Path

Disallow

rma

Rule

Path

Disallow

rico

Rule

Path

Disallow

robozilla

Rule

Path

Disallow

reget

Rule

Path

Disallow

siphon

Rule

Path

Disallow

sitesnagger

Rule

Path

Disallow

sitecheck.internetseer.com

Rule

Path

Disallow

smartdownload

Rule

Path

Disallow

snake

Rule

Path

Disallow

spanner

Rule

Path

Disallow

stealer

Rule

Path

Disallow

spacebison
spankbot

Rule

Path

Disallow

spinne

Rule

Path

Disallow

stripper

Rule

Path

Disallow

slysearch

Rule

Path

Disallow

sucker

Rule

Path

Disallow

snoopy

Rule

Path

Disallow

scoutabout

Rule

Path

Disallow

scooter

Rule

Path

Disallow

superbot

Rule

Path

Disallow

superhttp

Rule

Path

Disallow

snapbot

Rule

Path

Disallow

surfbot

Rule

Path

Disallow

suzuran

Rule

Path

Disallow

szukacz

Rule

Path

Disallow

sqworm

Rule

Path

Disallow

takeout

Rule

Path

Disallow

teleport\ pro

Rule

Path

Disallow

telesoft

Rule

Path

Disallow

turnitinbot

Rule

Path

Disallow

turingos

Rule

Path

Disallow

tocrawl

Rule

Path

Disallow

tighttwatbot

Rule

Path

Disallow

true_robot

Rule

Path

Disallow

the\ intraformant

Rule

Path

Disallow

thenomad

Rule

Path

Disallow

titan

Rule

Path

Disallow

urldispatcher

Rule

Path

Disallow

urly\ warning

Rule

Path

Disallow

vayala

Rule

Path

Disallow

vagabondo

Rule

Path

Disallow

vintage

Rule

Path

Disallow

vacuum

Rule

Path

Disallow

vci

Rule

Path

Disallow

voideye

Rule

Path

Disallow

w3c_validator

Rule

Path

Disallow

webdownloader
web\ downloader

Rule

Path

Disallow

webhook

Rule

Path

Disallow

webmole

Rule

Path

Disallow

webminer

Rule

Path

Disallow

webmirror

Rule

Path

Disallow

websucker

Rule

Path

Disallow

websites

Rule

Path

Disallow

web\ image\ collector

Rule

Path

Disallow

web\ sucker

Rule

Path

Disallow

webauto

Rule

Path

Disallow

webcopier

Rule

Path

Disallow

webfetch

Rule

Path

Disallow

webreaper

Rule

Path

Disallow

websauger

Rule

Path

Disallow

website

Rule

Path

Disallow

webster
webstripper

Rule

Path

Disallow

webcopier

Rule

Path

Disallow

webviewer

Rule

Path

Disallow

webwhacker

Rule

Path

Disallow

webenhancer

Rule

Path

Disallow

wells

Rule

Path

Disallow

webzip

Rule

Path

Disallow

wget

Rule

Path

Disallow

whacker

Rule

Path

Disallow

widow

Rule

Path

Disallow

xaldon

Rule

Path

Disallow

wildsoft\ surfer

Rule

Path

Disallow

winhttprequest

Rule

Path

Disallow

winhttp

Rule

Path

Disallow

webster\ pro

Rule

Path

Disallow

web\ image\ collector

Rule

Path

Disallow

webzip

Rule

Path

Disallow

webauto

Rule

Path

Disallow

website\ quester

Rule

Path

Disallow

wwwoffle

Rule

Path

Disallow

www-collector-e

Rule

Path

Disallow

xaldon\ webspider

Rule

Path

Disallow

xenu

Rule

Path

Disallow

xara

Rule

Path

Disallow

y!tunnelpro

Rule

Path

Disallow

yahooysmcm

Rule

Path

Disallow

zade

Rule

Path

Disallow

zbot

Rule

Path

Disallow

zeus

Rule

Path

Disallow

rogerbot

Rule

Path

Disallow

exabot

Rule

Path

Disallow

mj12bot

Rule

Path

Disallow

dotbot

Rule

Path

Disallow

gigabot

Rule

Path

Disallow

ahrefsbot

Rule

Path

Disallow

ahrefs

Rule

Path

Disallow

blackwidow

Rule

Path

Disallow

bot\ mailto:craftbot@yahoo.com

Rule

Path

Disallow

chinaclaw

Rule

Path

Disallow

custo

Rule

Path

Disallow

disco

Rule

Path

Disallow

download\ demon

Rule

Path

Disallow

ecatch

Rule

Path

Disallow

eirgrabber

Rule

Path

Disallow

emailsiphon

Rule

Path

Disallow

emailwolf

Rule

Path

Disallow

express\ webpictures

Rule

Path

Disallow

extractorpro

Rule

Path

Disallow

eyenetie

Rule

Path

Disallow

flashget

Rule

Path

Disallow

getright

Rule

Path

Disallow

getweb!

Rule

Path

Disallow

go!zilla

Rule

Path

Disallow

go-ahead-got-it

Rule

Path

Disallow

grabnet

Rule

Path

Disallow

grafula

Rule

Path

Disallow

hmview

Rule

Path

Disallow

httrack

Rule

Path

Disallow

image\ stripper

Rule

Path

Disallow

image\ sucker

Rule

Path

Disallow

indy\ library

Rule

Path

Disallow

interget

Rule

Path

Disallow

internet\ ninja

Rule

Path

Disallow

jetcar

Rule

Path

Disallow

joc\ web\ spider

Rule

Path

Disallow

larbin

Rule

Path

Disallow

leechftp

Rule

Path

Disallow

mass\ downloader

Rule

Path

Disallow

midown\ tool

Rule

Path

Disallow

mister\ pix

Rule

Path

Disallow

navroad

Rule

Path

Disallow

nearsite

Rule

Path

Disallow

netants

Rule

Path

Disallow

netspider

Rule

Path

Disallow

net\ vampire

Rule

Path

Disallow

netzip

Rule

Path

Disallow

octopusdisallow: /

Rule

Path

Disallow

offline\ explorer

Rule

Path

Disallow

offline\ navigator

Rule

Path

Disallow

pagegrabber

Rule

Path

Disallow

papa\ foto

Rule

Path

Disallow

pavuk

Rule

Path

Disallow

pcbrowser

Rule

Path

Disallow

realdownload

Rule

Path

Disallow

reget

Rule

Path

Disallow

sitesnagger

Rule

Path

Disallow

smartdownload

Rule

Path

Disallow

superbot

Rule

Path

Disallow

superhttp

Rule

Path

Disallow

surfbot

Rule

Path

Disallow

takeout

Rule

Path

Disallow

teleport\ pro

Rule

Path

Disallow

voideye

Rule

Path

Disallow

web\ image\ collector

Rule

Path

Disallow

web\ sucker

Rule

Path

Disallow

webauto

Rule

Path

Disallow

webcopier

Rule

Path

Disallow

webfetch

Rule

Path

Disallow

webgo\ is

Rule

Path

Disallow

webleacher

Rule

Path

Disallow

webreaper

Rule

Path

Disallow

websauger

Rule

Path

Disallow

website\ extractor

Rule

Path

Disallow

website\ quester

Rule

Path

Disallow

webstripper

Rule

Path

Disallow

webwhacker

Rule

Path

Disallow

webzip

Rule

Path

Disallow

wget

Rule

Path

Disallow

widow

Rule

Path

Disallow

wwwoffle

Rule

Path

Disallow

xaldon\ webspider

Rule

Path

Disallow

twitterbot

Rule

Path

Disallow

Comments

robots.txt
This file is to prevent the crawling and indexing of certain parts
of your site by web crawlers and spiders run by sites like Yahoo!
and Google. By telling these "robots" where not to go on your site,
you save bandwidth and server resources.
This file will be ignored unless it is at the root of your host:
Used: http://example.com/robots.txt
Ignored: http://example.com/site/robots.txt
For more information about the robots.txt standard, see:
http://www.robotstxt.org/robotstxt.html
CSS, JS, Images
Directories
Files
Paths (clean URLs)
Paths (no clean URLs)

Warnings

2 invalid lines.

hatarszel.hurobots.txt

Resource Scan

Scan Details

Last Successful Scan

Groups

*

trendictionbot

sistrix

sistrix crawler

sistrix

seokicks-robot

jobs.de-robot

ahrefsbot

unisterbot

dotbot

dotbot

searchmetricsbot

mj12bot

surveybot

seodiver

spbot

wotbox

meanpathbot

backlinkcrawler

magpie-crawler

obot

fr-crawler

blexbot

megaindex.ru

megaindex.com

cloudservermarketspider

trendictionbot

exabot

careerbot

lipperhey-kaus-australis

seoscanners.net

metajobbot

spiderbot

linkstats

jobboersebot

iccrawler

plista

domain re-animator bot

lipperhey-kaus-australis

turnitinbot

coccoc

um-ic

mindupbot

sg-orbiter

ccbot

qwantify

kraken

plukkie

safednsbot

haosouspider

rogerbot

openhosebot

screaming frog seo spider

thumbsniper

r6_commentreader

implisensebot

cliqzbot

aihitbot

adscanner

crawler4j

wbsearchbot

python/3.5 aiohttp

toweya.com

netestate

bubing

linguee

semrushbot

semrushbot-sa

sentibot

sentibot

velenpublicwebcrawler

domaincrawler

rogerbot

indeedbot

garlikcrawler

hatarszel.hu
robots.txt