Log In Sign Up

niidg.ru
robots.txt

Robots Exclusion Standard data for niidg.ru

Resource Scan

Scan Details

Site Domain niidg.ru
Base Domain niidg.ru
Scan Status Ok
Last Scan2024-10-19T21:22:34+00:00
Next Scan 2024-11-18T21:22:34+00:00

Last Scan

Scanned2024-10-19T21:22:34+00:00
URL http://niidg.ru/robots.txt
Domain IPs 141.8.194.164
Response IP 141.8.194.164
Found Yes
Hash 89afb9b78f1c729d88db749d8db412648ffa65f4b8f94d7182e7e522e8123bf3
SimHash e343ff5707af

Groups

screaming frog seo spider

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

mail.ru_bot/fast/2.0

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

webalta

Rule Path
Disallow /

msnbot

Rule Path
Disallow /

slurp

Rule Path
Disallow /

teoma

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

w3c-checklink

Rule Path
Disallow /

scrubby

Rule Path
Disallow /

robozilla

Rule Path
Disallow /

nutch

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

yahoo-mmcrawler

Rule Path
Disallow /

psbot

Rule Path
Disallow /

asterias

Rule Path
Disallow /

yahoo-blogs/v3.9

Rule Path
Disallow /

accelerator

Rule Path
Disallow /

accoona

Rule Path
Disallow /

alexibot

Rule Path
Disallow /

almaden.ibm.com

Rule Path
Disallow /

ants

Rule Path
Disallow /

aport

Rule Path
Disallow /

applewebkit

Rule Path
Disallow /

ask\ jeeves

Rule Path
Disallow /

asterias

Rule Path
Disallow /

attach

Rule Path
Disallow /

babaloospider

Rule Path
Disallow /

backdoorbot

Rule Path
Disallow /

backweb

Rule Path
Disallow /

badex-bot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

bandit

Rule Path
Disallow /

becomebot

Rule Path
Disallow /

bilgibot

Rule Path
Disallow /

bilgifeedbot/experimental

Rule Path
Disallow /

bingbot

Rule Path
Disallow /

birubot

Rule Path
Disallow /

black.hole

Rule Path
Disallow /

blackwidow

Rule Path
Disallow /

blowfish

Rule Path
Disallow /

botalot

Rule Path
Disallow /

brontok.a\ browser

Rule Path
Disallow /

buddy

Rule Path
Disallow /

builtbottough

Rule Path
Disallow /

bullseye

Rule Path
Disallow /

bunnyslippers

Rule Path
Disallow /

butterfly

Rule Path
Disallow /

camontspider

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

cegbfeieh

Rule Path
Disallow /

cfnetwork

Rule Path
Disallow /

cheesebot

Rule Path
Disallow /

cherrypicker

Rule Path
Disallow /

chinaclaw

Rule Path
Disallow /

collector

Rule Path
Disallow /

converacrawler

Rule Path
Disallow /

copier

Rule Path
Disallow /

copyrightcheck

Rule Path
Disallow /

cosmos

Rule Path
Disallow /

crescent

Rule Path
Disallow /

custo

Rule Path
Disallow /

dead\ link\ checker

Rule Path
Disallow /

depspid

Rule Path
Disallow /

diibot

Rule Path
Disallow /

disco

Rule Path
Disallow /

dittospyder

Rule Path
Disallow /

dolphin

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

dotnetdotcom

Rule Path
Disallow /

download\ demon

Rule Path
Disallow /

download\ master

Rule Path
Disallow /

downloader

Rule Path
Disallow /

drip

Rule Path
Disallow /

e-societyrobot

Rule Path
Disallow /

ecatch

Rule Path
Disallow /

eirgrabber

Rule Path
Disallow /

emailcollector

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

emailwolf

Rule Path
Disallow /

erocrawler

Rule Path
Disallow /

exabot

Rule Path
Disallow /

express\ webpictures

Rule Path
Disallow /

extractorpro

Rule Path
Disallow /

eyenetie

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

fairshare

Rule Path
Disallow /

fast\ metaweb\ crawler

Rule Path
Disallow /

feedfetcher-google

Rule Path
Disallow /

filehound

Rule Path
Disallow /

flashget

Rule Path
Disallow /

flexum\ spider

Rule Path
Disallow /

foobot

Rule Path
Disallow /

frontpage

Rule Path
Disallow /

getright

Rule Path
Disallow /

gets

Rule Path
Disallow /

getweb!

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

go-ahead-got-it

Rule Path
Disallow /

go!zilla

Rule Path
Disallow /

gotit

Rule Path
Disallow /

gozilla

Rule Path
Disallow /

grabber

Rule Path
Disallow /

grabnet

Rule Path
Disallow /

grafula

Rule Path
Disallow /

gulliver

Rule Path
Disallow /

harvest

Rule Path
Disallow /

help.naver.com

Rule Path
Disallow /

hloader

Rule Path
Disallow /

hmview

Rule Path
Disallow /

holmes

Rule Path
Disallow /

hp\ web\ printsmart

Rule Path
Disallow /

htmlparser

Rule Path
Disallow /

httplib

Rule Path
Disallow /

httrack

Rule Path
Disallow /

humanlinks

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

ibrowse

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

idbot

Rule Path
Disallow /

image\ stripper

Rule Path
Disallow /

image\ sucker

Rule Path
Disallow /

indy\ library

Rule Path
Disallow /

ineturl

Rule Path
Disallow /

infonavirobot

Rule Path
Disallow /

interget

Rule Path
Disallow /

internet\ ninja

Rule Path
Disallow /

internetseer

Rule Path
Disallow /

iria

Rule Path
Disallow /

irlbot

Rule Path
Disallow /

java

Rule Path
Disallow /

jennybot

Rule Path
Disallow /

jetcar

Rule Path
Disallow /

joc

Rule Path
Disallow /

js-kit

Rule Path
Disallow /

justview

Rule Path
Disallow /

kenjin.spider

Rule Path
Disallow /

keyword.density

Rule Path
Disallow /

kmsearchbot

Rule Path
Disallow /

konqueror

Rule Path
Disallow /

larbin

Rule Path
Disallow /

leech

Rule Path
Disallow /

leechftp

Rule Path
Disallow /

lexibot

Rule Path
Disallow /

lftp

Rule Path
Disallow /

libweb/clshttp

Rule Path
Disallow /

libwww

Rule Path
Disallow /

likse

Rule Path
Disallow /

linguee

Rule Path
Disallow /

linkexchanger

Rule Path
Disallow /

linkextractorpro

Rule Path
Disallow /

linkscan

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

listchecker

Rule Path
Disallow /

lupa\.ru

Rule Path
Disallow /

lwp-trivial

Rule Path
Disallow /

lwp::simple

Rule Path
Disallow /

mag-net

Rule Path
Disallow /

magnet

Rule Path
Disallow /

mass\ downloader

Rule Path
Disallow /

mata.hari

Rule Path
Disallow /

memo

Rule Path
Disallow /

microsoft.url

Rule Path
Disallow /

midown\ tool

Rule Path
Disallow /

miixpc

Rule Path
Disallow /

mirror

Rule Path
Disallow /

missigua

Rule Path
Disallow /

mister\ pix

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mlbot

Rule Path
Disallow /

moget

Rule Path
Disallow /

moreoverbot

Rule Path
Disallow /

msfrontpage

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

msnbot

Rule Path
Disallow /

naverbot/1.0

Rule Path
Disallow /

navroad

Rule Path
Disallow /

nearsite

Rule Path
Disallow /

net.*reaper

Rule Path
Disallow /

net\ vampire

Rule Path
Disallow /

netants

Rule Path
Disallow /

netcache

Rule Path
Disallow /

netmechanic

Rule Path
Disallow /

netspider

Rule Path
Disallow /

netzip

Rule Path
Disallow /

nextgensearchbot

Rule Path
Disallow /

nicebot

Rule Path
Disallow /

nicerspro

Rule Path
Disallow /

ninja

Rule Path
Disallow /

njuicebot

Rule Path
Disallow /

nokia6682

Rule Path
Disallow /

noxtrumbot

Rule Path
Disallow /

npbot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

octopus

Rule Path
Disallow /

offline

Rule Path
Disallow /

omniexplorer_bot

Rule Path
Disallow /

onetouchsystems

Rule Path
Disallow /

openfind

Rule Path
Disallow /

page.*saver

Rule Path
Disallow /

pagegrabber

Rule Path
Disallow /

papa\ foto

Rule Path
Disallow /

pavuk

Rule Path
Disallow /

pcbrowser

Rule Path
Disallow /

pear

Rule Path
Disallow /

php.*/

Rule Path
Disallow /

ping

Rule Path
Disallow /

pita

Rule Path
Disallow /

pockey

Rule Path
Disallow /

port\ huron\ labs

Rule Path
Disallow /

postrank

Rule Path
Disallow /

propowerbot

Rule Path
Disallow /

prowebwalker

Rule Path
Disallow /

psbot

Rule Path
Disallow /

ptd-crawler

Rule Path
Disallow /

pump

Rule Path
Disallow /

purebot

Rule Path
Disallow /

pycurl

Rule Path
Disallow /

python

Rule Path
Disallow /

qihoobot

Rule Path
Disallow /

queryn.metasearch

Rule Path
Disallow /

realdownload

Rule Path
Disallow /

reaper

Rule Path
Disallow /

recorder

Rule Path
Disallow /

reget

Rule Path
Disallow /

repomonkey

Rule Path
Disallow /

rma

Rule Path
Disallow /

rpt-httpclient

Rule Path
Disallow /

rufusbot

Rule Path
Disallow /

rulinki

Rule Path
Disallow /

sbider

Rule Path
Disallow /

shelob

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

slurp

Rule Path
Disallow /

slysearch

Rule Path
Disallow /

smartdownload

Rule Path
Disallow /

smile

Rule Path
Disallow /

snagger

Rule Path
Disallow /

snake

Rule Path
Disallow /

sogou\ web\ spider

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

soup

Rule Path
Disallow /

spacebison

Rule Path
Disallow /

spankbot

Rule Path
Disallow /

spanner

Rule Path
Disallow /

spbot

Rule Path
Disallow /

speedy

Rule Path
Disallow /

speedy\ spider

Rule Path
Disallow /

stackrambler

Rule Path
Disallow /

stripper

Rule Path
Disallow /

sucker

Rule Path
Disallow /

suggybot

Rule Path
Disallow /

superbot

Rule Path
Disallow /

superhttp

Rule Path
Disallow /

surfbot

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

suzuran

Rule Path
Disallow /

sweb

Rule Path
Disallow /

szukacz

Rule Path
Disallow /

takeout

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleport\ pro

Rule Path
Disallow /

telesoft

Rule Path
Disallow /

the.intraformant

Rule Path
Disallow /

thefreedictionary.com

Rule Path
Disallow /

thenomad

Rule Path
Disallow /

tighttwatbot

Rule Path
Disallow /

titan

Rule Path
Disallow /

tmcrawler

Rule Path
Disallow /

tocrawl/urldispatcher

Rule Path
Disallow /

triton

Rule Path
Disallow /

true_robot

Rule Path
Disallow /

ttcrawler

Rule Path
Disallow /

turingos

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

turtlescanner

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

universalfeedparser

Rule Path
Disallow /

urly.warning

Rule Path
Disallow /

user-agent

Rule Path
Disallow /

vacuum

Rule Path
Disallow /

vci

Rule Path
Disallow /

voideye

Rule Path
Disallow /

voilabot

Rule Path
Disallow /

voyager

Rule Path
Disallow /

weazel

Rule Path
Disallow /

web.*image.*collector

Rule Path
Disallow /

web.*spy

Rule Path
Disallow /

web.*sucker

Rule Path
Disallow /

web.image.collector

Rule Path
Disallow /

web\ image\ collector

Rule Path
Disallow /

web\ sucker

Rule Path
Disallow /

webalta

Rule Path
Disallow /

webauto

Rule Path
Disallow /

webbandit

Rule Path
Disallow /

webcapture

Rule Path
Disallow /

webcollage

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webdata

Rule Path
Disallow /

webemailextrac.*

Rule Path
Disallow /

webenhancer

Rule Path
Disallow /

webex

Rule Path
Disallow /

webextractor

Rule Path
Disallow /

webfetch

Rule Path
Disallow /

webgo\ is

Rule Path
Disallow /

webleacher

Rule Path
Disallow /

webmasterworldforumbot

Rule Path
Disallow /

webmirror

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

webrecorder

Rule Path
Disallow /

websauger

Rule Path
Disallow /

website.*extractor

Rule Path
Disallow /

website.*quester

Rule Path
Disallow /

website.quester

Rule Path
Disallow /

website\ extractor

Rule Path
Disallow /

website\ quester

Rule Path
Disallow /

webspy

Rule Path
Disallow /

webster

Rule Path
Disallow /

webster.pro

Rule Path
Disallow /

webster\ pro

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webwhacker

Rule Path
Disallow /

webzip

Rule Path
Disallow /

wget

Rule Path
Disallow /

whacker

Rule Path
Disallow /

widow

Rule Path
Disallow /

wordpress

Rule Path
Disallow /

www-collector-e

Rule Path
Disallow /

www.cuill.com

Rule Path
Disallow /

www.metacrawler.com

Rule Path
Disallow /

www\.asona\.org

Rule Path
Disallow /

www\.cys\.ru

Rule Path
Disallow /

wwwoffle

Rule Path
Disallow /

wysigot

Rule Path
Disallow /

xaldon\ webspider

Rule Path
Disallow /

xenu's

Rule Path
Disallow /

yahoo

Rule Path
Disallow /

yahooseeker

Rule Path
Disallow /

yanga

Rule Path
Disallow /

yeti

Rule Path
Disallow /

yottosbot

Rule Path
Disallow /

zeus

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

*

Rule Path
Disallow /wp-admin
Disallow /wp-content/plugins
Disallow /wp-content/cache
Disallow /xmlrpc.php
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /trackback/
Disallow /tag/
Disallow /feed/
Disallow */*/feed/*/
Disallow */feed
Disallow */*/feed
Disallow /*?*
Disallow /?feed=
Disallow /?s=

Other Records

Field Value
crawl-delay 10

yandex

Rule Path
Disallow /wp-admin
Disallow /wp-content/plugins
Disallow /wp-content/cache
Disallow /xmlrpc.php
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /trackback/
Disallow /tag/
Disallow /feed/
Disallow */*/feed/*/
Disallow */feed
Disallow */*/feed
Disallow /*?*
Disallow /?feed=
Disallow /?s=

Other Records

Field Value
crawl-delay 10

serpstatbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

bubing

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap http://niidg.ru/sitemap_index.xml/
Back to top

Warnings

  • 4 invalid lines.
  • `host` is not a known field.
Back to top