itmag.pro
robots.txt
Robots Exclusion Standard data for itmag.pro
Resource Scan
Scan Details
Site Domain | itmag.pro |
Base Domain | itmag.pro |
Scan Status | Ok |
Last Scan | 2024-06-28T05:04:05+00:00 |
Next Scan | 2024-07-05T05:04:05+00:00 |
Last Scan
Scanned | 2024-06-28T05:04:05+00:00 |
URL | https://itmag.pro/robots.txt |
Domain IPs | 104.21.83.193, 172.67.181.3, 2606:4700:3032::6815:53c1, 2606:4700:3035::ac43:b503 |
Response IP | 104.21.83.193 |
Found | Yes |
Hash | 487b7a80e773bf19913eb8f5c73c973aa82351d2fc092ffb69eb3f7e35c451a0 |
SimHash | 6e3e559142d6 |
Groups
*
Rule | Path |
---|---|
Disallow | /?* |
Disallow | /index.php?* |
Disallow | /*act%3Dprofile%3Bcode%3D* |
Disallow | /*controller%3Dtagging%26view%3Dtagging* |
Disallow | /*device%3Ddesktop* |
Disallow | /*format%3Dpdf* |
Disallow | /*format%3Dfeed* |
Disallow | /go.php* |
Disallow | /*layout%3Dblog%26id%3D19* |
Disallow | /*option%3Dcom_* |
Disallow | /*print%3D* |
Disallow | /*showall%3D* |
Disallow | /*sort%3D* |
Disallow | /*sncmode%3D* |
Disallow | /*task%3Dedit* |
Disallow | /*task%3Dcaptcha* |
Disallow | /*task%3Dredirect* |
Disallow | /*task%3Dweblink.go* |
Disallow | /*PHPSESSID%3D* |
Disallow | /*view%3Dweblink* |
Disallow | /*view%3Dcategory%26id%3D2%26Itemid%3D480* |
Disallow | /*uncategorised* |
Disallow | /administrator/ |
Disallow | /component/banners/click/ |
Disallow | /component/content/ |
Disallow | /component/search/ |
Disallow | /component/uddeim/ |
Disallow | /click/ |
Disallow | /archive/ |
Disallow | /search |
Disallow | /all-articles* |
Disallow | /forum/*/edit/* |
Disallow | /forum/*/list$ |
Disallow | /forum/*/moderate$ |
Disallow | /forum/profile |
Disallow | /forum/search |
Disallow | /forum/user |
Disallow | /gauth |
Disallow | /authorization |
Disallow | /price/new |
Disallow | /login |
Disallow | /profile |
Disallow | /cache/ |
Disallow | /download/ |
Disallow | /includes/ |
Disallow | /installation/ |
Disallow | /language/ |
Disallow | /libraries/ |
Disallow | /mod/ |
Disallow | /tmp/ |
Disallow | /xmlrpc/ |
Disallow | /video |
Disallow | /wp-admin/ |
Disallow | /wp-includes/ |
Disallow | /fzsdk |
Disallow | /mrcsdk |
Disallow | /mrsdk |
Disallow | /test |
Disallow | /s/ |
Disallow | /ar$ |
Disallow | /af$ |
Disallow | /az$ |
Disallow | /fi$ |
Disallow | /be$ |
Disallow | /bg$ |
Disallow | /ca$ |
Disallow | /cs$ |
Disallow | /cy$ |
Disallow | /da$ |
Disallow | /de$ |
Disallow | /el$ |
Disallow | /es$ |
Disallow | /en$ |
Disallow | /eu$ |
Disallow | /et$ |
Disallow | /gl$ |
Disallow | /ga$ |
Disallow | /fr$ |
Disallow | /fa$ |
Disallow | /is$ |
Disallow | /it$ |
Disallow | /iw$ |
Disallow | /id$ |
Disallow | /hi$ |
Disallow | /hu$ |
Disallow | /hy$ |
Disallow | /ht$ |
Disallow | /hr$ |
Disallow | /ka$ |
Disallow | /ko$ |
Disallow | /lv$ |
Disallow | /lt$ |
Disallow | /mt$ |
Disallow | /mk$ |
Disallow | /ms$ |
Disallow | /nl$ |
Disallow | /no$ |
Disallow | /pl$ |
Disallow | /pt$ |
Disallow | /ru$ |
Disallow | /ro$ |
Disallow | /sk$ |
Disallow | /sl$ |
Disallow | /sv$ |
Disallow | /sr$ |
Disallow | /sq$ |
Disallow | /sw$ |
Disallow | /tl$ |
Disallow | /tr$ |
Disallow | /th$ |
Disallow | /ja$ |
Disallow | /uk$ |
Disallow | /ur$ |
Disallow | /vi$ |
Disallow | /yi$ |
Disallow | /zh-TW$ |
Disallow | /zh-CN$ |
Disallow | /ar/* |
Disallow | /af/* |
Disallow | /az/* |
Disallow | /fi/* |
Disallow | /be/* |
Disallow | /bg/* |
Disallow | /ca/* |
Disallow | /cs/* |
Disallow | /cy/* |
Disallow | /da/* |
Disallow | /de/* |
Disallow | /el/* |
Disallow | /es/* |
Disallow | /en/* |
Disallow | /eu/* |
Disallow | /et/* |
Disallow | /gl/* |
Disallow | /ga/* |
Disallow | /fr/* |
Disallow | /fa/* |
Disallow | /is/* |
Disallow | /it/* |
Disallow | /iw/* |
Disallow | /id/* |
Disallow | /hi/* |
Disallow | /hu/* |
Disallow | /hy/* |
Disallow | /ht/* |
Disallow | /hr/* |
Disallow | /ka/* |
Disallow | /ko/* |
Disallow | /lv/* |
Disallow | /lt/* |
Disallow | /mt/* |
Disallow | /mk/* |
Disallow | /ms/* |
Disallow | /nl/* |
Disallow | /no/* |
Disallow | /pl/* |
Disallow | /pt/* |
Disallow | /ru/* |
Disallow | /ro/* |
Disallow | /sk/* |
Disallow | /sl/* |
Disallow | /sv/* |
Disallow | /sr/* |
Disallow | /sq/* |
Disallow | /sw/* |
Disallow | /tl/* |
Disallow | /tr/* |
Disallow | /th/* |
Disallow | /ja/* |
Disallow | /uk/* |
Disallow | /ur/* |
Disallow | /vi/* |
Disallow | /yi/* |
Disallow | /zh-TW/* |
Disallow | /zh-CN/* |
Other Records
Field | Value |
---|---|
crawl-delay | 10 |
mediapartners-google
adsbot-google
yandexdirect
Rule | Path |
---|---|
Disallow | /go.php* |
Other Records
Field | Value |
---|---|
crawl-delay | 10 |
e5z7jfia9oiabpih
sl-10666-60666
alexibot
aqua_products
a .net web crawler
a1 website download/1.* (*) miggibot
abot/*
acadiauniversitywebcensusclient
activerefresh*
ad muncher*
aiderss/2.0 (aiderss.com)
amico alpha * (*) gecko/* amicoalpha/*
androiddownloadmanager
annotate_google; http://ponderer.org/*
anonymisiert*
anonymizer/*
anonymizied*
anonymous*
anonymous/*
artera (version *)
atomic_email
atomic_email_hunter/*
autohotkey
automate5
backdoorbot*
black.hole
blackwidow
blowfish*
bookmark search tool
bot mailto:craftbot@yahoo.com
botalot
botrighthere
builtbottough
bullseye*
bunnyslippers
b2w/*
backstreet browser *
basichttp/*
bdfetch
beamer*
bilgibot/*
bitbeamer/*
bittorrent/*
blocknote.net
bluecoat proxysg
bot/* (bot; *bot@bot.bot)
barkrowler
busiversebot/v1.0 (http://www.busiverse.com/bot.php)
cegbfeieh
cheesebot
cherrypickerelite/*
cherrypickerse/*
chinaclaw
copernic
crescent
crescent internet toolpak http ole control*
camcrawler*
cast
cazoodlebot/*
ce-preload
cerberiandrtrs/*
cfnetwork/*
cfschedule*
cherrypicker*
chilkat/*
cms crawler (?http://buytaert.net/crawler/)
cobweb/*
cocoal.icio.us/* (*)*
coldfusion*
contactbot/*
copyright sheriff (*)
copyrightcheck*
crawl_application
cterm/*
curl*
custo*
cyberpatrol*
cydralspider/*
cz32ts
da *
datacha0s/*
datafountains/dmoz downloader*
deepindexer*
der gro\xdfe bildersauger*
desktop sidebar*
disco*
domainsbotbot/1.*
dotbot/* (http://www.dotnetdotcom.org/*)
download*
e-societyrobot(http://www.yama.info.waseda.ac.jp/~yamana/es/)
easydl/*
ecatch*
eirgrabber
emailcollector*
emailsearcher
emailsiphon*
emailwolf*
erocrawler
express webpictures*
envolk/* (?http://www.envolk.com/envolk*)
envolk?its?spider/* (?http://www.envolk.com/envolk*)
epsilon softworks' mailmunky
estylesearch * (compatible; msie 6.0; windows nt 5.0)
exabot-images/1.0
exabot-test/*
exabot/2.0
exabot/3.0
exactseek-pagereaper-* (crawler@exactseek.com)
exalead ng/*
extractorpro*
eyenetie
extreme picture finder
ezic.com http agent *
fairad client*
fangcrawl/*
favorstarbot/*
fdm 1.x
feed::find/*
feedfetcher-google*
feedfetcher-google-igooglegadgets*
fetch libfetch/*
fget*
findfiles.net/* (robot;test_robot@gmx-topmail.de)
flaming attackbot*
flashget
flatarts_favico
followsite.com (*)
foobot*
fooky.com/scorpionbot/scoutout;*
forschungsportal/*
fotochecker
franklin locator*
freshdownload/*
frontpage
fyberspider*
gaisbot
gamespyhttp/*
getright
getright/*
getrightpro/*
getring*
getsmart/*
gnome-vfs/*
getweb*
go!zilla*
go-ahead-got-it*
gozilla/*
grabnet
grafula
gsa-crawler*
gulper web *
gurujibot/1.*
harvest/*
hatena antenna/*
hatena bookmark/*
hatena rss/*
hatena::crawler/*
hatenascreenshot*
hcat/*
healthbot/health_and_longevity_project_(healthhaven.com)
hiddenmarket-*
hitcrawler_0.*
hloader
holmes/*
hoowwwer/*
html2jpg blackbox, http://www.html2jpg.com
htmlparser/*
http generic
http://anonymouse.org/*
http://arachnode.net*
http://hilfe.acont.de/bot.html acontbot
httpclient*
httperf/*
httpfetch/*
httpgrab
httpsession
httpunit/*
hyperestraier/*
ice_getfile
iconsurf/2.*
icopyright conductor*
ie/6.01 (cp/m; 8-bit*)
iexplore.exe
igetter/*
inet - eureka app
inetbot/* (?http://www.inetbot.com/bot.html)
ineturl/*
ineturl:/*
infociousbot (?http://corp.infocious.com/tech_crawler.php)
inne: mozilla/4.0 (compatible; cerberian drtrs*)
internet exploiter/*
internet explore *
internet explorer *
internet ninja*
ip*works!*/*
ipiumbot laurion(dot)com
irlbot/*
irssiurllog/*
iwagent/*
jetbrains omea reader*
jpluck/*
just-crawler(*)
kapere (http://www.kapere.com)
kbeebot/0.*
kevin http://*
kolinka forum search (www.kolinka.com)
kontiki client*
kretrieve/
lachesis
leechftp
leechget*
letscrawl.com/1.0*
lftp/3.2.1
libcurl-agent/*
libweb/clshttp*
liferea/1.* (linux; *; http://liferea.sf.net/)
lightningdownload/*
lincoln state web browser
link valet online*
linkextractorpro*
links4us-crawler,*
lmqueuebot/*
looq/0.1*
lorkyll *.* -- lorkyll@444.net
lsearch/sondeur
lucidmedia clicksense/4.?
lwp*
made by zmeu @ whitehat v0.* (www.whitehat.ro)
mapoftheinternet.com?(?http://mapoftheinternet.com)
metaproducts download express/*
metatagsdir/*
mfc foundation class library*
mfc_tear_sample
mfhttpscan
microsoft bits/*
microsoft data access internet publishing provider cache manager
microsoft data access internet publishing provider dav*
microsoft data access internet publishing provider protocol discovery
microsoft internet explorer
microsoft office existence discovery
microsoft office protocol discovery
microsoft office/* (*picture manager*)
microsoft url control*
microsoft visio msie
microsoft windows network diagnostics
microsoft-webdav-miniredir/*
missigua locator*
mister pix*
mono browser capabilities updater*
moozilla
morfeus fucking scanner
movabletype/*
mozilla/* (compatible; linktiger/*; *http://www.linktiger.com*)
mozilla/* (compatible; offbyone; windows*) webster pro v3.*
mozilla/* (turingos; turing machine; 0.0)
mozilla/0.9* no dos :) (linux*)
mozilla/2.0 (compatible; newt activex; win32)
mozilla/3.0 (compatible; indy library)
mozilla/4.0 (compatible; advanced email extractor*)
mozilla/4.0 (compatible; bordermanager*)
mozilla/4.0 (compatible; botw spider; *http://botw.org)
mozilla/4.0 (compatible; cerberian drtrs*)
mozilla/4.0 (compatible; getleft*)
mozilla/4.0 (compatible; http://search.thunderstone.com/texis/websearch/about.html)
mozilla/4.0 (compatible; msie ?.0; safersurf*)
mozilla/4.0 (compatible; msie 4.01; vonna.com b o t)
mozilla/4.0 (compatible; msie 6.0; bluecoat drtr)
mozilla/4.0 (compatible; scumbot/*; linux/*)
mozilla/4.0 (compatible; spider; linux)
mozilla/4.0 (compatible; trend micro tmdr 1.*
mozilla/4.0 (compatible; win32)
mozilla/5.0 (*) gecko/* firefox/2.0 oneriot/1.0 (http://www.oneriot.com)
mozilla/5.0 (*) voilabot*
mozilla/5.0 (*http://gnomit.com/) gecko/* gnomit/1.0
mozilla/5.0 (compatible; aboutusbot/*)
mozilla/5.0 (compatible; buzzrankingbot/*)
mozilla/5.0 (compatible; charlotte/*; *)
mozilla/5.0 (compatible; clixsense; http://www.clixsense.com/)
mozilla/5.0 (compatible; crawly/1.*; +http://*/crawler.html)
mozilla/5.0 (compatible; del.icio.us-thumbnails/*; *) khtml/* (like gecko)
mozilla/5.0 (compatible; dkimrepbot/*)
mozilla/5.0 (compatible; dotbot/*; http://www.dotnetdotcom.org/*)
mozilla/5.0 (compatible; exabot-images/3.0*)
mozilla/5.0 (compatible; exabot/3.0*)
mozilla/5.0 (compatible; ipcheck server monitor*)
mozilla/5.0 (compatible; jadynavebot; *http://www.jadynave.com/robot*
mozilla/5.0 (compatible; kaloogabot; http://www.kalooga.com/info.html?page=crawler)
mozilla/5.0 (compatible; legalanalysisagent/1.*; http://www.legalx.net)
mozilla/5.0 (compatible; mj12bot/v1.*)
mozilla/5.0 (compatible; netcraftsurveyagent/1.0; *info@netcraft.com)
mozilla/5.0 (compatible; nextthing.org/*)
mozilla/5.0 (compatible; ngbot/*)
mozilla/5.0 (compatible; oso;*
mozilla/5.0 (compatible; scoutjet; +http://www.scoutjet.com/)
mozilla/5.0 (compatible; scrubby/*; +http://www.scrubtheweb.com/abs/meta-check.html)
mozilla/5.0 (compatible; seznam screenshot-generator 2.0;*)
mozilla/5.0 (compatible; speedy spider; http://www.entireweb.com/about/search_tech/speedy_spider/)
mozilla/5.0 (compatible; theophrastus/*)
mozilla/5.0 (compatible; twitturls; +http://twitturls.com)
mozilla/5.0 (compatible; viralheat bot/*)
mozilla/5.0 (compatible; webbot/*)
mozilla/5.0 (compatible; webscan v0.*; +http://otc.dyndns.org/webscan/)
mozilla/5.0 (compatible; yodaobot/1.*)
mozilla/5.0 (compatible;yodaobot-image/1.*)
mozilla/5.0 (macintosh; intel mac os x) excel/12.*
mozilla/5.0 (macintosh; u; *mac os x; *) applewebkit/* (*) pandora/2.*
mozilla/5.0 (snappreviewbot) gecko/* firefox/*
mozilla/5.0 (twiceler*)
mozilla/5.0 (windows; u; windows nt 5.1; en-us) speedy spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
mozilla/5.0 gurlchecker/*
mp3spider cn-search-devel at yahoo-inc dot com
mqbot*
msproxy/*
myzilla
naofavicon4ie*
net vampire/*
net_vampire*
netants*
netcarta_webmapper/*
netchart adv crawler*
netid.com bot*
netprospector*
netpumper*
netsucker*
netzip downloader*
newsgator/*
nextgensearchbot*(for information visit *)
nextools webagent*
ng-search/*
ng/*
nicebot
nozilla/p.n (just for ids woring)
np/*
npbot*
nso_debugger_user/2.0
nudelsalat/*
nutch/0.? (openx spider)
nutscrape
nutscrape/* (cp/m; 8-bit*)
nv32ts
obot
ocn-soc/*
offline downloader*
offline explorer*
online link validator (http://www.dead-links.com/)
open web analytics bot*
oracle enterprise search
ossproxy*
outfoxbot/*
p3p client
pagedown*
pageload*
pagenest/*
pajaczek/*
panscient.com
pavuk/*
pear http_request*
pete-spider/1.*
php*
picaloader*
pigblock (windows nt 5.1; u)*
pixfinder/*
plantynet_webrobot*
pmafind
pockey*
poe-component-client-http/*
polybot?*
privoxy/*
prowebwalker*
proxytester*
prozilla*
psbot/* (?http://www.picsearch.com/bot.html)
pycurl/*
python*
quickfinder crawler
radiation retriever*
realdownload/*
reget
reget*
redcarpet/*
repomonkey*
rpt-httpclient/*
rssimagesbot/0.1 (*http://herbert.groot.jebbink.nl/?app=rssimages)
sbl-bot*
scollspider/2.*
scoutabout*
searchbot admin@google.com
seasydl/*
seeker.lookseek.com
seznambot/*
shaboyi spider
shareaza*
shelob (shelob@gmx.net)
shelob v1.*
sherlock/*
shim?crawler*
showxml/1.0 libwww/5.4.0
silentsurf*
site valet online*
siteparser/*
sitesnagger*
sitesucker/*
sitewinder*
slysearch/*
smallproxy*
smartdownload/*
sna-0.0.*
snapbot/*
snoopy*
softwing_tear_agent*
sogou develop spider/*
sogou head spider*
sogou js robot(*)
sogou orion spider/*
sogou pic agent
sogou pic spider/*
sogou push spider/*
sogou spider
sogou web spider*
sogou-test-spider/*
sohu*
space*bison/*
spankbot*
spbot
speeddownload/*
speedy spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
spider (tspyyp@tom.com)
sqeobot/0.*
squigglebotbot/*
sqworm/*
star*downloader/*
steeler/*
steroid download
strategic board bot (?http://www.strategicboard.com)
sunrise/0.*
superbot/*
superhttp/*
surf knight
surfcontrol
surveybot/*
synapticsearch/ai crawler 1.?
taiga web spider
talkro web-shot/*
tarantula/*
tasap-image-robot/0.* (http://www.tasap.com)
tcl http client package*
teleport*
terrawizbot/*
theinformant*
theme spider*
titanium 2005 (4.02.01)
tmcrawler
toata dragostea*
turnitinbot/*
tutorgigbot/*
tutorial crawler*
twingly recon
twisted pagegetter
twitturly*
uoftdb_experiment* (leehyun@cs.toronto.edu)
uri::fetch/*
url2file/*
user*agent:*
user_agent
usyd-nlp-spider*
updownerbot
utilmind httpget
vci webviewer*
vegas95/*
vengabot/*
virus_detector*
vobsub
wadaino.jp-crawler*
wap_browser/5.0 (compatible; yodaobot/1.*)
wbsearchbot
web downloader*
web downloader/*
web image collector*
web magnet*
webalta crawler/*
webauto/*
webbandit/*
webclipping.com
webcollage*
webcopier*
webcorp/*
webdownloader*
webenhancer*
webfetch
webfetch/*
webgatherer*
webget
webimages * (?http://herbert.groot.jebbink.nl/?app=webimages?)
webminer*
webpix*
webreaper*
webripper*
websauger*
website downloader*
website extractor*
website quester*
website.quester*
websiteextractor*
websnatcher*
webster pro*
webster.pro*
webstripper*
webwhacker*
webzip*
west wind internet protocols*
wget*
winhttp*
winscripter inet tools
wintools
wire/* (linux*bot,robot,spider,crawler)
wisebot/*
wordpress-b-/2.*
wordpress-do-p-/2.*
woriobot*
www-mechanize/*
wwwster/* (beta, mailto:gue@cis.uni-muenchen.de)
xaldon webspider*
xenu* link sleuth*
xerka webbot v1.*
xspider*
y!oasis*
yahoo-mmcrawler*
yodaobot/*
yodaobot/1.* (*)
yoow!/* (?http://www.yoow.eu)
yrl_odp_crawler
zao-crawler
zao/*
zend_http_client
zibb crawler (email address / www address)
surdotlybot
trendictionbot
mj12bot
Rule | Path |
---|---|
Disallow | / |
Other Records
Field | Value |
---|---|
sitemap | https://itmag.pro/sitemap.xml |
Warnings
- 29 invalid lines.
- `clean-param` is not a known field.
- `host` is not a known field.
Comments