Log In Sign Up

yesjav.net
robots.txt

Robots Exclusion Standard data for yesjav.net

Resource Scan

Scan Details

Site Domain yesjav.net
Base Domain yesjav.net
Scan Status Ok
Last Scan2024-10-25T19:52:25+00:00
Next Scan 2024-11-24T19:52:25+00:00

Last Scan

Scanned2024-10-25T19:52:25+00:00
URL https://yesjav.net/robots.txt
Redirect https://www.yesjav100.com/robots.txt
Redirect Domain www.yesjav100.com
Redirect Base yesjav100.com
Domain IPs 104.21.10.168, 172.67.146.31, 2606:4700:3030::ac43:921f, 2606:4700:3036::6815:aa8
Redirect IPs 104.21.59.192, 172.67.182.237, 2606:4700:3030::6815:3bc0, 2606:4700:3031::ac43:b6ed
Response IP 104.21.59.192
Found Yes
Hash 4414d0326d38be74f6eec23b22eda341586ee3a0f19d2ce945f6361c49b86072
SimHash 591761c0a710

Groups

seekport
datanyze
experibot
indeedbot
extlinksbot
crawler4j
dataprovider
daum
mauibot
mauibot (crawler.feedback+wc@gmail.com)
panscient.com
vscooter
psbot
ia_archiver
mj12bot
twiceler
velenpublicwebcrawler
serpstatbot
megaindex.ru/2.0
megaindex.ru
megaindex.ru
bytespider
taptubot
googlebot-image
twengabot
sitebot
ahrefsbot
ezooms
sistrix
aihitbot
infopath
infopath.2
swebot
ec2linkfinder
turnitinbot
the knowledge ai
mappy
grapeshotcrawler
steeler
jooblebot
adsbot
amazonbot
petalbot
rainbot
meta-externalagent
facebookexternalhit
gptbot

Rule Path
Disallow /

searchmetericsbot
wbsearchbot
exabot
sosospider
ip-web-crawler.com
netestate ne crawler
aboundexbot
aboundex
meanpathbot
mail.ru
spbot
archive.org_bot
linkpadbot
easouspider
seznambot
wotbox
blexbot
xovibot
semrushbot
a6-indexer
riddler
loadtimebot
obot
mojeekbot
memorybot
ltx71

Rule Path
Disallow /

advbot
smtbot
yisouspider
lssrocketcrawler
gsa-crawler
nutch
tbot-nutch
thunderstone
yacybot
ranksonicbot
betabot
parsijoo-bot
nextgensearchbot
gocrawl
plukkie
applebot
lipperhey
safednsbot
rankactivelinkbot
uptimebot
seeker
cliqzbot
domaincrawler
yoozbot
coccocbot-web
qwantify
siteexplorer
findxbot
garlikcrawler
zoominfobot
bubing
barkrowler
rogerbot
dotbot
jamesbot
contacts-crawler
ccbot
idbot
dnyzbot
piplbot
alphabot
alphaseobot
alphaseobot-sa
seokicks-robot
bingbot

Rule Path
Disallow /
Allow /zb_users/upload/
Disallow /zb_system/
Disallow /zb_users/

googlebot

Rule Path
Allow /zb_system/function/c_html_js_add.asp
Allow /zb_system/function/c_html_js.asp
Allow /zb_system/function/script/common.js
Allow /zb_users/THEME/Newdur/style/default.css

googlebot

Rule Path
Allow /zb_system/function/c_html_js_add.asp
Allow /zb_system/function/c_html_js.asp
Allow /zb_system/function/script/common.js
Allow /zb_users/THEME/Newdur/style/default.css
Disallow /*.jpg$
Disallow /*.JPG$
Disallow /*.png$
Disallow /*.PDF$
Disallow /*.pdf$
Disallow /*.mp3$
Disallow /*.MOV$
Disallow /*.mov$
Disallow /*.AVI$
Disallow /*.avi$
Disallow /*.csv$
Disallow /*.svg$
Disallow /*.data$
Allow /zb_users/upload/
Disallow /zb_system/
Disallow /zb_users/

*

Rule Path
Allow /zb_system/function/c_html_js_add.asp
Allow /zb_system/function/c_html_js.asp
Allow /zb_system/function/script/common.js
Allow /zb_users/THEME/Newdur/style/default.css
Disallow /*.jpg$
Disallow /*.JPG$
Disallow /*.png$
Disallow /*.PDF$
Disallow /*.pdf$
Disallow /*.mp3$
Disallow /*.MOV$
Disallow /*.mov$
Disallow /*.AVI$
Disallow /*.avi$
Disallow /*.csv$
Disallow /*.svg$
Disallow /*.data$
Allow /zb_users/upload/
Disallow /zb_system/
Disallow /zb_users/
Back to top

Other Records

Field Value
sitemap http://www.yesjav.info/sitemap.xml
Back to top

Comments

  • User-agent: Yandex
  • User-agent: Baiduspider
  • User-agent: Sogou blog
  • User-agent: Sogou inst spider
  • User-agent: Sogou News Spider
  • User-agent: Sogou Orion spider
  • User-agent: Sogou spider2
  • User-agent: Sogou web spider
  • Disallow: /index_0*$
  • Disallow: /*index_0*$
  • Disallow: /xmas_*
  • Disallow: /index_0*$
  • Disallow: /*index_0*$
  • Disallow: /xmas_*
Back to top

Warnings

  • 1 invalid line.
Back to top