Log In Sign Up

yesjav.info
robots.txt

Robots Exclusion Standard data for yesjav.info

Resource Scan

Scan Details

Site Domain yesjav.info
Base Domain yesjav.info
Scan Status Ok
Last Scan2024-11-11T03:54:28+00:00
Next Scan 2024-12-11T03:54:28+00:00

Last Scan

Scanned2024-11-11T03:54:28+00:00
URL https://yesjav.info/robots.txt
Redirect https://www.yesjav100.com/robots.txt
Redirect Domain www.yesjav100.com
Redirect Base yesjav100.com
Domain IPs 104.21.39.37, 172.67.142.233, 2606:4700:3031::ac43:8ee9, 2606:4700:3034::6815:2725
Redirect IPs 104.21.59.192, 172.67.182.237, 2606:4700:3030::6815:3bc0, 2606:4700:3031::ac43:b6ed
Response IP 104.21.59.192
Found Yes
Hash b3b7d92a5fc71febc3ddc8d3a0314357d92b80c758d16b267b33e0bf0ffcb51b
SimHash 591761c0a510

Groups

seekport
datanyze
experibot
indeedbot
extlinksbot
crawler4j
dataprovider
daum
mauibot
mauibot (crawler.feedback+wc@gmail.com)
panscient.com
vscooter
psbot
ia_archiver
mj12bot
twiceler
velenpublicwebcrawler
serpstatbot
megaindex.ru/2.0
megaindex.ru
megaindex.ru
bytespider
taptubot
googlebot-image
twengabot
sitebot
ahrefsbot
ezooms
sistrix
aihitbot
infopath
infopath.2
swebot
ec2linkfinder
turnitinbot
the knowledge ai
mappy
grapeshotcrawler
steeler
jooblebot
adsbot
amazonbot
petalbot
rainbot
meta-externalagent
facebookexternalhit
gptbot
searchbot

Rule Path
Disallow /

searchmetericsbot
wbsearchbot
exabot
sosospider
ip-web-crawler.com
netestate ne crawler
aboundexbot
aboundex
meanpathbot
mail.ru
spbot
archive.org_bot
linkpadbot
easouspider
seznambot
wotbox
blexbot
xovibot
semrushbot
a6-indexer
riddler
loadtimebot
obot
mojeekbot
memorybot
ltx71

Rule Path
Disallow /

advbot
smtbot
yisouspider
lssrocketcrawler
gsa-crawler
nutch
tbot-nutch
thunderstone
yacybot
ranksonicbot
betabot
parsijoo-bot
nextgensearchbot
gocrawl
plukkie
applebot
lipperhey
safednsbot
rankactivelinkbot
uptimebot
seeker
cliqzbot
domaincrawler
yoozbot
coccocbot-web
qwantify
siteexplorer
findxbot
garlikcrawler
zoominfobot
bubing
barkrowler
rogerbot
dotbot
jamesbot
contacts-crawler
ccbot
idbot
dnyzbot
piplbot
alphabot
alphaseobot
alphaseobot-sa
seokicks-robot
bingbot

Rule Path
Disallow /
Allow /zb_users/upload/
Disallow /zb_system/
Disallow /zb_users/

googlebot

Rule Path
Allow /zb_system/function/c_html_js_add.asp
Allow /zb_system/function/c_html_js.asp
Allow /zb_system/function/script/common.js
Allow /zb_users/THEME/Newdur/style/default.css

googlebot

Rule Path
Allow /zb_system/function/c_html_js_add.asp
Allow /zb_system/function/c_html_js.asp
Allow /zb_system/function/script/common.js
Allow /zb_users/THEME/Newdur/style/default.css
Disallow /*.jpg$
Disallow /*.JPG$
Disallow /*.png$
Disallow /*.PDF$
Disallow /*.pdf$
Disallow /*.mp3$
Disallow /*.MOV$
Disallow /*.mov$
Disallow /*.AVI$
Disallow /*.avi$
Disallow /*.csv$
Disallow /*.svg$
Disallow /*.data$
Allow /zb_users/upload/
Disallow /zb_system/
Disallow /zb_users/

*

Rule Path
Allow /zb_system/function/c_html_js_add.asp
Allow /zb_system/function/c_html_js.asp
Allow /zb_system/function/script/common.js
Allow /zb_users/THEME/Newdur/style/default.css
Disallow /*.jpg$
Disallow /*.JPG$
Disallow /*.png$
Disallow /*.PDF$
Disallow /*.pdf$
Disallow /*.mp3$
Disallow /*.MOV$
Disallow /*.mov$
Disallow /*.AVI$
Disallow /*.avi$
Disallow /*.csv$
Disallow /*.svg$
Disallow /*.data$
Allow /zb_users/upload/
Disallow /zb_system/
Disallow /zb_users/
Back to top

Other Records

Field Value
sitemap http://www.yesjav.info/sitemap.xml
Back to top

Comments

  • User-agent: Yandex
  • User-agent: Baiduspider
  • User-agent: Sogou blog
  • User-agent: Sogou inst spider
  • User-agent: Sogou News Spider
  • User-agent: Sogou Orion spider
  • User-agent: Sogou spider2
  • User-agent: Sogou web spider
  • Disallow: /index_0*$
  • Disallow: /*index_0*$
  • Disallow: /xmas_*
  • Disallow: /index_0*$
  • Disallow: /*index_0*$
  • Disallow: /xmas_*
Back to top

Warnings

  • 1 invalid line.
Back to top