Log In Sign Up

app.shufti.jp
robots.txt

Robots Exclusion Standard data for app.shufti.jp

Resource Scan

Scan Details

Site Domain app.shufti.jp
Base Domain shufti.jp
Scan Status Ok
Last Scan2024-11-15T12:18:58+00:00
Next Scan 2024-12-15T12:18:58+00:00

Last Scan

Scanned2024-11-15T12:18:58+00:00
URL https://app.shufti.jp/robots.txt
Domain IPs 13.227.254.115, 13.227.254.129, 13.227.254.14, 13.227.254.79, 2600:9000:200a:1400:10:e855:8180:93a1, 2600:9000:200a:1e00:10:e855:8180:93a1, 2600:9000:200a:3e00:10:e855:8180:93a1, 2600:9000:200a:5400:10:e855:8180:93a1, 2600:9000:200a:a200:10:e855:8180:93a1, 2600:9000:200a:b200:10:e855:8180:93a1, 2600:9000:200a:e800:10:e855:8180:93a1, 2600:9000:200a:fc00:10:e855:8180:93a1
Response IP 13.227.254.79
Found Yes
Hash 139fdfc81290888381a58706ca5babaf6e0393f2c24c44179fbecd9b9ef11f98
SimHash 135e7b654cb3

Groups

*

Rule Path
Disallow /test.php
Disallow /*/ajax_*

baiduspider

Rule Path
Disallow /

baiduimagespider

Rule Path
Disallow /

baidumobaider

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

speedy

Rule Path
Disallow /

yeti

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mlbot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

sitebot/0.1

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

holmes

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

yodaobot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandexmedia

Rule Path
Disallow /

lexxebot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

nerdbynature.bot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

sindicebot

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://app.shufti.jp/static/sitemap.xml
Back to top

Comments

  • Exclude all robots access to ajax actions #13742
  • http://www.baidu.jp/spider/
  • http://help.soso.com/webspider.htm
  • http://www.entireweb.com/about/search_tech/speedy_spider/
  • http://help.naver.com/robots/
  • http://www.dotnetdotcom.org/
  • http://www.metadatalabs.com/mlbot
  • http://www.alexa.com/site/help/webmasters
  • http://www.sitebot.org/robot/
  • http://www.majestic12.co.uk/bot.php
  • http://www.youdao.com/help/webmaster/spider/
  • http://morfeo.centrum.cz/bot
  • http://help.soso.com/webspider.htm
  • http://www.yodao.com/help/webmaster/spider/
  • http://yandex.com/bots
  • http://www.nerdbynature.net/bot
  • http://www.turnitin.com/robot/crawlerinfo.html
  • http://sindice.com/developers/bot
  • http://www.warebay.com/bot.html
Back to top