Log In Sign Up

app.shufti.jp
robots.txt

Robots Exclusion Standard data for app.shufti.jp

Resource Scan

Scan Details

Site Domain app.shufti.jp
Base Domain shufti.jp
Scan Status Ok
Last Scan2024-09-16T11:57:25+00:00
Next Scan 2024-10-16T11:57:25+00:00

Last Scan

Scanned2024-09-16T11:57:25+00:00
URL https://app.shufti.jp/robots.txt
Domain IPs 2600:9000:2003:200:10:e855:8180:93a1, 2600:9000:2003:3000:10:e855:8180:93a1, 2600:9000:2003:6200:10:e855:8180:93a1, 2600:9000:2003:6600:10:e855:8180:93a1, 2600:9000:2003:6a00:10:e855:8180:93a1, 2600:9000:2003:7000:10:e855:8180:93a1, 2600:9000:2003:b800:10:e855:8180:93a1, 2600:9000:2003:e400:10:e855:8180:93a1, 52.84.229.106, 52.84.229.110, 52.84.229.56, 52.84.229.77
Response IP 52.84.229.106
Found Yes
Hash 139fdfc81290888381a58706ca5babaf6e0393f2c24c44179fbecd9b9ef11f98
SimHash 135e7b654cb3

Groups

*

Rule Path
Disallow /test.php
Disallow /*/ajax_*

baiduspider

Rule Path
Disallow /

baiduimagespider

Rule Path
Disallow /

baidumobaider

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

speedy

Rule Path
Disallow /

yeti

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mlbot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

sitebot/0.1

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

holmes

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

yodaobot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandexmedia

Rule Path
Disallow /

lexxebot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

nerdbynature.bot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

sindicebot

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://app.shufti.jp/static/sitemap.xml
Back to top

Comments

  • Exclude all robots access to ajax actions #13742
  • http://www.baidu.jp/spider/
  • http://help.soso.com/webspider.htm
  • http://www.entireweb.com/about/search_tech/speedy_spider/
  • http://help.naver.com/robots/
  • http://www.dotnetdotcom.org/
  • http://www.metadatalabs.com/mlbot
  • http://www.alexa.com/site/help/webmasters
  • http://www.sitebot.org/robot/
  • http://www.majestic12.co.uk/bot.php
  • http://www.youdao.com/help/webmaster/spider/
  • http://morfeo.centrum.cz/bot
  • http://help.soso.com/webspider.htm
  • http://www.yodao.com/help/webmaster/spider/
  • http://yandex.com/bots
  • http://www.nerdbynature.net/bot
  • http://www.turnitin.com/robot/crawlerinfo.html
  • http://sindice.com/developers/bot
  • http://www.warebay.com/bot.html
Back to top