Log In Sign Up

shufti.jp
robots.txt

Robots Exclusion Standard data for shufti.jp

Resource Scan

Scan Details

Site Domain shufti.jp
Base Domain shufti.jp
Scan Status Ok
Last Scan2024-09-18T22:19:40+00:00
Next Scan 2024-10-18T22:19:40+00:00

Last Scan

Scanned2024-09-18T22:19:40+00:00
URL https://shufti.jp/robots.txt
Redirect https://app.shufti.jp/robots.txt
Redirect Domain app.shufti.jp
Redirect Base shufti.jp
Domain IPs 108.157.254.112, 108.157.254.35, 108.157.254.62, 108.157.254.70
Redirect IPs 2600:9000:2003:1400:10:e855:8180:93a1, 2600:9000:2003:200:10:e855:8180:93a1, 2600:9000:2003:5200:10:e855:8180:93a1, 2600:9000:2003:7000:10:e855:8180:93a1, 2600:9000:2003:9600:10:e855:8180:93a1, 2600:9000:2003:d000:10:e855:8180:93a1, 2600:9000:2003:e000:10:e855:8180:93a1, 2600:9000:2003:e800:10:e855:8180:93a1, 52.84.229.106, 52.84.229.110, 52.84.229.56, 52.84.229.77
Response IP 52.84.229.110
Found Yes
Hash 139fdfc81290888381a58706ca5babaf6e0393f2c24c44179fbecd9b9ef11f98
SimHash 135e7b654cb3

Groups

*

Rule Path
Disallow /test.php
Disallow /*/ajax_*

baiduspider

Rule Path
Disallow /

baiduimagespider

Rule Path
Disallow /

baidumobaider

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

speedy

Rule Path
Disallow /

yeti

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mlbot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

sitebot/0.1

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

holmes

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

yodaobot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandexmedia

Rule Path
Disallow /

lexxebot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

nerdbynature.bot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

sindicebot

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://app.shufti.jp/static/sitemap.xml
Back to top

Comments

  • Exclude all robots access to ajax actions #13742
  • http://www.baidu.jp/spider/
  • http://help.soso.com/webspider.htm
  • http://www.entireweb.com/about/search_tech/speedy_spider/
  • http://help.naver.com/robots/
  • http://www.dotnetdotcom.org/
  • http://www.metadatalabs.com/mlbot
  • http://www.alexa.com/site/help/webmasters
  • http://www.sitebot.org/robot/
  • http://www.majestic12.co.uk/bot.php
  • http://www.youdao.com/help/webmaster/spider/
  • http://morfeo.centrum.cz/bot
  • http://help.soso.com/webspider.htm
  • http://www.yodao.com/help/webmaster/spider/
  • http://yandex.com/bots
  • http://www.nerdbynature.net/bot
  • http://www.turnitin.com/robot/crawlerinfo.html
  • http://sindice.com/developers/bot
  • http://www.warebay.com/bot.html
Back to top