Log In Sign Up

poetsandquants.com
robots.txt

Robots Exclusion Standard data for poetsandquants.com

Resource Scan

Scan Details

Site Domain poetsandquants.com
Base Domain poetsandquants.com
Scan Status Ok
Last Scan2024-09-28T16:44:21+00:00
Next Scan 2024-10-28T16:44:21+00:00

Last Scan

Scanned2024-09-28T16:44:21+00:00
URL https://poetsandquants.com/robots.txt
Domain IPs 141.193.213.10, 141.193.213.11
Response IP 141.193.213.10
Found Yes
Hash 34ecd8e7bfe22ae0bea29969bd2dd554a92b95e13a34b28c0868c51ca2279404
SimHash 80644002c21b

Groups

yandex

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

gluten

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

rogerbot

Rule Path
Allow /

moget
ichiro

Rule Path
Disallow /

naverbot
yeti

Rule Path
Disallow /

baiduspider
baiduspider-video
baiduspider-image

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

*

Rule Path
Allow /wp-content/uploads/
Disallow /wp-content/plugins/
Allow /wp-content/plugins/sdac-mba-watch/images/
Allow /wp-content/plugins/*.js
Allow /wp-content/plugins/*.css
Disallow /readme.html

Other Records

Field Value
crawl-delay 5
Back to top

Comments

  • Yandex
  • Ahrefs.com
  • http://moz.com/products
  • https://www.distilnetworks.com/bot-directory/bot/gluten-free-crawler/
  • http://www.majestic12.co.uk/projects/dsearch/mj12bot.php
  • https://moz.com/help/guides/moz-procedures/what-is-rogerbot
  • Goo (JP)
  • Info (Japanese): http://help.goo.ne.jp/help/article/704/
  • Info (English): http://help.goo.ne.jp/help/article/853/
  • Naver (KR)
  • Info: http://help.naver.com/customer/etc/webDocument02.nhn
  • Baidu (CN)
  • Info: http://www.baidu.com/search/spider.htm
  • SoGou (CN)
  • Info: http://www.sogou.com/docs/help/webmasters.htm#07
  • Youdao (CN)
  • Info: http://www.youdao.com/help/webmaster/spider/
  • Petalbot (CN)
  • Info: https://aspiegel.com/petalbot
  • Bytespider (CN)
  • Info: https://www.feitsui.com/en/article/32
Back to top