Log In Sign Up

cdn.powered-by-nitrosell.com
robots.txt

Robots Exclusion Standard data for cdn.powered-by-nitrosell.com

Resource Scan

Scan Details

Site Domain cdn.powered-by-nitrosell.com
Base Domain powered-by-nitrosell.com
Scan Status Ok
Last Scan2024-05-22T18:12:57+00:00
Next Scan 2024-06-21T18:12:57+00:00

Last Scan

Scanned2024-05-22T18:12:57+00:00
URL https://cdn.powered-by-nitrosell.com/robots.txt
Domain IPs 13.35.18.123, 13.35.18.24, 13.35.18.60, 13.35.18.93
Response IP 13.35.18.123
Found Yes
Hash 3451a85c2186145de3edadb8ebd05e442729eee9ae91d0aede88f75339575e16
SimHash b21c5502da0b

Groups

ahrefsbot

Rule Path
Disallow /

*

Rule Path
Disallow /store/filtered/

moget
ichiro

Rule Path
Disallow /

naverbot
yeti

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

wesee

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 7

binglocalsearch

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 7

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5
Back to top

Comments

  • netstorage
  • http://help.goo.ne.jp/help/article/853/
  • Info: http://help.naver.com/customer/etc/webDocument02.nhn
  • Info: http://www.youdao.com/help/webmaster/spider/
  • Info: http://www.sogou.com/docs/help/webmasters.htm#07
  • Info: http://www.majestic12.co.uk/projects/dsearch/mj12bot.php
  • http://openlinkprofiler.org/bot
  • http://www.exalead.com/search/webmasterguide
  • http://www.xovibot.net/
  • http://www.wesee.com/bot/
  • http://moz.com/researchtools/ose/dotbot
  • disallowing spidering of PFS urls to prevent the bot
  • spidering in to an eternal abyss
Back to top