Log In Sign Up

angelos-supplies.com
robots.txt

Robots Exclusion Standard data for angelos-supplies.com

Resource Scan

Scan Details

Site Domain angelos-supplies.com
Base Domain angelos-supplies.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-10-08T22:38:06+00:00
Next Scan 2025-01-06T22:38:06+00:00

Last Successful Scan

Scanned2023-03-12T17:47:25+00:00
URL https://angelos-supplies.com/robots.txt
Domain IPs 162.13.135.188, 162.13.135.189, 72.3.177.104, 72.3.177.107
Response IP 162.13.135.189
Found Yes
Hash 27e8c444b20f67a42f67113e0eddd2553aebfb2be73d5e65db4d65ee40342265
SimHash b23c5d62ca29

Groups

*

Rule Path
Disallow /bk/

ahrefsbot

Rule Path
Allow /

ahrefsbot

Rule Path
Disallow /

*

Rule Path
Disallow /store/filtered/

moget
ichiro

Rule Path
Disallow /

naverbot
yeti

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

wesee

Rule Path
Disallow /

dotbot

Rule Path
Disallow /store/filtered/

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 7

semrushbot-sa

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 7

mojeekbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 7

binglocalsearch

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 7

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5
Back to top

Comments

  • http://help.goo.ne.jp/help/article/853/
  • Info: http://help.naver.com/customer/etc/webDocument02.nhn
  • Info: http://www.youdao.com/help/webmaster/spider/
  • Info: http://www.sogou.com/docs/help/webmasters.htm#07
  • Info: http://www.majestic12.co.uk/projects/dsearch/mj12bot.php
  • http://openlinkprofiler.org/bot
  • http://www.exalead.com/search/webmasterguide
  • http://www.xovibot.net/
  • http://www.wesee.com/bot/
  • http://moz.com/researchtools/ose/dotbot
  • disallowing spidering of PFS urls to prevent the bot
  • spidering in to an eternal abyss
Back to top