Log In Sign Up

2guyscigars.com
robots.txt

Robots Exclusion Standard data for 2guyscigars.com

Resource Scan

Scan Details

Site Domain 2guyscigars.com
Base Domain 2guyscigars.com
Scan Status Ok
Last Scan2024-06-04T03:28:16+00:00
Next Scan 2024-06-18T03:28:16+00:00

Last Scan

Scanned2024-06-04T03:28:16+00:00
URL https://www.2guyscigars.com/robots.txt
Domain IPs 72.3.177.104, 72.3.177.107
Response IP 72.3.177.107
Found Yes
Hash 24c294ff2fdd8f6ef4702161f0fab91ff6b0c1d1e9e260411604c10303fb1212
SimHash b23c5d22ca0b

Groups

ahrefsbot

Rule Path
Disallow /

*

Rule Path
Disallow /store/filtered/

moget
ichiro

Rule Path
Disallow /

naverbot
yeti

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

wesee

Rule Path
Disallow /

dotbot

Rule Path
Disallow /store/filtered/

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 7

semrushbot-sa

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 7

mojeekbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 7

binglocalsearch

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 7

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5
Back to top

Comments

  • http://help.goo.ne.jp/help/article/853/
  • Info: http://help.naver.com/customer/etc/webDocument02.nhn
  • Info: http://www.youdao.com/help/webmaster/spider/
  • Info: http://www.sogou.com/docs/help/webmasters.htm#07
  • Info: http://www.majestic12.co.uk/projects/dsearch/mj12bot.php
  • http://openlinkprofiler.org/bot
  • http://www.exalead.com/search/webmasterguide
  • http://www.xovibot.net/
  • http://www.wesee.com/bot/
  • http://moz.com/researchtools/ose/dotbot
  • disallowing spidering of PFS urls to prevent the bot
  • spidering in to an eternal abyss
Back to top