Log In Sign Up

theceramicshop.com
robots.txt

Robots Exclusion Standard data for theceramicshop.com

Resource Scan

Scan Details

Site Domain theceramicshop.com
Base Domain theceramicshop.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't establish SSL connection.
Last Scan2024-09-15T02:11:04+00:00
Next Scan 2024-12-14T02:11:04+00:00

Last Successful Scan

Scanned2024-02-12T01:50:19+00:00
URL https://theceramicshop.com/robots.txt
Redirect https://www.theceramicshop.com/robots.txt
Redirect Domain www.theceramicshop.com
Redirect Base theceramicshop.com
Domain IPs 72.3.177.104, 72.3.177.107
Redirect IPs 72.3.177.104, 72.3.177.107
Response IP 72.3.177.107
Found Yes
Hash a76ae998db86469cd132c3dbf8fbcd57b72c08275037150d877a3c654868959c
SimHash b23c5d22ca0b

Groups

*

Rule Path
Disallow /store/filtered/

ahrefsbot

Rule Path
Disallow /

*

Rule Path
Disallow /store/filtered/

moget
ichiro

Rule Path
Disallow /

naverbot
yeti

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

wesee

Rule Path
Disallow /

dotbot

Rule Path
Disallow /store/filtered/

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 7

semrushbot-sa

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 7

mojeekbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 7

binglocalsearch

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 7

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5
Back to top

Comments

  • http://help.goo.ne.jp/help/article/853/
  • Info: http://help.naver.com/customer/etc/webDocument02.nhn
  • Info: http://www.youdao.com/help/webmaster/spider/
  • Info: http://www.sogou.com/docs/help/webmasters.htm#07
  • Info: http://www.majestic12.co.uk/projects/dsearch/mj12bot.php
  • http://openlinkprofiler.org/bot
  • http://www.exalead.com/search/webmasterguide
  • http://www.xovibot.net/
  • http://www.wesee.com/bot/
  • http://moz.com/researchtools/ose/dotbot
  • disallowing spidering of PFS urls to prevent the bot
  • spidering in to an eternal abyss
Back to top