Log In Sign Up

mypetwarehouse.com.au
robots.txt

Robots Exclusion Standard data for mypetwarehouse.com.au

Resource Scan

Scan Details

Site Domain mypetwarehouse.com.au
Base Domain mypetwarehouse.com.au
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't establish SSL connection.
Last Scan2024-04-21T06:23:00+00:00
Next Scan 2024-06-20T06:23:00+00:00

Last Successful Scan

Scanned2024-02-15T02:50:01+00:00
URL https://mypetwarehouse.com.au/robots.txt
Redirect https://www.mypetwarehouse.com.au/robots.txt
Redirect Domain www.mypetwarehouse.com.au
Redirect Base mypetwarehouse.com.au
Domain IPs 72.3.177.107
Redirect IPs 72.3.177.104, 72.3.177.107
Response IP 72.3.177.107
Found Yes
Hash 205ccf9852f9d7f9cad2e08c80713f903f09269ce507e04401fd09bb748d8172
SimHash a27d5d43d209

Groups

yandex

Rule Path
Disallow /

googlebot

Rule Path
Disallow

adsbot-google

Rule Path
Disallow

googlebot-image

Rule Path
Disallow

ahrefsbot

Rule Path
Disallow /

*

Rule Path
Disallow /store/filtered/

moget
ichiro

Rule Path
Disallow /

naverbot
yeti

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

wesee

Rule Path
Disallow /

dotbot

Rule Path
Disallow /store/filtered/

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 7

semrushbot-sa

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 7

mojeekbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 7

binglocalsearch

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 7

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5
Back to top

Other Records

Field Value
sitemap https://www.mypetwarehouse.com.au/sitemap.xml
sitemap https://www.mypetwarehouse.com.au/sitemap_new.xml
sitemap https://www.mypetwarehouse.com.au/sitemap_http.xml
Back to top

Comments

  • http://help.goo.ne.jp/help/article/853/
  • Info: http://help.naver.com/customer/etc/webDocument02.nhn
  • Info: http://www.youdao.com/help/webmaster/spider/
  • Info: http://www.sogou.com/docs/help/webmasters.htm#07
  • Info: http://www.majestic12.co.uk/projects/dsearch/mj12bot.php
  • http://openlinkprofiler.org/bot
  • http://www.exalead.com/search/webmasterguide
  • http://www.xovibot.net/
  • http://www.wesee.com/bot/
  • http://moz.com/researchtools/ose/dotbot
  • disallowing spidering of PFS urls to prevent the bot
  • spidering in to an eternal abyss
Back to top