Log In Sign Up

shop.ago.ca
robots.txt

Robots Exclusion Standard data for shop.ago.ca

Resource Scan

Scan Details

Site Domain shop.ago.ca
Base Domain ago.ca
Scan Status Ok
Last Scan2024-05-26T15:03:15+00:00
Next Scan 2024-06-09T15:03:15+00:00

Last Scan

Scanned2024-05-26T15:03:15+00:00
URL https://shop.ago.ca/robots.txt
Domain IPs 104.22.42.92, 104.22.43.92, 172.67.25.120, 2606:4700:10::6816:2a5c, 2606:4700:10::6816:2b5c, 2606:4700:10::ac43:1978
Response IP 104.22.42.92
Found Yes
Hash 24c294ff2fdd8f6ef4702161f0fab91ff6b0c1d1e9e260411604c10303fb1212
SimHash b23c5d22ca0b

Groups

ahrefsbot

Rule Path
Disallow /

*

Rule Path
Disallow /store/filtered/

moget
ichiro

Rule Path
Disallow /

naverbot
yeti

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

wesee

Rule Path
Disallow /

dotbot

Rule Path
Disallow /store/filtered/

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 7

semrushbot-sa

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 7

mojeekbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 7

binglocalsearch

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 7

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5
Back to top

Comments

  • http://help.goo.ne.jp/help/article/853/
  • Info: http://help.naver.com/customer/etc/webDocument02.nhn
  • Info: http://www.youdao.com/help/webmaster/spider/
  • Info: http://www.sogou.com/docs/help/webmasters.htm#07
  • Info: http://www.majestic12.co.uk/projects/dsearch/mj12bot.php
  • http://openlinkprofiler.org/bot
  • http://www.exalead.com/search/webmasterguide
  • http://www.xovibot.net/
  • http://www.wesee.com/bot/
  • http://moz.com/researchtools/ose/dotbot
  • disallowing spidering of PFS urls to prevent the bot
  • spidering in to an eternal abyss
Back to top