Log In Sign Up

ledlightexpert.com
robots.txt

Robots Exclusion Standard data for ledlightexpert.com

Resource Scan

Scan Details

Site Domain ledlightexpert.com
Base Domain ledlightexpert.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-09-18T22:06:36+00:00
Next Scan 2024-12-17T22:06:36+00:00

Last Successful Scan

Scanned2023-08-03T09:24:08+00:00
URL https://ledlightexpert.com/robots.txt
Domain IPs 104.19.177.121, 104.19.178.121
Response IP 104.19.177.121
Found Yes
Hash 86cda4be06e0e744766e652ec5762622b624335f29d4e0df42fc558975a06288
SimHash 30ce0042cbe3

Groups

*

Rule Path
Disallow /checkout.asp
Disallow /add_cart.asp
Disallow /view_cart.asp
Disallow /add_cart.asp*
Disallow /view_cart.asp*
Disallow /error.asp
Disallow /error.asp*
Disallow /shipquote.asp
Disallow /shiptracking.asp*
Disallow /rssfeed.asp
Disallow /mobile/
Disallow /admin/
Disallow /search_quick.asp
Disallow /message.asp?msg=53
Disallow /myaccount.asp

yandexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

baiduspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

youdaobot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

ia_archiver

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

archive.org_bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

adbeat_bot

Rule Path
Disallow /

addthis.com

Rule Path
Disallow /

alexabot

Rule Path
Disallow

ahrefsbot/3.1

Rule Path
Disallow /

ahrefsbot/5.0

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

ahrefsbot/4.0

Rule Path
Disallow /

blexbot*

Rule Path
Disallow /

backlinkcrawler

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

comodo ssl checker

Rule Path
Disallow /

comodo-certificates-spider

Rule Path
Disallow /

crazywebcrawler-spider

Rule Path
Disallow /

curious** george**

Rule Path
Disallow /

curious george

Rule Path
Disallow /

curious

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

ip-web-crawler.com

Rule Path
Disallow /

ips-agent

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

feedthebot

Rule Path
Disallow /

findlinks

Rule Path
Disallow /

gsitecrawler

Rule Path
Disallow /

goodzer

Rule Path
Disallow /

istellabot*

Rule Path
Disallow /

halebot

Rule Path
Disallow /

linkdex

Rule Path
Disallow /

linkdexbot v2.0

Rule Path
Disallow /

linkdexbot v2.1

Rule Path
Disallow /

linkdexbot v2.2

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

phantomjs

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

spyfu

Rule Path
Disallow /

crawler@wishpond.com

Rule Path
Disallow /

wishpond

Rule Path
Disallow /

xenu link sleuth

Rule Path
Disallow /

xenu's

Rule Path
Disallow /

xenu's link sleuth 1.1c

Rule Path
Disallow /

xenu link sleuth

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

zeus

Rule Path
Disallow /

comodospider/nutch-1.2
mj12bot/v1.4.3
plukkie

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 9

acoon
careerbot
findlinks/2.6
openwebindex/nutch-1.6
openwebindex
sistrixcrawler
sistrix
sistrix crawler
sistrix

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

psbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

kaloogabot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

seznambot
seznambot/1.0
seznambot/1.1
seznambot/3.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

baiduspider-video
baiduspider-image
baiduspider+
baiduspider+(+http://www.baidu.com/search/spider.htm)
baiduspider/2.0;+http://www.baidu.com/search/spider.html
baiduspider/2.0
easouspider
jikespider
sosospider+
sosospider
sosospider/2.0
yisouspider
youdaobot/1.0
youdaobot/1.0
sogou spider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 15

exabot/3.0
exalead
exalead crawler

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

wada.vn
wada.vn vietnamese search
wada.vn vietnamese search/2.1

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 15

naverbot
yeti

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 15

yandex/1.01.001
yandexbot/3.0-mirrordetector
yandeximages/3.0
yandexsomething/1.
yandex.com
yandexbot/3.0
aihitbot
aihitbot/1.0
aihitbot/1.1
mail.ru_bot/2.0
mail.ru
mail.ru
mail.ru_bot/2.0; +http://go.mail.ru/help/robots

Rule Path
Disallow /

moget
ichiro
ichiro
ichiro 3.0
arachmo
zao

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 15
Back to top

Other Records

Field Value
sitemap https://www.ledlightexpert.com/sitemap.xml
Back to top

Comments

  • Please respect robot rules
  • slow certain bots
  • block excessive bots
  • Block/slow International Spiders
  • United Kingdom
  • Germany
  • Sweden
  • Netherlands
  • Czech
  • China
  • France
  • Vietnam
  • South Korea
  • Russia
  • Japan
Back to top

Warnings

  • 8 invalid lines.
Back to top