noel.org
robots.txt

Robots Exclusion Standard data for noel.org

Resource Scan

Scan Details

Site Domain noel.org
Base Domain noel.org
Scan Status Ok
Last Scan2024-11-14T08:57:42+00:00
Next Scan 2024-11-21T08:57:42+00:00

Last Scan

Scanned2024-11-14T08:57:42+00:00
URL https://noel.org/robots.txt
Domain IPs 176.31.37.233
Response IP 176.31.37.233
Found Yes
Hash 7689832a54871dd4add5605ad2a0b8ecaabf1e159182867b5af891e3e5215613
SimHash aa64d3a2d190

Groups

google
googlebot
googlebot-news
googlebot-image
googlebot-video
googlebot-mobile
mediapartners-google
mediapartners
adsbot-google
adsbot-google-mobile-apps
bingbot
msnbot
msnbot-media
adidxbot
bingpreview
slurp
orangebot
duckduckbot
facebookexternalhit
pinterest
twitterbot

Rule Path
Disallow
Disallow /weepeggle

yandex

Rule Path
Disallow /

moget
ichiro

Rule Path
Disallow /

naverbot
yeti

Rule Path
Disallow /

baiduspider
baiduspider-video
baiduspider-image

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

domaincrawler

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

Comments

  • ------------------------------------------------------------------------------
  • Whitelist
  • ------------------------------------------------------------------------------
  • Googlebot
  • Bing
  • Yahoo
  • Orange
  • DuckDuckGo
  • Social network
  • ------------------------------------------------------------------------------
  • Blacklist
  • ------------------------------------------------------------------------------
  • Yandex
  • Goo
  • Naver
  • Baidu
  • SoGou
  • Barkrowler
  • Youdao
  • GrapeshotCrawler
  • Majestic-12
  • Exabot
  • Alexa
  • DomainCrawler.com