Log In Sign Up

oeyo.no
robots.txt

Robots Exclusion Standard data for oeyo.no

Resource Scan

Scan Details

Site Domain oeyo.no
Base Domain oeyo.no
Scan Status Ok
Last Scan2024-05-11T09:46:53+00:00
Next Scan 2024-06-10T09:46:53+00:00

Last Scan

Scanned2024-05-11T09:46:53+00:00
URL https://oeyo.no/robots.txt
Domain IPs 104.21.71.89, 172.67.170.76, 2606:4700:3033::ac43:aa4c, 2606:4700:3035::6815:4759
Response IP 172.67.170.76
Found Yes
Hash e86119077ab09f84be668bceff1670d187ff431751376444e6d4b48774192a8c
SimHash 8754fce25997

Groups

*

Rule Path
Disallow /selection
Disallow /*/selection
Disallow /*/*/selection
Disallow /selection/*
Disallow /*/selection/*
Disallow /*/*/selection/*

adsbot-google

Rule Path
Disallow /selection
Disallow /*/selection
Disallow /*/*/selection
Disallow /selection/*
Disallow /*/selection/*
Disallow /*/*/selection/*

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

ahrefssiteaudit

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

nutch

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

lexxebot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

alligator

Rule Path
Disallow /

sq\ webscanner

Rule Path
Disallow /

allsubmitter

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

anonymous

Rule Path
Disallow /

asterias

Rule Path
Disallow /

autoemailspider

Rule Path
Disallow /

badass

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

becomebot

Rule Path
Disallow /

bitacle

Rule Path
Disallow /

dragonfly

Rule Path
Disallow /

dreampassport

Rule Path
Disallow /

dsurf

Rule Path
Disallow /

ebrowse

Rule Path
Disallow /

ecatch

Rule Path
Disallow /

edgeio

Rule Path
Disallow /

dreampassport

Rule Path
Disallow /

dts agent

Rule Path
Disallow /

fsurf

Rule Path
Disallow /

gaisbot

Rule Path
Disallow /

gamespy_arcade

Rule Path
Disallow /

geniebot

Rule Path
Disallow /

getbot

Rule Path
Disallow /

getright

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

go!zilla

Rule Path
Disallow /

go-ahead-got-it

Rule Path
Disallow /

goforitbot

Rule Path
Disallow /

email\ extractor

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

jeteye

Rule Path
Disallow /

jeteyebot

Rule Path
Disallow /

jobo

Rule Path
Disallow /

joc\ web\ spider

Rule Path
Disallow /

kapere

Rule Path
Disallow /

larbin

Rule Path
Disallow /

leechget

Rule Path
Disallow /

lightningdownload

Rule Path
Disallow /

linkie

Rule Path
Disallow /

mac\ finder

Rule Path
Disallow /

mail\ sweeper

Rule Path
Disallow /

outfoxbot

Rule Path
Disallow /

omniexplorer

Rule Path
Disallow /

p3p

Rule Path
Disallow /

psycheclone

Rule Path
Disallow /

puf

Rule Path
Disallow /

pussycat

Rule Path
Disallow /

puxarapido

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

realdownload

Rule Path
Disallow /

redkernel

Rule Path
Disallow /

relevantnoise

Rule Path
Disallow /

y!tunnelpro

Rule Path
Disallow /

yahooysmcm

Rule Path
Disallow /

zade

Rule Path
Disallow /

zbot

Rule Path
Disallow /

zerxbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://oeyo.no/sitemap.xml
Back to top

Comments

  • Google Asdbots ignores robots.txt unless specifically named
  • Slow down some bots
  • Disallow bad bots
Back to top