ourbob.com
robots.txt

Robots Exclusion Standard data for ourbob.com

Resource Scan

Scan Details

Site Domain ourbob.com
Base Domain ourbob.com
Scan Status Ok
Last Scan2024-07-03T04:52:11+00:00
Next Scan 2024-07-10T04:52:11+00:00

Last Scan

Scanned2024-07-03T04:52:11+00:00
URL https://www.ourbob.com/robots.txt
Domain IPs 108.128.9.71, 52.212.189.176
Response IP 52.212.189.176
Found Yes
Hash 5ac9553ac53a27b11abfa0b2fc4d5f17ccdd1e7d34cd31b288fa916879746825
SimHash 4f6fc8e0ee18

Groups

sirdatabot

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

seznambot/3.2

Rule Path
Disallow /

bidswitchbot/1.0

Rule Path
Disallow /

jooblebot

Rule Path
Allow /feeds/
Disallow /

coccocbot-web

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

buck/2.2

Rule Path
Disallow /

weborama-fetcher

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

uipbot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

qwant-news

Rule Path
Disallow /

mauibot (crawler.feedback+wc@gmail.com)

Rule Path
Disallow /

*

Rule Path
Disallow

Other Records

Field Value
crawl-delay 5

Comments

  • Block MegaIndex.ru