Log In Sign Up

dou.ua
robots.txt

Robots Exclusion Standard data for dou.ua

Resource Scan

Scan Details

Site Domain dou.ua
Base Domain dou.ua
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-04-01T20:43:28+00:00
Next Scan 2024-06-30T20:43:28+00:00

Last Successful Scan

Scanned2022-11-13T13:51:54+00:00
URL https://dou.ua/robots.txt
Response IP 104.22.52.237, 104.22.53.237, 172.67.21.118
Found Yes
Hash 618a08c7ef14528ca701002437916c152a4a911030a0ebbd23d09248f81a1cab
SimHash cc9d5ee3be82

Groups

*

Rule Path
Disallow /ajax-impressions-track/

yandex

Rule Path
Disallow /users/*/activities/

yandex

Rule Path
Disallow /vacancies/*search%3D*

yandex
slurp

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

vegebot

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

linkpadbot/1.06

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

trovitbot

Rule Path
Disallow /

x-cad-se/1.75

Rule Path
Disallow /

domain re-animator bot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

sputnikbot

Rule Path
Disallow /

sentibot

Rule Path
Disallow /

buckyohare

Rule Path
Disallow /

buck

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

riddler

Rule Path
Disallow /

garlikcrawler

Rule Path
Disallow /

bubing

Rule Path
Disallow /

alphaseobot

Rule Path
Disallow /

extlinksbot

Rule Path
Disallow /

alphaseobot-sa

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

rsiteauditor

Rule Path
Disallow /

mtrobot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

adsbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /
Back to top

Warnings

  • `clean-param` is not a known field.
Back to top