referatwork.ru
robots.txt

Robots Exclusion Standard data for referatwork.ru

Resource Scan

Scan Details

Site Domain referatwork.ru
Base Domain referatwork.ru
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-11-11T00:15:41+00:00
Next Scan 2024-12-11T00:15:41+00:00

Last Successful Scan

Scanned2024-10-12T23:23:39+00:00
URL http://referatwork.ru/robots.txt
Domain IPs 185.221.152.233
Response IP 185.221.152.233
Found Yes
Hash 61089df4aa372e0952540f646d1101d06aca37eb2ba740455baa677a73a83b90
SimHash 47450357c5b5

Groups

yandex

Rule Path
Disallow /temporary/
Allow /refs/source/*
Disallow /refwork_agency/
Disallow /js/
Disallow /upload_files/
Disallow /user/
Disallow /terms/

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Disallow /temporary/
Disallow /refwork_agency/
Disallow /upload_files/
Disallow /user/

Other Records

Field Value
sitemap https://referatwork.ru/sitemap.xml
sitemap https://referatwork.ru/sitemap.xml

Warnings

  • `host` is not a known field.