webry.info
robots.txt

Robots Exclusion Standard data for webry.info

Resource Scan

Scan Details

Site Domain webry.info
Base Domain webry.info
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-05-26T13:31:24+00:00
Next Scan 2024-08-24T13:31:24+00:00

Last Successful Scan

Scanned2024-01-05T13:24:46+00:00
URL https://www.webry.info/robots.txt
Domain IPs 52.84.251.120, 52.84.251.33, 52.84.251.60, 52.84.251.82
Response IP 52.222.236.4
Found Yes
Hash adff72aa093d2afb33bbd14c3db38f61568f783bef2d78e5a619b28762d9754a
SimHash 670fd840c37a

Groups

bingbot

Rule Path
Disallow /

msnbot

Rule Path
Disallow /

y!j-asr/0.1 crawler

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

seokicks-robot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

yandexcalendar

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandexmobilebot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /