raillab.jp
robots.txt

Robots Exclusion Standard data for raillab.jp

Resource Scan

Scan Details

Site Domain raillab.jp
Base Domain raillab.jp
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-10-31T14:44:26+00:00
Next Scan 2024-11-07T14:44:26+00:00

Last Successful Scan

Scanned2024-10-23T14:38:32+00:00
URL https://raillab.jp/robots.txt
Domain IPs 13.35.238.105, 13.35.238.121, 13.35.238.16, 13.35.238.29
Response IP 13.35.238.121
Found Yes
Hash 2610e07b91e65f2012b8afed64c2cf6ad4a7c520bbff25f339b3091091667588
SimHash 42399e3af591

Groups

*

Rule Path
Disallow /auth/
Disallow /api/
Disallow /utility/
Disallow /news/list?f_cat=
Disallow /car/*/photos/list?photo_type=
Disallow /carset/*/photos/list?photo_type=
Disallow /company/*/photos/list?photo_type=
Disallow /member/*/photo?photo_type=
Disallow /transport/*/photos/list?photo_type=
Disallow /station/*/photos/list?photo_type=
Disallow /terminal/*/photos/list?photo_type=
Disallow /train/*/photos/list?photo_type=
Disallow /car/*/records/list?sort=
Disallow /carset/*/records/list?sort=
Disallow /member/*/record/list?sort=
Disallow /transport/*/records/list?sort=
Disallow /station/*/records/list?sort=
Disallow /terminal/*/records/list?sort=
Disallow /train/*/records/list?sort=

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

blexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

grapeshot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

googlebot-news

Rule Path
Disallow /
Allow /news*