Log In Sign Up

train-directory.net
robots.txt

Robots Exclusion Standard data for train-directory.net

Resource Scan

Scan Details

Site Domain train-directory.net
Base Domain train-directory.net
Scan Status Ok
Last Scan2024-09-27T02:06:33+00:00
Next Scan 2024-10-04T02:06:33+00:00

Last Scan

Scanned2024-09-27T02:06:33+00:00
URL https://train-directory.net/robots.txt
Domain IPs 160.16.114.43, 2001:e42:102:1533:160:16:114:43
Response IP 160.16.114.43
Found Yes
Hash 2bd12a4842d974d60fbbd86bf60d12b8d8f4f3de6da22ec00dee63405102ab3a
SimHash 467dd770d2fd

Groups

petalbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

*

Rule Path
Disallow /photos/index/page*
Disallow /photos/index/sort*
Disallow /photos/photodays/*
Disallow /photos/company/*
Disallow /photos/trainno/*
Disallow /memberlists/index/*
Disallow /photos/lines/*/sort*
Disallow /photos/lines/*/company*
Disallow /photos/lines/*/photodays*
Disallow /photos/formation/*/company*
Disallow /members/photo/*/page*
Disallow /photos/photopoints/*/sort*
Disallow /photos/formation/*/photodays*
Disallow /photos/photopoints/*/photodays*
Disallow /photos/series/*/photodays*
Disallow /photos/photodays/*/photodays*
Disallow /members/photo/*/photodays*
Disallow /likes/user/*

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

blexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

grapeshot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

seekportbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30
Back to top