keskustelu.kauppalehti.fi
robots.txt
Robots Exclusion Standard data for keskustelu.kauppalehti.fi
Resource Scan
Scan Details
| Site Domain | keskustelu.kauppalehti.fi |
| Base Domain | kauppalehti.fi |
| Scan Status | Ok |
| Last Scan | 2025-02-23T18:42:39+00:00 |
| Next Scan | 2025-03-25T18:42:39+00:00 |
Last Scan
| Scanned | 2025-02-23T18:42:39+00:00 |
| URL | https://keskustelu.kauppalehti.fi/robots.txt |
| Domain IPs | 108.157.194.116, 108.157.194.21, 108.157.194.30, 108.157.194.7 |
| Response IP | 18.165.122.129 |
| Found | Yes |
| Hash | f863d02955612ae0828c5ce172c8c466d686c544435c08c68799c989f67dd69c |
| SimHash | 7296d940c033 |
Groups
sogou spider
baiduspider
baiduspider-video
baiduspider-image
youdaobot
moget
ichiro
ahrefsbot
chatgpt-user
ccbot
gptbot
google-extended
| Rule | Path |
|---|---|
| Disallow | / |
yandex
No rules defined. All paths allowed.
Other Records
| Field | Value | Comment |
|---|---|---|
| crawl-delay | 40 | specifies a 40 second delay between crawls |
msnbot-media
No rules defined. All paths allowed.
Other Records
| Field | Value | Comment |
|---|---|---|
| crawl-delay | 40 | specifies a 40 second delay between crawls |
bingbot
No rules defined. All paths allowed.
Other Records
| Field | Value | Comment |
|---|---|---|
| crawl-delay | 40 | specifies a 40 second delay between crawls |
ahrefsbot
No rules defined. All paths allowed.
Other Records
| Field | Value | Comment |
|---|---|---|
| crawl-delay | 40 | specifies a 40 second delay between crawls |
trendictionbot
No rules defined. All paths allowed.
Other Records
| Field | Value | Comment |
|---|---|---|
| crawl-delay | 40 | specifies a 40 second delay between crawls |
*
No rules defined. All paths allowed.
Other Records
| Field | Value | Comment |
|---|---|---|
| crawl-delay | 40 | specifies a 40 second delay between crawls |