perhokalastajat.net
robots.txt

Robots Exclusion Standard data for perhokalastajat.net

Resource Scan

Scan Details

Site Domain perhokalastajat.net
Base Domain perhokalastajat.net
Scan Status Ok
Last Scan2024-05-27T07:53:00+00:00
Next Scan 2024-06-26T07:53:00+00:00

Last Scan

Scanned2024-05-27T07:53:00+00:00
URL https://perhokalastajat.net/robots.txt
Domain IPs 5.44.245.39
Response IP 5.44.245.39
Found Yes
Hash 4127786096b63b0113db09ee45f691208cbe8e84e4de164a593d48feb4dca6c1
SimHash 4a3c7441f698

Groups

*

Rule Path
Disallow /

googlebot

Rule Path
Disallow /keskustelu/files/
Disallow /kuvagalleria/albums/

ahrefsbot

Rule Path
Disallow /

msnbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 5

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

exabot

Rule Path
Disallow /

yahoo! slurp

Rule Path
Disallow /

slurp

Rule Path
Disallow /

linkdex

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

alexa [bot]

Rule Path
Disallow /

majestic-12 [bot]

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

psbot [picsearch]

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

bingbot

Rule Path
Disallow /

mediapartners-google

Rule Path
Disallow /

linkdexbot/2.0

Rule Path
Disallow /

linkdexbot/2.1

Rule Path
Disallow /

linkdexbot/2.2

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

istellabot/t.1

Rule Path
Disallow /

ltx71 - (http://ltx71.com/)

Rule Path
Disallow /

twitterbot/1.0

Rule Path
Disallow /

googlebot-image

Rule Path
Disallow /kuvagalleria/albums/userpics/
Disallow /keskustelu/files/
Disallow /keskustelu/download/

msnbot-media/1.1

Rule Path
Disallow /

bingbot/2.0

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /