muslumcu.net
robots.txt

Robots Exclusion Standard data for muslumcu.net

Resource Scan

Scan Details

Site Domain muslumcu.net
Base Domain muslumcu.net
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-08-10T20:12:46+00:00
Next Scan 2025-11-08T20:12:46+00:00

Last Successful Scan

Scanned2025-04-13T20:11:27+00:00
URL https://muslumcu.net/robots.txt
Domain IPs 89.252.182.2
Response IP 89.252.182.2
Found Yes
Hash 83bc8d66881dae6b8df3379d65c410f988963ea42faa03e4e5facfc0f7631f8b
SimHash 3154deb015bb

Groups

slurp

Rule Path
Disallow /

yahoo-mmcrawler

Rule Path
Disallow /

*

Rule Path
Disallow /*.zip$
Disallow /*.Zif$
Disallow /*.ZIP$
Disallow /*.rar$
Disallow /*.RAR$
Disallow /arsiv/dosyalar/
Disallow /modules/bilgi/
Disallow modules.php?name=bilgi&op=preview&id=6

robozilla

Rule Path
Disallow /

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

teoma

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

aipbot

Rule Path
Disallow /

becomebot

Rule Path
Disallow /

psbot

Rule Path
Disallow /

fast

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

voyager

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

backrub/*.*

Rule Path
Disallow /

grub.org

Rule Path
Disallow /

botrighthere

Rule Path
Disallow /

larbin

Rule Path
Disallow /

psbot

Rule Path
Disallow /

walhello appie

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

cherrypicker

Rule Path
Disallow /

emailcollector

Rule Path
Disallow /

webbandit

Rule Path
Disallow /

emailwolf

Rule Path
Disallow /

copyrightcheck

Rule Path
Disallow /

crescent

Rule Path
Disallow /

cuil

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

twiceler-0.9

Rule Path
Disallow /