webmfiles.org
robots.txt

Robots Exclusion Standard data for webmfiles.org

Resource Scan

Scan Details

Site Domain webmfiles.org
Base Domain webmfiles.org
Scan Status Ok
Last Scan2024-09-12T13:29:01+00:00
Next Scan 2024-10-12T13:29:01+00:00

Last Scan

Scanned2024-09-12T13:29:01+00:00
URL https://webmfiles.org/robots.txt
Domain IPs 5.253.86.18
Response IP 5.253.86.18
Found Yes
Hash 3750668b54290a04499fdbccd9ef0879fca7680cbdf8362ae7f266ddf3d3c4b8
SimHash fb187142812b

Groups

*

Rule Path
Disallow /categories.php
Disallow /brands.php
Disallow /reviews.php
Disallow /review/
Disallow /members/
Disallow /admin/
Disallow /images/
Disallow /merchant/
Disallow /infarbe/
Disallow /margi*/
Disallow /video/
Disallow /material/
Disallow /goto.php
Disallow /sperrmich.php
Disallow /search.php
Disallow /jump.php

ia_archiver

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

domaincrawler/1.0

Rule Path
Disallow /

domaincrawler

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

linguee

Rule Path
Disallow /

exabot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

discobot

Rule Path
Disallow /

yottaamonitor

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

docomo

Rule Path
Disallow /

bspider

Rule Path
Disallow /

psbot

Rule Path
Disallow /

ng

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandex bot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

spbot

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

bixolabs

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

purebot

Rule Path
Disallow /

yeti/1.0

Rule Path
Disallow /

yeti

Rule Path
Disallow /

docomo

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

sogou

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

sbider

Rule Path
Disallow /

geohasher

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

mlbot

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

flightdeckreportsbot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

nerdbynature.bot

Rule Path
Disallow /

yottaamonitor

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mlbot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

lmspider

Rule Path
Disallow /

speedy

Rule Path
Disallow /

lmspider

Rule Path
Disallow /

yodaobot

Rule Path
Disallow /

aboundexbot

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

crystalsemanticsbot

Rule Path
Disallow /

linkdex.com

Rule Path
Disallow /

nextgensearchbot

Rule Path
Disallow /

r6_commentreader

Rule Path
Disallow /

motor

Rule Path
Disallow /

my nutch spider

Rule Path
Disallow /

swebot

Rule Path
Disallow /

Warnings

  • 4 invalid lines.