Log In Sign Up

earthstation1.com
robots.txt

Robots Exclusion Standard data for earthstation1.com

Resource Scan

Scan Details

Site Domain earthstation1.com
Base Domain earthstation1.com
Scan Status Ok
Last Scan2026-01-29T00:14:34+00:00
Next Scan 2026-02-05T00:14:34+00:00

Last Scan

Scanned2026-01-29T00:14:34+00:00
URL https://earthstation1.com/robots.txt
Domain IPs 20.127.197.157
Response IP 20.127.197.157
Found Yes
Hash f707a4690283cf3b77edb6dba19951f647d2f21f506ba04162382cc135996372
SimHash 9004c2c3ef77

Groups

*

Rule Path
Disallow /mysite/pgs/

*

Rule Path
Disallow /pgs/

freefind

Rule Path
Disallow /mysite/pgs/

freefind

Rule Path
Disallow /pgs/

ahrefsbot

Rule Path
Disallow /

ahrefsbot/5.2

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

whizbang

Rule Path
Disallow /

miixpc

Rule Path
Disallow /

mfc_tear

Rule Path
Disallow /

diibot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

webzip

Rule Path
Disallow /

webzip/4.0 (http://www.spidersoft.com)

Rule Path
Disallow /

webzip/4.1 (http://www.spidersoft.com)

Rule Path
Disallow /

internetseer.com

Rule Path
Disallow /

teleport pro/1.29.1590

Rule Path
Disallow /

teleport pro/1.29.1616

Rule Path
Disallow /

teleport pro/1.29.1718

Rule Path
Disallow /

wget/1.6

Rule Path
Disallow /

wget

Rule Path
Disallow /

webbandit

Rule Path
Disallow /

webster pro v2.9 win32

Rule Path
Disallow /

msproxy/2.0

Rule Path
Disallow /

fastlwspider

Rule Path
Disallow /

surfwalker

Rule Path
Disallow /

getwebpage

Rule Path
Disallow /

lwp-trivial

Rule Path
Disallow /

nicerspro

Rule Path
Disallow /

go!zilla 3.3 (www.gozilla.com)

Rule Path
Disallow /

go!zilla 3.5 (www.gozilla.com)

Rule Path
Disallow /

go!zilla/4.0.0.26

Rule Path
Disallow /

crawler@alexa.com

Rule Path
Disallow /

ecatch/3.0

Rule Path
Disallow /

express webpictures (www.express-soft.com)

Rule Path
Disallow /

filehound 1.2

Rule Path
Disallow /

flashget

Rule Path
Disallow /

getright/3.3.3

Rule Path
Disallow /

getright/4.3

Rule Path
Disallow /

getright/4.5

Rule Path
Disallow /

getright/4.5b3

Rule Path
Disallow /

getright/4.5b7

Rule Path
Disallow /

ia_archiver/1.6

Rule Path
Disallow /

installshield digitalwizard

Rule Path
Disallow /

jobo/@jobo_version@(http://www.matuschek.net/jobo.html)

Rule Path
Disallow /

mass downloader 2.2

Rule Path
Disallow /

metaproducts download express/1.0

Rule Path
Disallow /

mister pix version.dll

Rule Path
Disallow /

mozilla/4.0 (compatible; dnloadmage 1.0)

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 5.5; windows nt 5.0; asptear 1.5)

Rule Path
Disallow /

mozilla/4.5 (compatible; httrack 3.0x; windows 98)

Rule Path
Disallow /

mozilla/4.5 (compatible; httrack 3.16; windows 98)

Rule Path
Disallow /

netants/1.23

Rule Path
Disallow /

offline explorer 1.*

Rule Path
Disallow /

pavuk/0.9pl29b i686-pc-linux-gnu

Rule Path
Disallow /

python-urllib/2.0a1

Rule Path
Disallow /

python-urllib/2.1.3

Rule Path
Disallow /

python-urllib/2.2.1

Rule Path
Disallow /

realdownload/4.0.0.42

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

sq webscanner

Rule Path
Disallow /

stamina/1.4

Rule Path
Disallow /

utilmind httpget

Rule Path
Disallow /

webcopier v2.5

Rule Path
Disallow /

webcopier v2.7a

Rule Path
Disallow /

webreaper v9 - www.webreaper.net

Rule Path
Disallow /

webreaper vx.x - www.webreaper.net

Rule Path
Disallow /

webreaper v9.8 - www.webreaper.net

Rule Path
Disallow /

wwwoffle/2.x

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

emailwolf

Rule Path
Disallow /

emailcollector

Rule Path
Disallow /

extractorpro

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

cherrypicker

Rule Path
Disallow /

webemailextractor

Rule Path
Disallow /

bullseye

Rule Path
Disallow /
Back to top

Comments

  • --------------------------------------------------------------------------------------------
  • This is how FreeFind told me to write my FreeFind Exclusionary script
  • This is how I've changed the FreeFind Exclusionary script
  • --------------------------------------------------------------------------------------------
  • Web bots:
  • Above added M 12/4/17
  • --------------------------------------------------------------------------------------------
  • Email Bots:
Back to top