Log In Sign Up

annieminogue.com
robots.txt

Robots Exclusion Standard data for annieminogue.com

Resource Scan

Scan Details

Site Domain annieminogue.com
Base Domain annieminogue.com
Scan Status Ok
Last Scan2024-10-20T23:16:34+00:00
Next Scan 2024-11-19T23:16:34+00:00

Last Scan

Scanned2024-10-20T23:16:34+00:00
URL https://annieminogue.com/robots.txt
Domain IPs 66.96.146.129
Response IP 66.96.146.129
Found Yes
Hash 8ccf999f8f79b841927ee3b0ab6798005bd2e30d673bd2693b2ad63a13bbab99
SimHash 2e7b121a3b93

Groups

googlebot

Rule Path
Disallow /cgi-bin/
Disallow /poll/
Disallow /images/
Disallow /MP3/
Disallow /asf/

*

Rule Path
Disallow /cgi-bin/

mozilla/3.01 (hotwired-test/0.1)

Rule Path
Disallow /cgi-bin/
Disallow /MP3/
Disallow /asf/

slurp

Rule Path
Disallow /cgi-bin/
Disallow /MP3/
Disallow /asf/

ultraseek

Rule Path
Disallow /cgi-bin/
Disallow /MP3/
Disallow /asf/

smallbear

Rule Path
Disallow /cgi-bin/
Disallow /MP3/
Disallow /asf/

webferret

Rule Path
Disallow

arachnophilia

Rule Path
Disallow

architextspider

Rule Path
Disallow

aspider/0.09

Rule Path
Disallow

auresys/1.0

Rule Path
Disallow

backrub/*.*

Rule Path
Disallow

big brother

Rule Path
Disallow

blackwidow

Rule Path
Disallow

bspider/1.0 libwww-perl/0.40

Rule Path
Disallow

cactvs chemistry spider

Rule Path
Disallow

digimarc cgireader/1.0

Rule Path
Disallow

checkbot/x.xx lwp/5.x

Rule Path
Disallow

cmc/0.01

Rule Path
Disallow

combine/0.0

Rule Path
Disallow

conceptbot/0.3

Rule Path
Disallow

crescent internet toolpak http ole control v.1.0

Rule Path
Disallow

root/0.1

Rule Path
Disallow

cs-hkust-indexserver/1.0

Rule Path
Disallow

cyberspyder/2.1

Rule Path
Disallow

deweb/1.01

Rule Path
Disallow

dragonbot/1.0 libwww/5.0

Rule Path
Disallow

eit-link-verifier-robot/0.2

Rule Path
Disallow

emacs-w3/v[0-9\.]+

Rule Path
Disallow

emailsiphon

Rule Path
Disallow

emc spider

Rule Path
Disallow

explorersearch

Rule Path
Disallow

explorer

Rule Path
Disallow

extractorpro

Rule Path
Disallow

felixide/1.0

Rule Path
Disallow

hazel's ferret web hopper,

Rule Path
Disallow

esirover v1.0

Rule Path
Disallow

fido/0.9 harvest/1.4.pl2

Rule Path
Disallow

hämähäkki/0.2

Rule Path
Disallow

kit-fireball/2.0 libwww/5.0a

Rule Path
Disallow

fish-search-robot

Rule Path
Disallow

mozilla/2.0 (compatible fouineur v2.0; fouineur.9bit.qc.ca)

Rule Path
Disallow

robot du crim 1.0a

Rule Path
Disallow

freecrawl

Rule Path
Disallow

funnelweb-1.0

Rule Path
Disallow

gcreep/1.0

Rule Path
Disallow

geturl.rexx v1.05

Rule Path
Disallow

golem/1.1

Rule Path
Disallow

gromit/1.0

Rule Path
Disallow

gulliver/1.1

Rule Path
Disallow

yes

Rule Path
Disallow

aitcsrobot/1.1

Rule Path
Disallow

wired-digital-newsbot/1.5

Rule Path
Disallow

htdig/3.0b3

Rule Path
Disallow

htmlgobble v2.2

Rule Path
Disallow

no

Rule Path
Disallow

ibm_planetwide,

Rule Path
Disallow

gestalticonoclast/1.0 libwww-fm/2.17

Rule Path
Disallow

ingrid/0.1

Rule Path
Disallow

incywincy/1.0b1

Rule Path
Disallow

informant

Rule Path
Disallow

infoseek robot 1.0

Rule Path
Disallow

infoseek sidewinder

Rule Path
Disallow

infospiders/0.1

Rule Path
Disallow
Back to top

Comments

  • robots, goaway
  • this bans robots from our cgi-bin and othe places they should not be
Back to top

Warnings

  • 4 invalid lines.
Back to top