Log In Sign Up

stmary.edu
robots.txt

Robots Exclusion Standard data for stmary.edu

Resource Scan

Scan Details

Site Domain stmary.edu
Base Domain stmary.edu
Scan Status Ok
Last Scan2024-09-20T02:16:54+00:00
Next Scan 2024-10-20T02:16:54+00:00

Last Scan

Scanned2024-09-20T02:16:54+00:00
URL https://stmary.edu/robots.txt
Domain IPs 209.41.65.36
Response IP 209.41.65.36
Found Yes
Hash e16ffa7ddcb9be2ade7c51be9ae62ce58aa9c68e1d6d69af5b0dbc21f0198b0e
SimHash 6214d2e242a1

Groups

mauibot

Rule Path
Disallow /

blp_bbot/0.1

Rule Path
Disallow /

blp_bbot

Rule Path
Disallow /

slurp

Rule Path
Disallow /

spinn3r

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduimagespider

Rule Path
Disallow /

speedy spider

Rule Path
Disallow /

psbot

Rule Path
Disallow /

voilabot

Rule Path
Disallow /

ask jeeves

Rule Path
Disallow /

daumoa

Rule Path
Disallow /

jaxified

Rule Path
Disallow /

yeti

Rule Path
Disallow /

icc-crawler

Rule Path
Disallow /

yesupbot

Rule Path
Disallow /

ezooms/1.0

Rule Path
Disallow /

dotspotsbot

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

willow internet crawler by twotrees

Rule Path
Disallow /

largesmall crawler

Rule Path
Disallow /

spbot

Rule Path
Disallow /

mxbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

influencebot/0.9

Rule Path
Disallow /

kwaclebot

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

msrbot

Rule Path
Disallow /

betabot

Rule Path
Disallow /

yahoo-newscrawler

Rule Path
Disallow /

lycos_spider

Rule Path
Disallow /

yahoomobile/1.0

Rule Path
Disallow /

domaincrawler 1.0

Rule Path
Disallow /

yahoo pipes 1.0

Rule Path
Disallow /

kscrawler

Rule Path
Disallow /

synapse

Rule Path
Disallow /

yandex

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

inagist.com url crawler

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /

lmspider

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

r6_commentreader

Rule Path
Disallow /

r6_feedfetcher

Rule Path
Disallow /

nu_tch-princeton

Rule Path
Disallow /

sheenbot

Rule Path
Disallow /

msr-isrccrawler

Rule Path
Disallow /

abby

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

purebot

Rule Path
Disallow /

discobot

Rule Path
Disallow /

msubot

Rule Path
Disallow /

cyberpatrol sitecat webbot

Rule Path
Disallow /

diribot

Rule Path
Disallow /

envolk

Rule Path
Disallow /

fast enterprise crawler 6

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

jyxobot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

postrank

Rule Path
Disallow /

hailoobot

Rule Path
Disallow /

agbot

Rule Path
Disallow /

unwindfetchor/1.0

Rule Path
Disallow /

voyager/1.0

Rule Path
Disallow /

huaweisymantecspider

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mozilla/5.0 (compatible; butterfly/1.0; +http://labs.topsy.com/butterfly/) gecko/2009032608 firefox/3.0.8

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

suggybot

Rule Path
Disallow /

lemurwebcrawler

Rule Path
Disallow /

flamingo_searchengine

Rule Path
Disallow /

www.integromedb.org/crawler

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

eds exif

Rule Path
Disallow /

googlebot

Rule Path
Disallow /omni-cms/
Disallow /_design-files/
Disallow /_faculty_import_11-5/
Disallow /_qa/
Disallow /_resources/
Disallow /_showcase/
Disallow /_training/
Disallow /_admissions-nav-demo/
Disallow /_archived/
Disallow /d/
Disallow /images/
Disallow /include-test/
Disallow /navigation-demo/
Disallow /search/
Disallow /usm-testing/
Disallow /usm_testing/
Allow /

facebookexternalhit/1.1

Rule Path
Disallow /omni-cms/
Disallow /_design-files/
Disallow /_faculty_import_11-5/
Disallow /_qa/
Disallow /_resources/
Disallow /_showcase/
Disallow /_training/
Disallow /_admissions-nav-demo/
Disallow /_archived/
Disallow /d/
Disallow /images/
Disallow /include-test/
Disallow /navigation-demo/
Disallow /search/
Disallow /usm-testing/
Disallow /usm_testing/
Allow /

facebot

Rule Path
Disallow /omni-cms/
Disallow /_design-files/
Disallow /_faculty_import_11-5/
Disallow /_qa/
Disallow /_resources/
Disallow /_showcase/
Disallow /_training/
Disallow /_admissions-nav-demo/
Disallow /_archived/
Disallow /d/
Disallow /images/
Disallow /include-test/
Disallow /navigation-demo/
Disallow /search/
Disallow /usm-testing/
Disallow /usm_testing/
Allow /

bingbot

Rule Path
Disallow /omni-cms/
Disallow /_design-files/
Disallow /_faculty_import_11-5/
Disallow /_qa/
Disallow /_resources/
Disallow /_showcase/
Disallow /_training/
Disallow /_admissions-nav-demo/
Disallow /_archived/
Disallow /d/
Disallow /images/
Disallow /include-test/
Disallow /navigation-demo/
Disallow /search/
Disallow /usm-testing/
Disallow /usm_testing/
Disallow /*.jpg$
Disallow /*.gif$
Allow /

Other Records

Field Value
crawl-delay 30

msnbot

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /omni-cms/
Disallow /_design-files/
Disallow /_faculty_import_11-5/
Disallow /_qa/
Disallow /_resources/
Disallow /_showcase/
Disallow /_training/
Disallow /_admissions-nav-demo/
Disallow /_archived/
Disallow /d/
Disallow /images/
Disallow /include-test/
Disallow /navigation-demo/
Disallow /search/
Disallow /usm-testing/
Disallow /usm_testing/
Disallow /*.axd
Allow /

Other Records

Field Value
crawl-delay 60

*

Rule Path
Disallow /omni-cms/
Disallow /_design-files/
Disallow /_faculty_import_11-5/
Disallow /_qa/
Disallow /_resources/
Disallow /_showcase/
Disallow /_training/
Disallow /_admissions-nav-demo/
Disallow /_archived/
Disallow /d/
Disallow /images/
Disallow /include-test/
Disallow /navigation-demo/
Disallow /search/
Disallow /usm-testing/
Disallow /usm_testing/
Disallow /*.js$
Disallow /*.css$
Disallow /*.jpg$
Disallow /*.gif$

Other Records

Field Value
crawl-delay 30

heritrix

Rule Path
Disallow /omni-cms/
Disallow /_design-files/
Disallow /_faculty_import_11-5/
Disallow /_qa/
Disallow /_resources/
Disallow /_showcase/
Disallow /_training/
Disallow /_admissions-nav-demo/
Disallow /_archived/
Disallow /d/
Disallow /images/
Disallow /include-test/
Disallow /navigation-demo/
Disallow /search/
Disallow /usm-testing/
Disallow /usm_testing/
Allow /

Other Records

Field Value
crawl-delay 5

twitterbot/1.0

Rule Path
Allow /

mozilla/4.0+(compatible;+t-h-u-n-d-e-r-s-t-o-n-e)

Rule Path
Disallow /omni-cms/
Disallow /_design-files/
Disallow /_faculty_import_11-5/
Disallow /_qa/
Disallow /_resources/
Disallow /_showcase/
Disallow /_training/
Disallow /_admissions-nav-demo/
Disallow /_archived/
Disallow /d/
Disallow /images/
Disallow /include-test/
Disallow /navigation-demo/
Disallow /search/
Disallow /usm-testing/
Disallow /usm_testing/
Allow /

Other Records

Field Value
crawl-delay 2

swiftbot

Rule Path
Disallow /omni-cms/
Disallow /_design-files/
Disallow /_faculty_import_11-5/
Disallow /_qa/
Disallow /_resources/
Disallow /_showcase/
Disallow /_training/
Disallow /_admissions-nav-demo/
Disallow /_archived/
Disallow /d/
Disallow /images/
Disallow /include-test/
Disallow /navigation-demo/
Disallow /search/
Disallow /usm-testing/
Disallow /usm_testing/
Allow /

Other Records

Field Value
crawl-delay 3
Back to top

Warnings

  • 2 invalid lines.
  • `request-rate` is not a known field.
  • `visit-time` is not a known field.
Back to top