Log In Sign Up

athletics.wesleyan.edu
robots.txt

Robots Exclusion Standard data for athletics.wesleyan.edu

Resource Scan

Scan Details

Site Domain athletics.wesleyan.edu
Base Domain wesleyan.edu
Scan Status Ok
Last Scan2024-11-14T10:00:42+00:00
Next Scan 2024-11-21T10:00:42+00:00

Last Scan

Scanned2024-11-14T10:00:42+00:00
URL https://athletics.wesleyan.edu/robots.txt
Domain IPs 67.192.28.16
Response IP 67.192.28.16
Found Yes
Hash 3375eeeaa942a23537d188e6a3a5707207fbf6d49aa6ebf8239e4129405d7826
SimHash 6a14d2e2c0b1

Groups

blp_bbot/0.1

Rule Path
Disallow /

blp_bbot

Rule Path
Disallow /

slurp

Rule Path
Disallow /

spinn3r

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduimagespider

Rule Path
Disallow /

speedy spider

Rule Path
Disallow /

psbot

Rule Path
Disallow /

voilabot

Rule Path
Disallow /

ask jeeves

Rule Path
Disallow /

daumoa

Rule Path
Disallow /

jaxified

Rule Path
Disallow /

yeti

Rule Path
Disallow /

icc-crawler

Rule Path
Disallow /

yesupbot

Rule Path
Disallow /

ezooms/1.0

Rule Path
Disallow /

dotspotsbot

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

willow internet crawler by twotrees

Rule Path
Disallow /

largesmall crawler

Rule Path
Disallow /

spbot

Rule Path
Disallow /

mxbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

influencebot/0.9

Rule Path
Disallow /

kwaclebot

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

msrbot

Rule Path
Disallow /

betabot

Rule Path
Disallow /

yahoo-newscrawler

Rule Path
Disallow /

lycos_spider

Rule Path
Disallow /

yahoomobile/1.0

Rule Path
Disallow /

domaincrawler 1.0

Rule Path
Disallow /

yahoo pipes 1.0

Rule Path
Disallow /

kscrawler

Rule Path
Disallow /

synapse

Rule Path
Disallow /

yandex

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /admin/
Disallow /services/
Disallow /*.axd
Allow /

Other Records

Field Value
crawl-delay 2

inagist.com url crawler

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /

lmspider

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

r6_commentreader

Rule Path
Disallow /

r6_feedfetcher

Rule Path
Disallow /

nu_tch-princeton

Rule Path
Disallow /

sheenbot

Rule Path
Disallow /

msr-isrccrawler

Rule Path
Disallow /

abby

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

purebot

Rule Path
Disallow /

discobot

Rule Path
Disallow /

msubot

Rule Path
Disallow /

cyberpatrol sitecat webbot

Rule Path
Disallow /

diribot

Rule Path
Disallow /

envolk

Rule Path
Disallow /

fast enterprise crawler 6

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

jyxobot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

postrank

Rule Path
Disallow /

hailoobot

Rule Path
Disallow /

agbot

Rule Path
Disallow /

unwindfetchor/1.0

Rule Path
Disallow /

voyager/1.0

Rule Path
Disallow /

huaweisymantecspider

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /admin/
Disallow /services/
Disallow /*.axd
Disallow /*print%3Dtrue*
Disallow /common/

Other Records

Field Value
crawl-delay 2

ahrefssiteaudit

Rule Path
Disallow /admin/
Disallow /services/
Disallow /*.axd
Disallow /*print%3Dtrue*
Disallow /common/

Other Records

Field Value
crawl-delay 2

mozilla/5.0 (compatible; butterfly/1.0; +http://labs.topsy.com/butterfly/) gecko/2009032608 firefox/3.0.8

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

suggybot

Rule Path
Disallow /

lemurwebcrawler

Rule Path
Disallow /

flamingo_searchengine

Rule Path
Disallow /

www.integromedb.org/crawler

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

siteimprovebot-crawler

Rule Path
Disallow /admin/
Disallow /services/
Disallow /*.axd
Disallow /*print%3Dtrue*
Allow /

powermapper

Rule Path
Disallow /admin/
Disallow /services/
Disallow /*.axd
Disallow /*print%3Dtrue*
Disallow /common/
Allow /

googlebot

Rule Path
Disallow /admin/
Disallow /services/
Disallow /*.axd
Disallow /*print%3Dtrue*
Disallow /common/
Allow /services/podcast_rss.ashx
Allow /

bingbot

Rule Path
Disallow /admin/
Disallow /images/
Disallow /admin/
Disallow /common/
Disallow /editor/
Disallow /services/
Disallow /site/
Disallow /*.js$
Disallow /*.css$
Disallow /*.jpg$
Disallow /*.gif$
Disallow /*.axd
Allow /documents/
Allow /

Other Records

Field Value
crawl-delay 30

msnbot

Rule Path
Disallow /admin/
Disallow /images/
Disallow /admin/
Disallow /common/
Disallow /editor/
Disallow /services/
Disallow /site/
Disallow /*.js$
Disallow /*.css$
Disallow /*.jpg$
Disallow /*.gif$
Disallow /*.axd
Allow /documents/
Allow /

Other Records

Field Value
crawl-delay 30

ltx71

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /admin/
Disallow /services/
Disallow /*.axd
Allow /

*

Rule Path
Disallow /common/
Disallow /images/
Disallow /documents/
Disallow /admin/
Disallow /services/
Disallow /site/
Disallow /hidden/
Disallow /*.js$
Disallow /*.css$
Disallow /*.jpg$
Disallow /*.gif$
Disallow /*.axd
Disallow /*print%3Dtrue*
Allow /

Other Records

Field Value
crawl-delay 30

heritrix

Rule Path
Disallow /admin/
Disallow /images/
Disallow /documents/
Disallow /admin/
Disallow /common/
Disallow /editor/
Disallow /services/
Disallow /site/
Disallow /*.axd
Allow /

Other Records

Field Value
crawl-delay 5

twitterbot/1.0

Rule Path
Allow /

mozilla/4.0+(compatible;+t-h-u-n-d-e-r-s-t-o-n-e)

Rule Path
Disallow /admin/
Disallow /services/
Disallow /site/
Disallow /common/
Disallow /*.axd
Allow /

Other Records

Field Value
crawl-delay 2

swiftbot

Rule Path
Disallow /admin/
Disallow /services/
Disallow /site/
Disallow /*.axd
Allow /

Other Records

Field Value
crawl-delay 3

gsa-crawler

Rule Path
Disallow /admin/
Disallow /common/
Disallow /services/
Disallow /*.axd
Allow /

Other Records

Field Value
crawl-delay 2
Back to top

Warnings

  • 2 invalid lines.
Back to top