Log In Sign Up

espritweb.hexagon.com
robots.txt

Robots Exclusion Standard data for espritweb.hexagon.com

Resource Scan

Scan Details

Site Domain espritweb.hexagon.com
Base Domain hexagon.com
Scan Status Ok
Last Scan2024-11-03T14:56:14+00:00
Next Scan 2024-12-03T14:56:14+00:00

Last Scan

Scanned2024-11-03T14:56:14+00:00
URL https://espritweb.hexagon.com/robots.txt
Domain IPs 20.3.128.197
Response IP 20.3.128.197
Found Yes
Hash 0cec139bd326b7650d287a842ea0cedbca0bab94560139227ad85cb015b170d7
SimHash 621440e247b1

Groups

blp_bbot/0.1

Rule Path
Disallow /

blp_bbot

Rule Path
Disallow /

slurp

Rule Path
Disallow /

spinn3r

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduimagespider

Rule Path
Disallow /

speedy spider

Rule Path
Disallow /

psbot

Rule Path
Disallow /

voilabot

Rule Path
Disallow /

ask jeeves

Rule Path
Disallow /

daumoa

Rule Path
Disallow /

jaxified

Rule Path
Disallow /

yeti

Rule Path
Disallow /

icc-crawler

Rule Path
Disallow /

yesupbot

Rule Path
Disallow /

ezooms/1.0

Rule Path
Disallow /

dotspotsbot

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

willow internet crawler by twotrees

Rule Path
Disallow /

largesmall crawler

Rule Path
Disallow /

spbot

Rule Path
Disallow /

mxbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

influencebot/0.9

Rule Path
Disallow /

kwaclebot

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

msrbot

Rule Path
Disallow /

betabot

Rule Path
Disallow /

yahoofeedseeker

Rule Path
Disallow /

yahoo-newscrawler

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

linkedinbot/1.0 (compatible; mozilla/5.0; jakarta commons-httpclient/3.1 +http://www.linkedin.com)

Rule Path
Disallow /

lycos_spider

Rule Path
Disallow /

yahoomobile/1.0

Rule Path
Disallow /

domaincrawler 1.0

Rule Path
Disallow /

yahoo pipes 1.0

Rule Path
Disallow /

googlebot-image

Rule Path
Disallow /

kscrawler

Rule Path
Disallow /

synapse

Rule Path
Disallow /

yandex

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

inagist.com url crawler

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /

lmspider

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

r6_commentreader

Rule Path
Disallow /

r6_feedfetcher

Rule Path
Disallow /

nu_tch-princeton

Rule Path
Disallow /

sheenbot

Rule Path
Disallow /

msr-isrccrawler

Rule Path
Disallow /

abby

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

purebot

Rule Path
Disallow /

discobot

Rule Path
Disallow /

msubot

Rule Path
Disallow /

cyberpatrol sitecat webbot

Rule Path
Disallow /

diribot

Rule Path
Disallow /

envolk

Rule Path
Disallow /

fast enterprise crawler 6

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

jyxobot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

postrank

Rule Path
Disallow /

linkedinbot

Rule Path
Disallow /

hailoobot

Rule Path
Disallow /

agbot

Rule Path
Disallow /

unwindfetchor/1.0

Rule Path
Disallow /

voyager/1.0

Rule Path
Disallow /

huaweisymantecspider

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mozilla/5.0 (compatible; butterfly/1.0; +http://labs.topsy.com/butterfly/) gecko/2009032608 firefox/3.0.8

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

suggybot

Rule Path
Disallow /

lemurwebcrawler

Rule Path
Disallow /

flamingo_searchengine

Rule Path
Disallow /

www.integromedb.org/crawler

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

googlebot

Rule Path
Disallow /ew/
Allow /

bingbot

Rule Path
Disallow /ew/
Allow /

Other Records

Field Value
crawl-delay 30

msnbot

Rule Path
Disallow /
Allow /

*

Rule Path
Disallow /ew/

Other Records

Field Value
crawl-delay 30

archive.org_bot

Rule Path
Disallow /ew/

Other Records

Field Value
crawl-delay 30

heritrix

Rule Path
Disallow /ew/
Allow /

Other Records

Field Value
crawl-delay 30
Back to top

Warnings

  • 2 invalid lines.
  • `request-rate` is not a known field.
  • `visit-time` is not a known field.
Back to top