Log In Sign Up

thorlabs.us
robots.txt

Robots Exclusion Standard data for thorlabs.us

Resource Scan

Scan Details

Site Domain thorlabs.us
Base Domain thorlabs.us
Scan Status Ok
Last Scan2024-11-03T22:43:42+00:00
Next Scan 2024-12-03T22:43:42+00:00

Last Scan

Scanned2024-11-03T22:43:42+00:00
URL https://thorlabs.us/robots.txt
Redirect https://www.thorlabs.us/robots.txt
Redirect Domain www.thorlabs.us
Redirect Base thorlabs.us
Domain IPs 199.83.129.169, 199.83.131.169
Redirect IPs 199.83.129.169, 199.83.131.169
Response IP 199.83.129.169
Found Yes
Hash f38a9cf4a447ad766fa542a908eee58d800167fcfee943b0fe226f20f541339d
SimHash 0204e2c1a0ca

Groups

zoomspider

Product Comment
zoomspider http://www.wrensoft.com/zoom/support/useragent.html
Rule Path
Disallow /

exabot

Product Comment
exabot http://www.exalead.com/search/webmasterguide
Rule Path
Disallow /

icc-crawler

Product Comment
icc-crawler http://www.nict.go.jp/en/univ-com/plan/crawl.html
Rule Path
Disallow /

ichiro

Product Comment
ichiro http://search.goo.ne.jp/option/use/sub4/sub4-1/
Rule Path
Disallow /

yeti

Product Comment
yeti http://www.botopedia.org/user-agent-list/search-bots/item/340-yeti-naverbot
Rule Path
Disallow /

ssearch_bot

Product Comment
ssearch_bot http://www.semantissimo.de/
Rule Path
Disallow /

changedetection

Product Comment
changedetection http://www.changedetection.com/bot.html
Rule Path
Disallow /

job roboter spider

Product Comment
job roboter spider http://www.webintegration.at/jobroboter_suchmaschine
Rule Path
Disallow /

xenu link sleuth

Product Comment
xenu link sleuth http://home.snafu.de/tilman/xenulink.html
Rule Path
Disallow /

hatena antenna

Product Comment
hatena antenna (BAD)Unknown URL
Rule Path
Disallow /

linkdexbot

Product Comment
linkdexbot http://www.linkdex.com/m/bots/
Rule Path
Disallow /

facebookexternalhit

Product Comment
facebookexternalhit http://www.facebook.com/externalhit_uatext.php
Rule Path
Disallow /

slackbot

Product Comment
slackbot https://api.slack.com/robots
Rule Path
Disallow /

qwantify

Product Comment
qwantify https://www.qwant.com/
Rule Path
Disallow /

feeddemon

Product Comment
feeddemon http://www.feeddemon.com/
Rule Path
Disallow /

dotbot

Product Comment
dotbot https://moz.com/researchtools/ose/dotbot
Rule Path
Disallow /

semrushbot

Product Comment
semrushbot http://www.semrush.com/bot/
Rule Path
Disallow /

seznambot

Product Comment
seznambot http://napoveda.seznam.cz/cz/seznambot/
Rule Path
Disallow /

feedly

Product Comment
feedly http://www.feedly.com/fetcher.html
Rule Path
Disallow /

mj12bot

Product Comment
mj12bot http://www.majestic12.co.uk/projects/dsearch/mj12bot.php
Rule Path
Disallow /

yisouspider

Product Comment
yisouspider (BAD)http://user-agents.me/crawler/yisouspider
Rule Path
Disallow /

magpie-crawler

Product Comment
magpie-crawler https://www.brandwatch.com/how-it-works/
Rule Path
Disallow /

alexabot

Product Comment
alexabot https://support.alexa.com/hc/en-us/articles/200462340-Certification-Crawler-Information
Rule Path
Disallow /

speedy spider

Product Comment
speedy spider (BAD)http://www.entireweb.com/
Rule Path
Disallow /

garlikcrawler

Product Comment
garlikcrawler (BAD)http://www.garlik.com/
Rule Path
Disallow /

ahrefsbot

Product Comment
ahrefsbot https://ahrefs.com/robot
Rule Path
Disallow /

slurp

Product Comment
slurp http://www.useragentstring.com/Yahoo!%20Slurp_id_75.php
Rule Path
Disallow /

yandexbot

Product Comment
yandexbot http://help.yandex.com/search/robots/agent.xml
Rule Path
Disallow /

sogou spider

Product Comment
sogou spider http://www.sogou.com/docs/help/webmasters.htm#07
Rule Path
Disallow /

maxum

Product Comment
maxum http://www.informedusa.com/t/phantom7.15.html
Rule Path
Disallow /

curious george

Product Comment
curious george http://www.analyticsseo.com/the-analytics-seo-crawler-curious-george/
Rule Path
Disallow /

wesee

Product Comment
wesee (BAD)http://www.wesee.com/bot/
Rule Path
Disallow /

rogerbot

Product Comment
rogerbot http://moz.com/help/pro/what-is-rogerbot-
Rule Path
Disallow /

dotbot

Product Comment
dotbot https://moz.com/researchtools/ose/dotbot
Rule Path
Disallow /

y!j-asr

Product Comment
y!j-asr https://help.yahoo.com/kb/search/SLN22600.html?impressions=true
Rule Path
Disallow /

y!j-bsc

Product Comment
y!j-bsc https://help.yahoo.com/kb/search/SLN22600.html?impressions=true
Rule Path
Disallow /

rambot xtreme x.x

Product Comment
rambot xtreme x.x (BAD)Unknown URL
Rule Path
Disallow /

daumoa

Product Comment
daumoa (BAD)https://www.webmasterworld.com/search_engine_spiders/3895299.htm
Rule Path
Disallow /

who.is bot

Product Comment
who.is bot https://www.webmasterworld.com/search_engine_spiders/4427797.htm
Rule Path
Disallow /

psbot

Product Comment
psbot http://www.picsearch.com/bot.html
Rule Path
Disallow /

yacybot

Product Comment
yacybot http://yacy.net/bot.html
Rule Path
Disallow /

nutch

Product Comment
nutch http://nutch.apache.org/bot.html
Rule Path
Disallow /

bubing

Product Comment
bubing http://law.di.unimi.it/software.php#buging
Rule Path
Disallow /

bingbot

Product Comment
bingbot http://www.bing.com/bingbot.htm
Rule Path
Disallow /thorproduct.cfm*

bingbot

Product Comment
bingbot http://www.bing.com/bingbot.htm
Rule Path
Disallow /ThorProduct.cfm*

mappy

Product Comment
mappy http://mappydata.net/#eng
Rule Path
Disallow /

*

Rule Path
Disallow /honey/
Disallow /thorcat/
Disallow /Thorcat/
Disallow /search/
Disallow /thorsearch.cfm*
Disallow /advSearch.cfm
Disallow /advSearchDetail.cfm
Disallow /*.dxf$
Disallow /*.sldrpt$
Disallow /*.step$
Disallow /*.vbi$
Disallow /*.zip$
Disallow /*.eprt$
Disallow /*.bak$
Disallow /*.exe$
Disallow /images/catalog/
Disallow /trackClick.cfc
Disallow /NewGroupPage9.cfm?ObjectGroup_ID=5569
Disallow /Navigation.cfm?Guide_ID=2184
Disallow /newgrouppage9_pf.cfm*
Disallow /newgrouppage9pf.cfm*
Disallow /cfc/familyPage/priceRequest.cfc*
Disallow /sitemap.cfm*
Disallow /action.cfm*
Disallow /RoHS_cert.cfm*
Disallow /_sd.cfm*
Disallow /AJAX/
Disallow /CFC/
Disallow /cfc/
Disallow /JS/
Disallow /CFIDE/
Disallow /JSON/
Disallow /*?*CurrencySelect=*
Disallow /*?*Language=*
Disallow /*?*isPreview=*
Disallow /*?*ispreview=*
Disallow /*.cfc$
Disallow /contentEditor/
Disallow /contenteditor/
Disallow /rest/library/
Disallow /_volPricing.cfm
Disallow /_volpricing.cfm
Disallow /RoHS_cert.cfm
Disallow /rohs_cert.cfm

Other Records

Field Value
crawl-delay 30
Back to top

Comments

  • Updated on 4/9/2018
  • User-agent Disallow List with URL Link
  • User-agent Crawl Delay Disallow
Back to top