thorlabs.com
robots.txt
Robots Exclusion Standard data for thorlabs.com
Resource Scan
Scan Details
| Site Domain | thorlabs.com |
| Base Domain | thorlabs.com |
| Scan Status | Ok |
| Last Scan | 2024-06-07T19:11:28+00:00 |
| Next Scan | 2024-07-07T19:11:28+00:00 |
Last Scan
| Scanned | 2024-06-07T19:11:28+00:00 |
| URL | https://thorlabs.com/robots.txt |
| Redirect | https://www.thorlabs.com/robots.txt |
| Redirect Domain | www.thorlabs.com |
| Redirect Base | thorlabs.com |
| Domain IPs | 199.83.132.153, 199.83.134.153 |
| Redirect IPs | 103.28.249.153 |
| Response IP | 103.28.249.153 |
| Found | Yes |
| Hash | f38a9cf4a447ad766fa542a908eee58d800167fcfee943b0fe226f20f541339d |
| SimHash | 0204e2c1a0ca |
Groups
zoomspider
| Product | Comment |
|---|---|
| zoomspider | http://www.wrensoft.com/zoom/support/useragent.html |
| Rule | Path |
|---|---|
| Disallow | / |
icc-crawler
| Product | Comment |
|---|---|
| icc-crawler | http://www.nict.go.jp/en/univ-com/plan/crawl.html |
| Rule | Path |
|---|---|
| Disallow | / |
yeti
| Product | Comment |
|---|---|
| yeti | http://www.botopedia.org/user-agent-list/search-bots/item/340-yeti-naverbot |
| Rule | Path |
|---|---|
| Disallow | / |
changedetection
| Product | Comment |
|---|---|
| changedetection | http://www.changedetection.com/bot.html |
| Rule | Path |
|---|---|
| Disallow | / |
job roboter spider
| Product | Comment |
|---|---|
| job roboter spider | http://www.webintegration.at/jobroboter_suchmaschine |
| Rule | Path |
|---|---|
| Disallow | / |
xenu link sleuth
| Product | Comment |
|---|---|
| xenu link sleuth | http://home.snafu.de/tilman/xenulink.html |
| Rule | Path |
|---|---|
| Disallow | / |
facebookexternalhit
| Product | Comment |
|---|---|
| facebookexternalhit | http://www.facebook.com/externalhit_uatext.php |
| Rule | Path |
|---|---|
| Disallow | / |
mj12bot
| Product | Comment |
|---|---|
| mj12bot | http://www.majestic12.co.uk/projects/dsearch/mj12bot.php |
| Rule | Path |
|---|---|
| Disallow | / |
yisouspider
| Product | Comment |
|---|---|
| yisouspider | (BAD)http://user-agents.me/crawler/yisouspider |
| Rule | Path |
|---|---|
| Disallow | / |
magpie-crawler
| Product | Comment |
|---|---|
| magpie-crawler | https://www.brandwatch.com/how-it-works/ |
| Rule | Path |
|---|---|
| Disallow | / |
alexabot
| Product | Comment |
|---|---|
| alexabot | https://support.alexa.com/hc/en-us/articles/200462340-Certification-Crawler-Information |
| Rule | Path |
|---|---|
| Disallow | / |
slurp
| Product | Comment |
|---|---|
| slurp | http://www.useragentstring.com/Yahoo!%20Slurp_id_75.php |
| Rule | Path |
|---|---|
| Disallow | / |
yandexbot
| Product | Comment |
|---|---|
| yandexbot | http://help.yandex.com/search/robots/agent.xml |
| Rule | Path |
|---|---|
| Disallow | / |
sogou spider
| Product | Comment |
|---|---|
| sogou spider | http://www.sogou.com/docs/help/webmasters.htm#07 |
| Rule | Path |
|---|---|
| Disallow | / |
curious george
| Product | Comment |
|---|---|
| curious george | http://www.analyticsseo.com/the-analytics-seo-crawler-curious-george/ |
| Rule | Path |
|---|---|
| Disallow | / |
y!j-asr
| Product | Comment |
|---|---|
| y!j-asr | https://help.yahoo.com/kb/search/SLN22600.html?impressions=true |
| Rule | Path |
|---|---|
| Disallow | / |
y!j-bsc
| Product | Comment |
|---|---|
| y!j-bsc | https://help.yahoo.com/kb/search/SLN22600.html?impressions=true |
| Rule | Path |
|---|---|
| Disallow | / |
daumoa
| Product | Comment |
|---|---|
| daumoa | (BAD)https://www.webmasterworld.com/search_engine_spiders/3895299.htm |
| Rule | Path |
|---|---|
| Disallow | / |
who.is bot
| Product | Comment |
|---|---|
| who.is bot | https://www.webmasterworld.com/search_engine_spiders/4427797.htm |
| Rule | Path |
|---|---|
| Disallow | / |
bingbot
| Product | Comment |
|---|---|
| bingbot | http://www.bing.com/bingbot.htm |
| Rule | Path |
|---|---|
| Disallow | /thorproduct.cfm* |
bingbot
| Product | Comment |
|---|---|
| bingbot | http://www.bing.com/bingbot.htm |
| Rule | Path |
|---|---|
| Disallow | /ThorProduct.cfm* |
*
| Rule | Path |
|---|---|
| Disallow | /honey/ |
| Disallow | /thorcat/ |
| Disallow | /Thorcat/ |
| Disallow | /search/ |
| Disallow | /thorsearch.cfm* |
| Disallow | /advSearch.cfm |
| Disallow | /advSearchDetail.cfm |
| Disallow | /*.dxf$ |
| Disallow | /*.sldrpt$ |
| Disallow | /*.step$ |
| Disallow | /*.vbi$ |
| Disallow | /*.zip$ |
| Disallow | /*.eprt$ |
| Disallow | /*.bak$ |
| Disallow | /*.exe$ |
| Disallow | /images/catalog/ |
| Disallow | /trackClick.cfc |
| Disallow | /NewGroupPage9.cfm?ObjectGroup_ID=5569 |
| Disallow | /Navigation.cfm?Guide_ID=2184 |
| Disallow | /newgrouppage9_pf.cfm* |
| Disallow | /newgrouppage9pf.cfm* |
| Disallow | /cfc/familyPage/priceRequest.cfc* |
| Disallow | /sitemap.cfm* |
| Disallow | /action.cfm* |
| Disallow | /RoHS_cert.cfm* |
| Disallow | /_sd.cfm* |
| Disallow | /AJAX/ |
| Disallow | /CFC/ |
| Disallow | /cfc/ |
| Disallow | /JS/ |
| Disallow | /CFIDE/ |
| Disallow | /JSON/ |
| Disallow | /*?*CurrencySelect=* |
| Disallow | /*?*Language=* |
| Disallow | /*?*isPreview=* |
| Disallow | /*?*ispreview=* |
| Disallow | /*.cfc$ |
| Disallow | /contentEditor/ |
| Disallow | /contenteditor/ |
| Disallow | /rest/library/ |
| Disallow | /_volPricing.cfm |
| Disallow | /_volpricing.cfm |
| Disallow | /RoHS_cert.cfm |
| Disallow | /rohs_cert.cfm |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 30 |
Comments