Log In Sign Up

merckmillipore.com
robots.txt

Robots Exclusion Standard data for merckmillipore.com

Resource Scan

Scan Details

Site Domain merckmillipore.com
Base Domain merckmillipore.com
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2024-03-10T05:30:38+00:00
Next Scan 2024-06-08T05:30:38+00:00

Last Successful Scan

Scanned2021-12-04T04:12:52+00:00
URL https://merckmillipore.com/robots.txt
Found Yes
Hash 21b8df4c539aead9cd85f29e3924d54ec8ab61b8edc7371c239e631e5a550cd1
SimHash 59939355ccb8

Groups

bingbot

Rule Path
Disallow /*/*/*/*/ViewUserAccount-*
Disallow /*/*/*/*/*/*/*/ViewUserAccount-*
Disallow /*/*/*/*/ViewCart-*
Disallow /*/*/*/*/*/*/*/ViewCart-*
Disallow /is-bin/
Disallow /test/
Disallow /mcsearch/
Disallow /mmsearch/
Disallow /chemdat/
Disallow /DISTR/
Disallow /INTERSHOP/
Disallow /Web-*-Site/
Disallow */search/-
Disallow */page.contact.customer.services.pdp
Disallow /formassembly/2767
Disallow /formassembly/form-builder/4.1.0/css/wforms-layout.css
Allow /********.png$
Allow /********.jpg$
Allow /********.gif$
Allow /********.PNG$
Allow /********.JPG$
Allow /********.GIF$
Allow /********.SWF$
Allow /********.swf$
Allow /********.ico$
Allow /********.ICO$
Allow /********.css
Allow /********.CSS
Allow /********.js
Allow /********.pdf$
Allow /********.PDF$
Allow */ShowDocument-Pronet?
Allow */ShowDocument-File?

Other Records

Field Value
crawl-delay 1

bingpreview

Rule Path
Disallow /
Allow /INTL/en/

googleweblight

Rule Path
Disallow /
Allow /INTL/en/

baiduspider

Rule Path
Disallow /
Disallow */search/-
Disallow */page.contact.customer.services.pdp
Allow /CN/
Allow /HK/
Allow /ID/
Allow /IN/
Allow /JA/
Allow /KR/
Allow /TW/

Other Records

Field Value
crawl-delay 1

yandex

Rule Path
Disallow /
Disallow */search/-
Disallow */page.contact.customer.services.pdp
Allow /RU/
Allow /INTL/
Allow /TR/
Allow /yandex

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Allow */ShowDocument-Pronet?
Allow */ShowDocument-File?
Allow /INTERSHOP/static/WFS/*/Mega-Nav/
Allow /INTERSHOP/static/WFS/*/images/
Allow /INTERSHOP/static/WFS/*/js/
Allow /INTERSHOP/static/WFS/*/bundles/
Allow /INTERSHOP/static/WFS/*/Homepage
Allow /INTERSHOP/static/WFS/*/Target-Banner
Allow /INTERSHOP/static/WFS/*/Freestyle/
Allow /INTERSHOP/static/WFS/*/Right-Nav-Banner
Allow /INTERSHOP/static/WFS/*/Hub-Banner-940x200/
Allow /INTERSHOP/static/WFS/*/Thumbs-100x100/
Allow /INTERSHOP/static/WFS/*/css
Allow /INTERSHOP/static/WFS/*/JP_FreeStyle/
Allow /INTERSHOP/static/WFS/*/Certipur
Allow /INTERSHOP/static/WFS/*/01_JP_Image/
Allow /INTERSHOP/static/WFS/*/JP_Common/
Allow /INTERSHOP/static/WFS/*/system/
Allow /INTERSHOP/static/WFS/*/homepage*/
Allow /INTERSHOP/static/WFS/*/icon
Allow */Textbox-*-188x120/
Allow */BI-Bioscience/
Allow */BM-BioMonitoring/
Allow */LE-Lab-Essentials/
Allow */ViewGdsImage-Start?
Allow /********.png$
Allow /********.jpg$
Allow /********.gif$
Allow /********.PNG$
Allow /********.JPG$
Allow /********.GIF$
Allow /********.SWF$
Allow /********.swf$
Allow /********.ico$
Allow /********.ICO$
Allow /********.css
Allow /********.CSS
Allow /********.js
Allow */productlistpage/sid/
Allow */productpage/sid/
Allow */video-js.css
Allow */Info-Ticker-188x120
Allow */NewsletterLink-Show
Allow /INTERSHOP/static/WFS/*/*/*/*/SWF/BI-Bioscience/
Allow /INTERSHOP/static/WFS/*/*/*/*/homepage*/
Allow /INTERSHOP/static/WFS/*/*/*/*/system/
Allow */ViewIntermediatePage-ViewContentCatalog?
Allow */ViewIntermediatePage-ViewContentPage?
Allow */ViewCountrySelection-RedirectToCountry
Allow */ViewCountrySelection-BrandDefenseRedirect?
Allow */systempage.country.selection.pagelet
Allow */EmergingBiotech/
Allow */ProcessMSDS-Start
Allow */forms/**/
Disallow /Web-*-Site/
Disallow /INTERSHOP/
Disallow /is-bin/
Disallow /test/
Disallow /mcsearch/
Disallow /mmsearch/
Disallow /chemdat/
Disallow /DISTR/
Disallow /PrudsysServer/
Disallow */flashmediaelement.swf
Disallow */search/-
Disallow */page.contact.customer.services.pdp
Disallow /formassembly/2767
Disallow /formassembly/form-builder/4.1.0/css/wforms-layout.css

*

Rule Path
Disallow /Web-US-Site/
Disallow /Web-CA-Site/
Disallow /Web-PR-Site/
Disallow /INTERSHOP/
Disallow /is-bin/
Disallow /test/
Disallow /mcsearch/
Disallow /mmsearch/
Disallow /chemdat/
Disallow /DISTR/
Disallow /PrudsysServer/
Disallow */flashmediaelement.swf
Disallow */search/-
Disallow */page.contact.customer.services.pdp
Disallow */systempage.*
Disallow /formassembly/2767
Disallow /formassembly/form-builder/4.1.0/css/wforms-layout.css

brightedge crawler

Rule Path
Disallow /
Disallow *.pdf
Disallow *.PDF
Disallow *.pdf?*
Disallow *.PDF?*
Allow /INTL/en/*%2Cnav
Allow /INTL/en/product/

converacrawler

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ingenieur

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

exabot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /
Allow /DE/
Allow /INTL/

seznambot

Rule Path
Disallow /
Allow /CZ/
Allow /INTL/

yeti/naverbot

Rule Path
Disallow /
Allow /KR/
Allow /INTL/

daumoa

Rule Path
Disallow /

docomo

Rule Path
Disallow /

rakuten

Rule Path
Disallow /

domaincrawler

Rule Path
Disallow /

blexbot

Rule Path
Disallow /
Allow /INTL/en/

Other Records

Field Value
crawl-delay 3

mappy

Rule Path
Disallow /

jamesbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

hubspot

Rule Path
Disallow /

caliperbot

Rule Path
Disallow /

linkcheck

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

optimizer

Rule Path
Disallow /

match

Rule Path
Disallow /

linkcheck

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap http://www.merckmillipore.com/AE/en/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/AR/es/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/AT/de/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/AU/en/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/BE/fr/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/BR/pt/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/CH/de/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/CL/es/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/CN/zh/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/CO/es/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/CZ/cs/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/DE/de/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/DK/en/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/ES/es/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/FI/en/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/FR/fr/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/GB/en/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/GT/es/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/HK/zh/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/HU/hu/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/ID/id/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/IE/en/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/IN/en/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/INTL/en/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/IT/it/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/JP/ja/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/KR/ko/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/LU/fr/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/MX/es/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/MY/en/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/NL/en/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/NO/en/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/NZ/en/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/PE/es/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/PL/en/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/PT/pt/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/RO/en/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/RU/ru/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/SE/en/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/SG/en/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/SK/sk/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/TH/en/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/TR/tr/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/TW/zh/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/VE/es/sitemap/sitemap.xml
sitemap http://www.merckmillipore.com/ZA/en/sitemap/sitemap.xml
Back to top

Comments

  • Bing this is uploaded 06172019
  • Account pages and legacy no crawl directories
  • forms disallow from BM data 09152017
  • bing logos images video sitemap files other media files in intershop or web-XX-site
  • document allows bing reach removed
  • for GetERP crawl issue 02272018
  • for GetERP crawl issue 02272018
  • Baidu
  • Allow only baidu country directories
  • Yandex crawl delay
  • yandex countries
  • yandex parameter clean
  • Google
  • documents msds coa reach allow removed 01262018
  • Images css files resources
  • logos images video sitemap files other media files in intershop or web-XX-site
  • 11-20-2015 from gwmt blocked resources pdps
  • video sitemap blocked resource 02102016
  • 5-9-2016
  • brand defense and channel redirects
  • 8-10-2016 emerging biotech
  • MSDS 08-22-2016
  • PS webinar forms
  • forms disallow from BM data 09152017
  • Global bot directives
  • forms disallow from BM data 09152017
  • brightedge recommendation crawl
  • added for crawl budgeting 06272017
  • cliqzbot
  • added for crawl budgeting 07272017
  • sitemaps
Back to top

Warnings

  • `clean-param` is not a known field.
Back to top