Log In Sign Up

technology.matthey.com
robots.txt

Robots Exclusion Standard data for technology.matthey.com

Resource Scan

Scan Details

Site Domain technology.matthey.com
Base Domain matthey.com
Scan Status Ok
Last Scan2025-11-20T11:12:24+00:00
Next Scan 2025-12-20T11:12:24+00:00

Last Scan

Scanned2025-11-20T11:12:24+00:00
URL https://technology.matthey.com/robots.txt
Domain IPs 104.18.8.61, 104.18.9.61
Response IP 104.18.9.61
Found Yes
Hash 8705e0fae6a18d2ebd774cadb2c4fa1d5455e01b9ee48cf4ed61e93c011cbd35
SimHash 25b5cd3aa397

Groups

googlebot

Rule Path
Disallow /admin/
Disallow /alerts/
Disallow /cache/
Disallow /carnet/
Disallow /cart
Disallow /cart/
Disallow /cartregistration/
Disallow /cc/
Disallow /commenting/
Disallow /counterstats/
Disallow /deliver/
Disallow /editcontent/
Disallow /email
Disallow /email/
Disallow /error/
Disallow /export/
Disallow /librarians/uploadbranding
Disallow /licence-admin/
Disallow /managesavedsearches
Disallow /markedlist/
Disallow /mostviewed/
Disallow /offer/
Disallow /openurl/profile
Disallow /orders
Disallow /orders/
Disallow /pricing/
Disallow /profile
Disallow /profile/
Disallow /pubstats/
Disallow /recommendto
Disallow /reference/
Disallow /regactivation
Disallow /regactivation/
Disallow /registration
Disallow /registration/
Disallow /resourcelookup
Disallow /runsavedsearch
Disallow /savedsearches
Disallow /search
Disallow /search/
Disallow /selfservice/
Disallow /session
Disallow /session/
Disallow /statslogredirect
Disallow /statslogwrapper
Disallow /subscribedtitles
Disallow /uploadfile
Disallow /user/
Disallow /workflow/
Disallow /content/contributor
Disallow /content/contributor/
Disallow /content/institution
Disallow /content/institution/

yandexbot

Rule Path
Disallow /admin/
Disallow /alerts/
Disallow /cache/
Disallow /carnet/
Disallow /cart
Disallow /cart/
Disallow /cartregistration/
Disallow /cc/
Disallow /commenting/
Disallow /counterstats/
Disallow /deliver/
Disallow /editcontent/
Disallow /email
Disallow /email/
Disallow /error/
Disallow /export/
Disallow /librarians/uploadbranding
Disallow /licence-admin/
Disallow /managesavedsearches
Disallow /markedlist/
Disallow /mostviewed/
Disallow /offer/
Disallow /openurl/profile
Disallow /orders
Disallow /orders/
Disallow /pricing/
Disallow /profile
Disallow /profile/
Disallow /pubstats/
Disallow /recommendto
Disallow /reference/
Disallow /regactivation
Disallow /regactivation/
Disallow /registration
Disallow /registration/
Disallow /resourcelookup
Disallow /runsavedsearch
Disallow /savedsearches
Disallow /search
Disallow /search/
Disallow /selfservice/
Disallow /session
Disallow /session/
Disallow /statslogredirect
Disallow /statslogwrapper
Disallow /subscribedtitles
Disallow /uploadfile
Disallow /user/
Disallow /workflow/
Disallow /content/contributor
Disallow /content/contributor/
Disallow /content/institution
Disallow /content/institution/

Other Records

Field Value
crawl-delay 4

yandeximages

Rule Path
Disallow /admin/
Disallow /alerts/
Disallow /cache/
Disallow /carnet/
Disallow /cart
Disallow /cart/
Disallow /cartregistration/
Disallow /cc/
Disallow /commenting/
Disallow /counterstats/
Disallow /deliver/
Disallow /editcontent/
Disallow /email
Disallow /email/
Disallow /error/
Disallow /export/
Disallow /librarians/uploadbranding
Disallow /licence-admin/
Disallow /managesavedsearches
Disallow /markedlist/
Disallow /mostviewed/
Disallow /offer/
Disallow /openurl/profile
Disallow /orders
Disallow /orders/
Disallow /pricing/
Disallow /profile
Disallow /profile/
Disallow /pubstats/
Disallow /recommendto
Disallow /reference/
Disallow /regactivation
Disallow /regactivation/
Disallow /registration
Disallow /registration/
Disallow /resourcelookup
Disallow /runsavedsearch
Disallow /savedsearches
Disallow /search
Disallow /search/
Disallow /selfservice/
Disallow /session
Disallow /session/
Disallow /statslogredirect
Disallow /statslogwrapper
Disallow /subscribedtitles
Disallow /uploadfile
Disallow /user/
Disallow /workflow/
Disallow /content/contributor
Disallow /content/contributor/
Disallow /content/institution
Disallow /content/institution/

Other Records

Field Value
crawl-delay 4

bingbot

Rule Path
Disallow /admin/
Disallow /alerts/
Disallow /cache/
Disallow /carnet/
Disallow /cart
Disallow /cart/
Disallow /cartregistration/
Disallow /cc/
Disallow /commenting/
Disallow /counterstats/
Disallow /deliver/
Disallow /editcontent/
Disallow /email
Disallow /email/
Disallow /error/
Disallow /export/
Disallow /librarians/uploadbranding
Disallow /licence-admin/
Disallow /managesavedsearches
Disallow /markedlist/
Disallow /mostviewed/
Disallow /offer/
Disallow /openurl/profile
Disallow /orders
Disallow /orders/
Disallow /pricing/
Disallow /profile
Disallow /profile/
Disallow /pubstats/
Disallow /recommendto
Disallow /reference/
Disallow /regactivation
Disallow /regactivation/
Disallow /registration
Disallow /registration/
Disallow /resourcelookup
Disallow /runsavedsearch
Disallow /savedsearches
Disallow /search
Disallow /search/
Disallow /selfservice/
Disallow /session
Disallow /session/
Disallow /statslogredirect
Disallow /statslogwrapper
Disallow /subscribedtitles
Disallow /uploadfile
Disallow /user/
Disallow /workflow/
Disallow /content/contributor
Disallow /content/contributor/
Disallow /content/institution
Disallow /content/institution/

Other Records

Field Value
crawl-delay 2

baiduspider

Rule Path
Disallow /admin/
Disallow /alerts/
Disallow /cache/
Disallow /carnet/
Disallow /cart
Disallow /cart/
Disallow /cartregistration/
Disallow /cc/
Disallow /commenting/
Disallow /counterstats/
Disallow /deliver/
Disallow /editcontent/
Disallow /email
Disallow /email/
Disallow /error/
Disallow /export/
Disallow /librarians/uploadbranding
Disallow /licence-admin/
Disallow /managesavedsearches
Disallow /markedlist/
Disallow /mostviewed/
Disallow /offer/
Disallow /openurl/profile
Disallow /orders
Disallow /orders/
Disallow /pricing/
Disallow /profile
Disallow /profile/
Disallow /pubstats/
Disallow /recommendto
Disallow /reference/
Disallow /regactivation
Disallow /regactivation/
Disallow /registration
Disallow /registration/
Disallow /resourcelookup
Disallow /runsavedsearch
Disallow /savedsearches
Disallow /search
Disallow /search/
Disallow /selfservice/
Disallow /session
Disallow /session/
Disallow /statslogredirect
Disallow /statslogwrapper
Disallow /subscribedtitles
Disallow /uploadfile
Disallow /user/
Disallow /workflow/
Disallow /content/contributor
Disallow /content/contributor/
Disallow /content/institution
Disallow /content/institution/

Other Records

Field Value
crawl-delay 2

*

Rule Path
Disallow /admin/
Disallow /alerts/
Disallow /cache/
Disallow /carnet/
Disallow /cart
Disallow /cart/
Disallow /cartregistration/
Disallow /cc/
Disallow /commenting/
Disallow /counterstats/
Disallow /deliver/
Disallow /editcontent/
Disallow /email
Disallow /email/
Disallow /error/
Disallow /export/
Disallow /librarians/uploadbranding
Disallow /licence-admin/
Disallow /managesavedsearches
Disallow /markedlist/
Disallow /mostviewed/
Disallow /offer/
Disallow /openurl/profile
Disallow /orders
Disallow /orders/
Disallow /pricing/
Disallow /profile
Disallow /profile/
Disallow /pubstats/
Disallow /recommendto
Disallow /reference/
Disallow /regactivation
Disallow /regactivation/
Disallow /registration
Disallow /registration/
Disallow /resourcelookup
Disallow /runsavedsearch
Disallow /savedsearches
Disallow /search
Disallow /search/
Disallow /selfservice/
Disallow /session
Disallow /session/
Disallow /statslogredirect
Disallow /statslogwrapper
Disallow /subscribedtitles
Disallow /uploadfile
Disallow /user/
Disallow /workflow/
Disallow /content/contributor
Disallow /content/contributor/
Disallow /content/institution
Disallow /content/institution/

Other Records

Field Value
crawl-delay 2
Back to top

Other Records

Field Value
sitemap https://technology.matthey.com/sitemap.xml
sitemap https://technology.matthey.com/sitemap.xml
Back to top

Comments

  • Agent-specific settings for indexing the site.
  • Allow Google for full text crawling
  • Google ignores Crawl-delay
  • Crawl-delay: 2
  • URL prefixes
  • URL wildcards
  • Yandex have two bots so they can work twice as slowly
  • Yandex have two bots so they can work twice as slowly
  • Bing specific configuration
  • Baidu specific configuration
  • Generic settings for indexing the site.
Back to top