Log In Sign Up

intellectdiscover.com
robots.txt

Robots Exclusion Standard data for intellectdiscover.com

Resource Scan

Scan Details

Site Domain intellectdiscover.com
Base Domain intellectdiscover.com
Scan Status Ok
Last Scan2025-12-06T12:04:05+00:00
Next Scan 2026-01-05T12:04:05+00:00

Last Scan

Scanned2025-12-06T12:04:05+00:00
URL https://intellectdiscover.com/robots.txt
Domain IPs 104.26.0.106, 104.26.1.106, 172.67.69.110
Response IP 104.26.1.106
Found Yes
Hash c5de5d4654ced78a9eef66be6e25d0705eb9cedb8be2e767ff94fee536799066
SimHash 259dcd18a317

Groups

googlebot

Rule Path
Disallow /admin/
Disallow /alerts/
Disallow /cache/
Disallow /carnet/
Disallow /cart
Disallow /cart/
Disallow /cartregistration/
Disallow /cc/
Disallow /commenting/
Disallow /counterstats/
Disallow /deliver/
Disallow /editcontent/
Disallow /email
Disallow /email/
Disallow /error/
Disallow /export/
Disallow /librarians/uploadbranding
Disallow /licence-admin/
Disallow /managesavedsearches
Disallow /markedlist/
Disallow /mostviewed/
Disallow /offer/
Disallow /openurl/profile
Disallow /orders
Disallow /orders/
Disallow /pricing/
Disallow /profile
Disallow /profile/
Disallow /pubstats/
Disallow /recommendto
Disallow /reference/
Disallow /regactivation
Disallow /regactivation/
Disallow /registration
Disallow /registration/
Disallow /resourcelookup
Disallow /runsavedsearch
Disallow /savedsearches
Disallow /search
Disallow /search/
Disallow /selfservice/
Disallow /session
Disallow /session/
Disallow /statslogredirect
Disallow /statslogwrapper
Disallow /subscribedtitles
Disallow /uploadfile
Disallow /user/
Disallow /workflow/
Disallow /content/contributor
Disallow /content/contributor/
Disallow /content/institution
Disallow /content/institution/

yandexbot

Rule Path
Disallow /admin/
Disallow /alerts/
Disallow /cache/
Disallow /carnet/
Disallow /cart
Disallow /cart/
Disallow /cartregistration/
Disallow /cc/
Disallow /commenting/
Disallow /counterstats/
Disallow /deliver/
Disallow /editcontent/
Disallow /email
Disallow /email/
Disallow /error/
Disallow /export/
Disallow /librarians/uploadbranding
Disallow /licence-admin/
Disallow /managesavedsearches
Disallow /markedlist/
Disallow /mostviewed/
Disallow /offer/
Disallow /openurl/profile
Disallow /orders
Disallow /orders/
Disallow /pricing/
Disallow /profile
Disallow /profile/
Disallow /pubstats/
Disallow /recommendto
Disallow /reference/
Disallow /regactivation
Disallow /regactivation/
Disallow /registration
Disallow /registration/
Disallow /resourcelookup
Disallow /runsavedsearch
Disallow /savedsearches
Disallow /search
Disallow /search/
Disallow /selfservice/
Disallow /session
Disallow /session/
Disallow /statslogredirect
Disallow /statslogwrapper
Disallow /subscribedtitles
Disallow /uploadfile
Disallow /user/
Disallow /workflow/
Disallow /content/contributor
Disallow /content/contributor/
Disallow /content/institution
Disallow /content/institution/

Other Records

Field Value
crawl-delay 10

yandeximages

Rule Path
Disallow /admin/
Disallow /alerts/
Disallow /cache/
Disallow /carnet/
Disallow /cart
Disallow /cart/
Disallow /cartregistration/
Disallow /cc/
Disallow /commenting/
Disallow /counterstats/
Disallow /deliver/
Disallow /editcontent/
Disallow /email
Disallow /email/
Disallow /error/
Disallow /export/
Disallow /librarians/uploadbranding
Disallow /licence-admin/
Disallow /managesavedsearches
Disallow /markedlist/
Disallow /mostviewed/
Disallow /offer/
Disallow /openurl/profile
Disallow /orders
Disallow /orders/
Disallow /pricing/
Disallow /profile
Disallow /profile/
Disallow /pubstats/
Disallow /recommendto
Disallow /reference/
Disallow /regactivation
Disallow /regactivation/
Disallow /registration
Disallow /registration/
Disallow /resourcelookup
Disallow /runsavedsearch
Disallow /savedsearches
Disallow /search
Disallow /search/
Disallow /selfservice/
Disallow /session
Disallow /session/
Disallow /statslogredirect
Disallow /statslogwrapper
Disallow /subscribedtitles
Disallow /uploadfile
Disallow /user/
Disallow /workflow/
Disallow /content/contributor
Disallow /content/contributor/
Disallow /content/institution
Disallow /content/institution/

Other Records

Field Value
crawl-delay 10

bingbot

Rule Path
Disallow /admin/
Disallow /alerts/
Disallow /cache/
Disallow /carnet/
Disallow /cart
Disallow /cart/
Disallow /cartregistration/
Disallow /cc/
Disallow /commenting/
Disallow /counterstats/
Disallow /deliver/
Disallow /editcontent/
Disallow /email
Disallow /email/
Disallow /error/
Disallow /export/
Disallow /librarians/uploadbranding
Disallow /licence-admin/
Disallow /managesavedsearches
Disallow /markedlist/
Disallow /mostviewed/
Disallow /offer/
Disallow /openurl/profile
Disallow /orders
Disallow /orders/
Disallow /pricing/
Disallow /profile
Disallow /profile/
Disallow /pubstats/
Disallow /recommendto
Disallow /reference/
Disallow /regactivation
Disallow /regactivation/
Disallow /registration
Disallow /registration/
Disallow /resourcelookup
Disallow /runsavedsearch
Disallow /savedsearches
Disallow /search
Disallow /search/
Disallow /selfservice/
Disallow /session
Disallow /session/
Disallow /statslogredirect
Disallow /statslogwrapper
Disallow /subscribedtitles
Disallow /uploadfile
Disallow /user/
Disallow /workflow/
Disallow /content/contributor
Disallow /content/contributor/
Disallow /content/institution
Disallow /content/institution/

Other Records

Field Value
crawl-delay 10

baiduspider

Rule Path
Disallow /admin/
Disallow /alerts/
Disallow /cache/
Disallow /carnet/
Disallow /cart
Disallow /cart/
Disallow /cartregistration/
Disallow /cc/
Disallow /commenting/
Disallow /counterstats/
Disallow /deliver/
Disallow /editcontent/
Disallow /email
Disallow /email/
Disallow /error/
Disallow /export/
Disallow /librarians/uploadbranding
Disallow /licence-admin/
Disallow /managesavedsearches
Disallow /markedlist/
Disallow /mostviewed/
Disallow /offer/
Disallow /openurl/profile
Disallow /orders
Disallow /orders/
Disallow /pricing/
Disallow /profile
Disallow /profile/
Disallow /pubstats/
Disallow /recommendto
Disallow /reference/
Disallow /regactivation
Disallow /regactivation/
Disallow /registration
Disallow /registration/
Disallow /resourcelookup
Disallow /runsavedsearch
Disallow /savedsearches
Disallow /search
Disallow /search/
Disallow /selfservice/
Disallow /session
Disallow /session/
Disallow /statslogredirect
Disallow /statslogwrapper
Disallow /subscribedtitles
Disallow /uploadfile
Disallow /user/
Disallow /workflow/
Disallow /content/contributor
Disallow /content/contributor/
Disallow /content/institution
Disallow /content/institution/

Other Records

Field Value
crawl-delay 10

*

Rule Path
Disallow /admin/
Disallow /alerts/
Disallow /cache/
Disallow /carnet/
Disallow /cart
Disallow /cart/
Disallow /cartregistration/
Disallow /cc/
Disallow /commenting/
Disallow /counterstats/
Disallow /deliver/
Disallow /editcontent/
Disallow /email
Disallow /email/
Disallow /error/
Disallow /export/
Disallow /librarians/uploadbranding
Disallow /licence-admin/
Disallow /managesavedsearches
Disallow /markedlist/
Disallow /mostviewed/
Disallow /offer/
Disallow /openurl/profile
Disallow /orders
Disallow /orders/
Disallow /pricing/
Disallow /profile
Disallow /profile/
Disallow /pubstats/
Disallow /recommendto
Disallow /reference/
Disallow /regactivation
Disallow /regactivation/
Disallow /registration
Disallow /registration/
Disallow /resourcelookup
Disallow /runsavedsearch
Disallow /savedsearches
Disallow /search
Disallow /search/
Disallow /selfservice/
Disallow /session
Disallow /session/
Disallow /statslogredirect
Disallow /statslogwrapper
Disallow /subscribedtitles
Disallow /uploadfile
Disallow /user/
Disallow /workflow/
Disallow /content/contributor
Disallow /content/contributor/
Disallow /content/institution
Disallow /content/institution/

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://intellectdiscover.com/sitemap.xml
sitemap https://intellectdiscover.com/sitemap.xml
Back to top

Comments

  • Agent-specific settings for indexing the site.
  • Allow Google for full text crawling
  • Google ignores Crawl-delay
  • Crawl-delay: 10
  • URL prefixes
  • URL wildcards
  • Yandex have two bots so they can work twice as slowly
  • Yandex have two bots so they can work twice as slowly
  • Bing specific configuration
  • Baidu specific configuration
  • Generic settings for indexing the site.
Back to top