Log In Sign Up

annualreviews.org
robots.txt

Robots Exclusion Standard data for annualreviews.org

Resource Scan

Scan Details

Site Domain annualreviews.org
Base Domain annualreviews.org
Scan Status Ok
Last Scan2024-09-17T23:19:56+00:00
Next Scan 2024-09-24T23:19:56+00:00

Last Scan

Scanned2024-09-17T23:19:56+00:00
URL https://annualreviews.org/robots.txt
Redirect https://www.annualreviews.org/robots.txt
Redirect Domain www.annualreviews.org
Redirect Base annualreviews.org
Domain IPs 104.18.24.141, 104.18.25.141
Redirect IPs 104.18.24.141, 104.18.25.141
Response IP 104.18.25.141
Found Yes
Hash 46f2432eb5ab88a64c5cdc47aafd5ec561ff5990284e1b884d94534ac6b3e617
SimHash 2db5cd3aa397

Groups

googlebot

Rule Path
Disallow /admin/
Disallow /alerts/
Disallow /cache/
Disallow /carnet/
Disallow /cart
Disallow /cart/
Disallow /cartregistration/
Disallow /cc/
Disallow /commenting/
Disallow /counterstats/
Disallow /deliver/
Disallow /editcontent/
Disallow /email
Disallow /email/
Disallow /error/
Disallow /export/
Disallow /librarians/uploadbranding
Disallow /licence-admin/
Disallow /managesavedsearches
Disallow /markedlist/
Disallow /mostviewed/
Disallow /offer/
Disallow /openurl/profile
Disallow /orders
Disallow /orders/
Disallow /pricing/
Disallow /profile
Disallow /profile/
Disallow /pubstats/
Disallow /recommendto
Disallow /reference/
Disallow /regactivation
Disallow /regactivation/
Disallow /registration
Disallow /registration/
Disallow /resourcelookup
Disallow /runsavedsearch
Disallow /savedsearches
Disallow /search
Disallow /search/
Disallow /selfservice/
Disallow /session
Disallow /session/
Disallow /statslogredirect
Disallow /statslogwrapper
Disallow /subscribedtitles
Disallow /uploadfile
Disallow /user/
Disallow /workflow/
Disallow /content/contributor
Disallow /content/contributor/
Disallow /content/institution
Disallow /content/institution/

yandexbot

Rule Path
Disallow /admin/
Disallow /alerts/
Disallow /cache/
Disallow /carnet/
Disallow /cart
Disallow /cart/
Disallow /cartregistration/
Disallow /cc/
Disallow /commenting/
Disallow /counterstats/
Disallow /deliver/
Disallow /editcontent/
Disallow /email
Disallow /email/
Disallow /error/
Disallow /export/
Disallow /librarians/uploadbranding
Disallow /licence-admin/
Disallow /managesavedsearches
Disallow /markedlist/
Disallow /mostviewed/
Disallow /offer/
Disallow /openurl/profile
Disallow /orders
Disallow /orders/
Disallow /pricing/
Disallow /profile
Disallow /profile/
Disallow /pubstats/
Disallow /recommendto
Disallow /reference/
Disallow /regactivation
Disallow /regactivation/
Disallow /registration
Disallow /registration/
Disallow /resourcelookup
Disallow /runsavedsearch
Disallow /savedsearches
Disallow /search
Disallow /search/
Disallow /selfservice/
Disallow /session
Disallow /session/
Disallow /statslogredirect
Disallow /statslogwrapper
Disallow /subscribedtitles
Disallow /uploadfile
Disallow /user/
Disallow /workflow/
Disallow /content/contributor
Disallow /content/contributor/
Disallow /content/institution
Disallow /content/institution/

Other Records

Field Value
crawl-delay 4

yandeximages

Rule Path
Disallow /admin/
Disallow /alerts/
Disallow /cache/
Disallow /carnet/
Disallow /cart
Disallow /cart/
Disallow /cartregistration/
Disallow /cc/
Disallow /commenting/
Disallow /counterstats/
Disallow /deliver/
Disallow /editcontent/
Disallow /email
Disallow /email/
Disallow /error/
Disallow /export/
Disallow /librarians/uploadbranding
Disallow /licence-admin/
Disallow /managesavedsearches
Disallow /markedlist/
Disallow /mostviewed/
Disallow /offer/
Disallow /openurl/profile
Disallow /orders
Disallow /orders/
Disallow /pricing/
Disallow /profile
Disallow /profile/
Disallow /pubstats/
Disallow /recommendto
Disallow /reference/
Disallow /regactivation
Disallow /regactivation/
Disallow /registration
Disallow /registration/
Disallow /resourcelookup
Disallow /runsavedsearch
Disallow /savedsearches
Disallow /search
Disallow /search/
Disallow /selfservice/
Disallow /session
Disallow /session/
Disallow /statslogredirect
Disallow /statslogwrapper
Disallow /subscribedtitles
Disallow /uploadfile
Disallow /user/
Disallow /workflow/
Disallow /content/contributor
Disallow /content/contributor/
Disallow /content/institution
Disallow /content/institution/

Other Records

Field Value
crawl-delay 4

bingbot

Rule Path
Disallow /admin/
Disallow /alerts/
Disallow /cache/
Disallow /carnet/
Disallow /cart
Disallow /cart/
Disallow /cartregistration/
Disallow /cc/
Disallow /commenting/
Disallow /counterstats/
Disallow /deliver/
Disallow /editcontent/
Disallow /email
Disallow /email/
Disallow /error/
Disallow /export/
Disallow /librarians/uploadbranding
Disallow /licence-admin/
Disallow /managesavedsearches
Disallow /markedlist/
Disallow /mostviewed/
Disallow /offer/
Disallow /openurl/profile
Disallow /orders
Disallow /orders/
Disallow /pricing/
Disallow /profile
Disallow /profile/
Disallow /pubstats/
Disallow /recommendto
Disallow /reference/
Disallow /regactivation
Disallow /regactivation/
Disallow /registration
Disallow /registration/
Disallow /resourcelookup
Disallow /runsavedsearch
Disallow /savedsearches
Disallow /search
Disallow /search/
Disallow /selfservice/
Disallow /session
Disallow /session/
Disallow /statslogredirect
Disallow /statslogwrapper
Disallow /subscribedtitles
Disallow /uploadfile
Disallow /user/
Disallow /workflow/
Disallow /content/contributor
Disallow /content/contributor/
Disallow /content/institution
Disallow /content/institution/

Other Records

Field Value
crawl-delay 2

baiduspider

Rule Path
Disallow /admin/
Disallow /alerts/
Disallow /cache/
Disallow /carnet/
Disallow /cart
Disallow /cart/
Disallow /cartregistration/
Disallow /cc/
Disallow /commenting/
Disallow /counterstats/
Disallow /deliver/
Disallow /editcontent/
Disallow /email
Disallow /email/
Disallow /error/
Disallow /export/
Disallow /librarians/uploadbranding
Disallow /licence-admin/
Disallow /managesavedsearches
Disallow /markedlist/
Disallow /mostviewed/
Disallow /offer/
Disallow /openurl/profile
Disallow /orders
Disallow /orders/
Disallow /pricing/
Disallow /profile
Disallow /profile/
Disallow /pubstats/
Disallow /recommendto
Disallow /reference/
Disallow /regactivation
Disallow /regactivation/
Disallow /registration
Disallow /registration/
Disallow /resourcelookup
Disallow /runsavedsearch
Disallow /savedsearches
Disallow /search
Disallow /search/
Disallow /selfservice/
Disallow /session
Disallow /session/
Disallow /statslogredirect
Disallow /statslogwrapper
Disallow /subscribedtitles
Disallow /uploadfile
Disallow /user/
Disallow /workflow/
Disallow /content/contributor
Disallow /content/contributor/
Disallow /content/institution
Disallow /content/institution/

Other Records

Field Value
crawl-delay 2

*

Rule Path
Disallow /admin/
Disallow /alerts/
Disallow /cache/
Disallow /carnet/
Disallow /cart
Disallow /cart/
Disallow /cartregistration/
Disallow /cc/
Disallow /commenting/
Disallow /counterstats/
Disallow /deliver/
Disallow /editcontent/
Disallow /email
Disallow /email/
Disallow /error/
Disallow /export/
Disallow /librarians/uploadbranding
Disallow /licence-admin/
Disallow /managesavedsearches
Disallow /markedlist/
Disallow /mostviewed/
Disallow /offer/
Disallow /openurl/profile
Disallow /orders
Disallow /orders/
Disallow /pricing/
Disallow /profile
Disallow /profile/
Disallow /pubstats/
Disallow /recommendto
Disallow /reference/
Disallow /regactivation
Disallow /regactivation/
Disallow /registration
Disallow /registration/
Disallow /resourcelookup
Disallow /runsavedsearch
Disallow /savedsearches
Disallow /search
Disallow /search/
Disallow /selfservice/
Disallow /session
Disallow /session/
Disallow /statslogredirect
Disallow /statslogwrapper
Disallow /subscribedtitles
Disallow /uploadfile
Disallow /user/
Disallow /workflow/
Disallow /content/contributor
Disallow /content/contributor/
Disallow /content/institution
Disallow /content/institution/

Other Records

Field Value
crawl-delay 2
Back to top

Other Records

Field Value
sitemap https://www.annualreviews.org/sitemap.xml
sitemap https://www.annualreviews.org/sitemap.xml
Back to top

Comments

  • Agent-specific settings for indexing the site.
  • Allow Google for full text crawling
  • Google ignores Crawl-delay
  • Crawl-delay: 2
  • URL prefixes
  • URL wildcards
  • Yandex have two bots so they can work twice as slowly
  • Yandex have two bots so they can work twice as slowly
  • Bing specific configuration
  • Baidu specific configuration
  • Generic settings for indexing the site.
Back to top