Log In Sign Up

jstor.com
robots.txt

Robots Exclusion Standard data for jstor.com

Resource Scan

Scan Details

Site Domain jstor.com
Base Domain jstor.com
Scan Status Ok
Last Scan2025-10-15T19:25:23+00:00
Next Scan 2025-11-14T19:25:23+00:00

Last Scan

Scanned2025-10-15T19:25:23+00:00
URL https://jstor.com/robots.txt
Redirect https://www.jstor.org/robots.txt
Redirect Domain www.jstor.org
Redirect Base jstor.org
Domain IPs 151.101.0.152, 151.101.128.152, 151.101.192.152, 151.101.64.152
Redirect IPs 151.101.0.152, 151.101.128.152, 151.101.192.152, 151.101.64.152
Response IP 199.232.112.152
Found Yes
Hash 78149d4862e485a692144bed7e41b7cd11079375be2b340ce8aff51b6dd266a6
SimHash 18d9995e4d5f

Groups

googlebot

Rule Path
Disallow /action
Disallow /api
Disallow /citation
Disallow /clockss-manifest
Disallow /doi/abs
Disallow /feedback
Disallow /purchase
Disallow /register/
Disallow /stable/full
Disallow /stable/suppl
Disallow /stable/view
Disallow /start-session
Disallow /stoken
Disallow /tc/accept
Disallow /token
Disallow /topic
Disallow /ui_log
Disallow /userimages
Allow /action/showLogin
Allow /action/showJournal
Allow /action/showPublication
Allow /action/showSubscriptionJournalsAsXml
Allow /action/showXml
Allow /doi/xml

*

Rule Path
Disallow /action
Disallow /api
Disallow /citation
Disallow /clockss-manifest
Disallow /doi/abs
Disallow /doi/xml
Disallow /feedback
Disallow /purchase
Disallow /register/
Disallow /stable/full
Disallow /stable/suppl
Disallow /stable/view
Disallow /start-session
Disallow /stoken
Disallow /tc/accept
Disallow /token
Disallow /topic
Disallow /ui_log
Disallow /userimages
Allow /action/showLogin
Allow /action/showJournal
Allow /action/showPublication
Allow /action/showSubscriptionJournalsAsXml
Allow /action/showXml

claudebot
ccbot
gptbot
google-extended

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.jstor.org/sitemap.xml
Back to top

Comments

  • Disallow crawling for the purposes of training models
  • https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
  • https://commoncrawl.org/faq
  • https://platform.openai.com/docs/gptbot
  • https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
Back to top