Log In Sign Up

iopscience.com
robots.txt

Robots Exclusion Standard data for iopscience.com

Resource Scan

Scan Details

Site Domain iopscience.com
Base Domain iopscience.com
Scan Status Ok
Last Scan2024-05-27T17:28:33+00:00
Next Scan 2024-06-26T17:28:33+00:00

Last Scan

Scanned2024-05-27T17:28:33+00:00
URL http://www.iopscience.com/robots.txt
Redirect https://iopscience.iop.org/robots.txt
Redirect Domain iopscience.iop.org
Redirect Base iop.org
Domain IPs 52.16.160.11
Redirect IPs 141.226.253.39
Response IP 141.226.253.39
Found Yes
Hash 84a873216277c026814db456d763e30ca52ac2c12b901bfff5c9500a087b7110
SimHash 20515352cdd5

Groups

googlebot

Rule Path
Disallow /EJ/openphysics/
Disallow /*sid%3DIOPP
Disallow /*jsessionid
Disallow /*fromSearchPage%3Dtrue
Disallow /*v_showaffiliations
Disallow /*relno
Disallow /nsearch*
Disallow /*fullsearch
Disallow /*searchType
Disallow /eprint
Disallow /cws
Disallow /findcontent
Disallow /*hdrSearch
Disallow /*site_preference

slurp

Rule Path
Disallow /EJ/openphysics/
Disallow /*sid%3DIOPP
Disallow /*jsessionid
Disallow /*fromSearchPage%3Dtrue
Disallow /*v_showaffiliations
Disallow /*relno
Disallow /*fullsearch
Disallow /*searchType
Disallow /eprint
Disallow /cws
Disallow /findcontent
Disallow /*hdrSearch
Disallow /*site_preference

bingbot

Rule Path
Disallow /EJ/openphysics/
Disallow /*sid%3DIOPP
Disallow /*jsessionid
Disallow /*fromSearchPage%3Dtrue
Disallow /*v_showaffiliations
Disallow /*relno
Disallow /nsearch*
Disallow /*fullsearch
Disallow /*searchType
Disallow /eprint
Disallow /cws
Disallow /findcontent
Disallow /*hdrSearch
Disallow /*site_preference

baiduspider

Rule Path
Disallow /EJ/openphysics/
Disallow /*sid%3DIOPP
Disallow /*jsessionid
Disallow /*fromSearchPage%3Dtrue
Disallow /*v_showaffiliations
Disallow /*relno
Disallow /nsearch*
Disallow /*fullsearch
Disallow /*searchType
Disallow /eprint
Disallow /cws
Disallow /findcontent
Disallow /*hdrSearch
Disallow /*site_preference

teoma

Rule Path
Disallow /EJ/openphysics/
Disallow /*sid%3DIOPP
Disallow /*jsessionid
Disallow /*fromSearchPage%3Dtrue
Disallow /*v_showaffiliations
Disallow /*relno
Disallow /nsearch*
Disallow /*fullsearch
Disallow /*searchType
Disallow /eprint
Disallow /cws
Disallow /findcontent
Disallow /*hdrSearch
Disallow /*site_preference

twitterbot

Rule Path
Disallow
Allow /

dragonbot
screaming frog seo spider
sogou web spider

Rule Path
Allow *

gptbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

*

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://iopscience.iop.org/sitemap.xml
Back to top

Comments

  • For existing directives, removed crawl-delay.
  • New directives from visibility audit
  • Duplicate content caused by crossrefs - see 3.5 from visibility audit
  • Duplicate content caused by serving jsessionids - see 3.6 from visibility audit
  • Search results pages - see 3.15 from visibility audit
  • Duplicate content due to affiliations link on article pages. Content is present in HTML anyway
  • Duplicate content caused by rel=ref and rel=rev related articles on right hand side of article page
  • Disallow crawling search results pages
  • Duplicate content caused by hdrSearch parameter tracking search options
  • Duplicate content caused by site_preference parameter, registering user's preference for desktop or mobile version
  • Yahoo! User agent is Yahoo! Slurp (http://www.inktomi.com/slurp.html)
  • New directives from visibility audit
  • Bing
  • New directives from visibility audit
  • Baidu
  • For existing directives, removed crawl-delay.
  • New directives from visibility audit
  • Duplicate content caused by crossrefs - see 3.5 from visibility audit
  • Duplicate content caused by serving jsessionids - see 3.6 from visibility audit
  • Search results pages - see 3.15 from visibility audit
  • Duplicate content due to affiliations link on article pages. Content is present in HTML anyway
  • Duplicate content caused by rel=ref and rel=rev related articles on right hand side of article page
  • Disallow crawling search results pages
  • Duplicate content caused by hdrSearch parameter tracking search options
  • Duplicate content caused by site_preference parameter, registering user's preference for desktop or mobile version
  • Ask Jeeves. User agent is Ask Jeeves/Teoma
  • New directives from visibility audit
  • Twitter cards
  • Removed other specific user agent strings as they were all set
  • to disallow: /
  • They will match the catch-all below.
Back to top

Warnings

  • 1 invalid line.
Back to top