Log In Sign Up

genome.cshlp.org
robots.txt

Robots Exclusion Standard data for genome.cshlp.org

Resource Scan

Scan Details

Site Domain genome.cshlp.org
Base Domain cshlp.org
Scan Status Ok
Last Scan2025-08-26T08:57:18+00:00
Next Scan 2025-09-25T08:57:18+00:00

Last Scan

Scanned2025-08-26T08:57:18+00:00
URL https://genome.cshlp.org/robots.txt
Domain IPs 104.232.26.111
Response IP 104.232.26.111
Found Yes
Hash 6c0a87738836d5c8a96865a00896978db83e96c7125e2622cc59aad9637e35b6
SimHash 3613fd70c1d1

Groups

*

Rule Path
Disallow /

googlebot

Rule Path
Disallow /accesslogs
Disallow /adsystem
Disallow /apps/
Disallow /backtocs/
Disallow /browse/
Disallow /careerfocus/
Disallow /cgi/adclick
Disallow /cgi/alerts
Disallow /cgi/authordata
Disallow /cgi/changeuserinfo
Disallow /cgi/citemap
Disallow /citemap
Disallow /cgi/citmgr
Disallow /citmgr
Disallow /cgi/cookietest
Disallow /cgi/crossref-forward-links
Disallow /cgi/ctalert
Disallow /cgi/ctmain
Disallow /cgi/eletter-submit
Disallow /cgi/etoc
Disallow /cgi/external_ref
Disallow /external_ref
Disallow /cgi/flagsearch
Disallow /cgi/folders
Disallow /cgi/issue_pdf
Disallow /cgi/login
Disallow /login
Disallow /cgi/mailafriend
Disallow /email
Disallow /cgi/markedcitation
Disallow /cgi/myjs
Disallow /cgi/pdf_extract
Disallow /cgi/powerpoint
Disallow /powerpoint
Disallow /cgi/register
Disallow /cgi/reprintsidebar
Disallow /cgi/savedsearch
Disallow /cgi/scopus
Disallow /cgi/search
Disallow /search
Disallow /cgi/searchhistory
Disallow /cgi/searchresults
Disallow /cgi/topics
Disallow /classifieds/
Disallow /conf/
Disallow /guides/
Disallow /help
Disallow /honeypot/
Disallow /math/
Disallow /misc/press/
Disallow /press
Disallow /searchall
Disallow /lens
Disallow /lensxml

Other Records

Field Value
crawl-delay 10

fasterfox

Rule Path
Disallow /

msnbot

Rule Path
Disallow /accesslogs
Disallow /adsystem
Disallow /apps/
Disallow /backtocs/
Disallow /browse/
Disallow /careerfocus/
Disallow /cgi/adclick
Disallow /cgi/alerts
Disallow /cgi/authordata
Disallow /cgi/changeuserinfo
Disallow /cgi/citemap
Disallow /citemap
Disallow /cgi/citmgr
Disallow /citmgr
Disallow /cgi/cookietest
Disallow /cgi/crossref-forward-links
Disallow /cgi/ctalert
Disallow /cgi/ctmain
Disallow /cgi/eletter-submit
Disallow /cgi/etoc
Disallow /cgi/external_ref
Disallow /external_ref
Disallow /cgi/flagsearch
Disallow /cgi/folders
Disallow /cgi/issue_pdf
Disallow /cgi/login
Disallow /login
Disallow /cgi/mailafriend
Disallow /email
Disallow /cgi/markedcitation
Disallow /cgi/myjs
Disallow /cgi/pdf_extract
Disallow /cgi/powerpoint
Disallow /powerpoint
Disallow /cgi/register
Disallow /cgi/reprintsidebar
Disallow /cgi/savedsearch
Disallow /cgi/scopus
Disallow /cgi/search
Disallow /search
Disallow /cgi/searchhistory
Disallow /cgi/searchresults
Disallow /cgi/topics
Disallow /classifieds/
Disallow /conf/
Disallow /guides/
Disallow /help
Disallow /honeypot/
Disallow /math/
Disallow /misc/press/
Disallow /press
Disallow /searchall
Disallow /lens
Disallow /lensxml

Other Records

Field Value
crawl-delay 10

ia_archiver

Rule Path
Disallow /accesslogs
Disallow /adsystem
Disallow /apps/
Disallow /backtocs/
Disallow /browse/
Disallow /careerfocus/
Disallow /cgi/adclick
Disallow /cgi/alerts
Disallow /cgi/authordata
Disallow /cgi/changeuserinfo
Disallow /cgi/citemap
Disallow /citemap
Disallow /cgi/citmgr
Disallow /citmgr
Disallow /cgi/cookietest
Disallow /cgi/crossref-forward-links
Disallow /cgi/ctalert
Disallow /cgi/ctmain
Disallow /cgi/eletter-submit
Disallow /cgi/etoc
Disallow /cgi/external_ref
Disallow /external_ref
Disallow /cgi/flagsearch
Disallow /cgi/folders
Disallow /cgi/issue_pdf
Disallow /cgi/login
Disallow /login
Disallow /cgi/mailafriend
Disallow /email
Disallow /cgi/markedcitation
Disallow /cgi/myjs
Disallow /cgi/pdf_extract
Disallow /cgi/powerpoint
Disallow /powerpoint
Disallow /cgi/register
Disallow /cgi/reprintsidebar
Disallow /cgi/savedsearch
Disallow /cgi/scopus
Disallow /cgi/search
Disallow /search
Disallow /cgi/searchhistory
Disallow /cgi/searchresults
Disallow /cgi/topics
Disallow /classifieds/
Disallow /conf/
Disallow /guides/
Disallow /help
Disallow /honeypot/
Disallow /math/
Disallow /misc/press/
Disallow /press
Disallow /searchall
Disallow /lens
Disallow /lensxml

Other Records

Field Value
crawl-delay 10

turnitinbot

Rule Path
Disallow /accesslogs
Disallow /adsystem
Disallow /apps/
Disallow /backtocs/
Disallow /browse/
Disallow /careerfocus/
Disallow /cgi/adclick
Disallow /cgi/alerts
Disallow /cgi/authordata
Disallow /cgi/changeuserinfo
Disallow /cgi/citemap
Disallow /citemap
Disallow /cgi/citmgr
Disallow /citmgr
Disallow /cgi/cookietest
Disallow /cgi/crossref-forward-links
Disallow /cgi/ctalert
Disallow /cgi/ctmain
Disallow /cgi/eletter-submit
Disallow /cgi/etoc
Disallow /cgi/external_ref
Disallow /external_ref
Disallow /cgi/flagsearch
Disallow /cgi/folders
Disallow /cgi/issue_pdf
Disallow /cgi/login
Disallow /login
Disallow /cgi/mailafriend
Disallow /email
Disallow /cgi/markedcitation
Disallow /cgi/myjs
Disallow /cgi/pdf_extract
Disallow /cgi/powerpoint
Disallow /powerpoint
Disallow /cgi/register
Disallow /cgi/reprintsidebar
Disallow /cgi/savedsearch
Disallow /cgi/scopus
Disallow /cgi/search
Disallow /search
Disallow /cgi/searchhistory
Disallow /cgi/searchresults
Disallow /cgi/topics
Disallow /classifieds/
Disallow /conf/
Disallow /guides/
Disallow /help
Disallow /honeypot/
Disallow /math/
Disallow /misc/press/
Disallow /press
Disallow /searchall
Disallow /lens
Disallow /lensxml

Other Records

Field Value
crawl-delay 10

baiduspider

Rule Path
Disallow /accesslogs
Disallow /adsystem
Disallow /apps/
Disallow /backtocs/
Disallow /browse/
Disallow /careerfocus/
Disallow /cgi/adclick
Disallow /cgi/alerts
Disallow /cgi/authordata
Disallow /cgi/changeuserinfo
Disallow /cgi/citemap
Disallow /citemap
Disallow /cgi/citmgr
Disallow /citmgr
Disallow /cgi/cookietest
Disallow /cgi/crossref-forward-links
Disallow /cgi/ctalert
Disallow /cgi/ctmain
Disallow /cgi/eletter-submit
Disallow /cgi/etoc
Disallow /cgi/external_ref
Disallow /external_ref
Disallow /cgi/flagsearch
Disallow /cgi/folders
Disallow /cgi/issue_pdf
Disallow /cgi/login
Disallow /login
Disallow /cgi/mailafriend
Disallow /email
Disallow /cgi/markedcitation
Disallow /cgi/myjs
Disallow /cgi/pdf_extract
Disallow /cgi/powerpoint
Disallow /powerpoint
Disallow /cgi/register
Disallow /cgi/reprintsidebar
Disallow /cgi/savedsearch
Disallow /cgi/scopus
Disallow /cgi/search
Disallow /search
Disallow /cgi/searchhistory
Disallow /cgi/searchresults
Disallow /cgi/topics
Disallow /classifieds/
Disallow /conf/
Disallow /guides/
Disallow /help
Disallow /honeypot/
Disallow /math/
Disallow /misc/press/
Disallow /press
Disallow /searchall
Disallow /lens
Disallow /lensxml

Other Records

Field Value
crawl-delay 10

infovell

Rule Path
Disallow /accesslogs
Disallow /adsystem
Disallow /apps/
Disallow /backtocs/
Disallow /browse/
Disallow /careerfocus/
Disallow /cgi/adclick
Disallow /cgi/alerts
Disallow /cgi/authordata
Disallow /cgi/changeuserinfo
Disallow /cgi/citemap
Disallow /citemap
Disallow /cgi/citmgr
Disallow /citmgr
Disallow /cgi/cookietest
Disallow /cgi/crossref-forward-links
Disallow /cgi/ctalert
Disallow /cgi/ctmain
Disallow /cgi/eletter-submit
Disallow /cgi/etoc
Disallow /cgi/external_ref
Disallow /external_ref
Disallow /cgi/flagsearch
Disallow /cgi/folders
Disallow /cgi/issue_pdf
Disallow /cgi/login
Disallow /login
Disallow /cgi/mailafriend
Disallow /email
Disallow /cgi/markedcitation
Disallow /cgi/myjs
Disallow /cgi/pdf_extract
Disallow /cgi/powerpoint
Disallow /powerpoint
Disallow /cgi/register
Disallow /cgi/reprintsidebar
Disallow /cgi/savedsearch
Disallow /cgi/scopus
Disallow /cgi/search
Disallow /search
Disallow /cgi/searchhistory
Disallow /cgi/searchresults
Disallow /cgi/topics
Disallow /classifieds/
Disallow /conf/
Disallow /guides/
Disallow /help
Disallow /honeypot/
Disallow /math/
Disallow /misc/press/
Disallow /press
Disallow /searchall
Disallow /lens
Disallow /lensxml

Other Records

Field Value
crawl-delay 10
Back to top

Comments

  • THIS IS A GENERATED FILE.
  • ANY CHANGES YOU MAKE DIRECTLY TO THIS FILE WILL BE OVERWRITTEN
  • AND LOST THE NEXT TIME THIS FILE IS GENERATED.
  • (This file last generated: Jul 22, 2025, 11:23 P.M.)
  • BELOW added by: file:/hwpnfs3/mayshare02/shared/webapp/modules/genome/resources/shared/build/robots-txt.base.xml
  • BELOW added by: file:/hwpnfs3/mayshare02/shared/webapp/modules/genome/resources/publisher/build/robots-txt.mix-in.xml
  • END OF ADDED BLOCK
  • END OF ADDED BLOCK
Back to top