Log In Sign Up

www.ncbi.nlm.nih.gov
robots.txt

Robots Exclusion Standard data for www.ncbi.nlm.nih.gov

Resource Scan

Scan Details

Site Domain www.ncbi.nlm.nih.gov
Base Domain nih.gov
Scan Status Ok
Last Scan2024-05-21T17:49:10+00:00
Next Scan 2024-06-20T17:49:10+00:00

Last Scan

Scanned2024-05-21T17:49:10+00:00
URL https://www.ncbi.nlm.nih.gov/robots.txt
Domain IPs 130.14.29.110, 2607:f220:41e:4290::110
Response IP 130.14.29.110
Found Yes
Hash 603c3bb4727454809bb64bd8ba5a4e653c3a6ea1afd7a2d34d8887251f4bb9cf
SimHash 0b3e12000382

Groups

htdig/3.1.6

Rule Path
Disallow /cgi-bin
Disallow /entrez
Disallow /stat
Disallow /COG
Disallow /Entrez
Disallow /mailman
Disallow /myncbi
Disallow /sutils
Disallow /Taxonomy/Selector
Disallow /Taxonomy/CommonTree
Disallow /entrez/sutils
Disallow /mapview
Disallow /blast/BlastAlign.cgi
Disallow /blast/bl2seq/wblast2.cgi
Disallow /portal
Disallow /pmc/utilities/
Disallow /pmc/issues/
Disallow /pmc/articles/
Disallow /pmc/ivip/
Disallow /pmc/articlerender.fcgi
Disallow /pmc/pagerender.fcgi
Disallow /pmc/picrender.fcgi
Disallow /pmc/tocrender.fcgi
Disallow /pmc/*report%3D
Disallow /pmc/*page%3D
Disallow /pmc/articles/*/citedby/
Disallow /pmc/publisherportal/api/
Disallow /pmc/publisherportal/application/
Disallow /pmc/publisherportal/download
Disallow /pmc/publisherportal/journalmanager/
Disallow /labs/pmc/
Disallow /geo/tools
Disallow /projects/geo/tools
Disallow /geo/download
Disallow /projects/geo/download
Disallow /projects/gtr/gt
Disallow /Structure/flink/flink.cgi
Disallow /Structure/vast/vastsrv.cgi
Disallow /viewvc
Disallow /sars-cov-2

baiduspider

Rule Path
Disallow /cgi-bin
Disallow /entrez
Disallow /stat
Disallow /COG
Disallow /Entrez
Disallow /mailman
Disallow /myncbi
Disallow /sutils
Disallow /Taxonomy/Selector
Disallow /Taxonomy/CommonTree
Disallow /entrez/sutils
Disallow /mapview
Disallow /blast/BlastAlign.cgi
Disallow /blast/bl2seq/wblast2.cgi
Disallow /portal
Disallow /pmc/utilities/
Disallow /pmc/issues/
Disallow /pmc/articles/
Disallow /pmc/ivip/
Disallow /pmc/articlerender.fcgi
Disallow /pmc/pagerender.fcgi
Disallow /pmc/picrender.fcgi
Disallow /pmc/tocrender.fcgi
Disallow /pmc/*report%3D
Disallow /pmc/*page%3D
Disallow /pmc/articles/*/citedby/
Disallow /pmc/publisherportal/api/
Disallow /pmc/publisherportal/application/
Disallow /pmc/publisherportal/download
Disallow /pmc/publisherportal/journalmanager/
Disallow /labs/pmc/
Disallow /geo/tools
Disallow /projects/geo/tools
Disallow /geo/download
Disallow /projects/geo/download
Disallow /projects/gtr/gt
Disallow /Structure/flink/flink.cgi
Disallow /Structure/vast/vastsrv.cgi
Disallow /viewvc
Disallow /sars-cov-2

googlebot

Rule Path
Disallow /cgi-bin
Disallow /stat
Disallow /COG
Disallow /mailman
Disallow /sutils
Disallow /entrez/viewer.fcgi
Disallow /Taxonomy/Selector
Disallow /Taxonomy/CommonTree
Disallow /entrez/sutils
Disallow /SAGE
Disallow /SNP
Disallow /snp
Disallow /IEB/ToolBox/SB
Disallow /Structure/dgs
Disallow /Structure/VA
Disallow /Structure/cblast/cblast.cgi
Disallow /Coffeebreak
Disallow /gorf
Disallow /mapview
Disallow /myncbi
Disallow /blast/BlastAlign.cgi
Disallow /blast/bl2seq/wblast2.cgi
Disallow /entrez/eutils
Disallow /entrez/utils
Disallow /portal
Disallow /pmc/utilities/
Disallow /pmc/issues/
Disallow /pmc/*/bin/
Disallow /pmc/*/pdf/
Disallow /pmc/*/equ/
Disallow /pmc/*.cgi
Disallow /pmc/*report%3D
Disallow /pmc/*page%3D
Disallow /pmc/articles/*/citedby/
Disallow /pmc/publisherportal/api/
Disallow /pmc/publisherportal/application/
Disallow /pmc/publisherportal/download
Disallow /pmc/publisherportal/journalmanager/
Disallow /labs/pmc/
Disallow /geo/tools
Disallow /projects/geo/tools
Disallow /geo/download
Disallow /projects/geo/download
Disallow /projects/gtr/gt
Disallow /Structure/flink/flink.cgi
Disallow /Structure/vast/vastsrv.cgi
Disallow /viewvc
Disallow /clone/library/genomic/organisms
Disallow /sars-cov-2
Allow /portal/portal3rc.fcgi/*/*.js
Allow /portal/portal3rc.fcgi/*/*.css

teleport pro

Rule Path
Disallow /cgi-bin
Disallow /entrez
Disallow /stat
Disallow /COG
Disallow /Entrez
Disallow /myncbi
Disallow /mailman
Disallow /sutils
Disallow /Taxonomy/Selector
Disallow /Taxonomy/CommonTree
Disallow /entrez/sutils
Disallow /mapview
Disallow /blast/BlastAlign.cgi
Disallow /blast/bl2seq/wblast2.cgi
Disallow /portal
Disallow /pmc/utilities/
Disallow /pmc/issues/
Disallow /pmc/articles/
Disallow /pmc/ivip/
Disallow /pmc/articlerender.fcgi
Disallow /pmc/pagerender.fcgi
Disallow /pmc/picrender.fcgi
Disallow /pmc/tocrender.fcgi
Disallow /pmc/publisherportal/api/
Disallow /pmc/publisherportal/application/
Disallow /pmc/publisherportal/download
Disallow /pmc/publisherportal/journalmanager/
Disallow /labs/pmc/
Disallow /geo/tools
Disallow /projects/geo/tools
Disallow /geo/download
Disallow /projects/geo/download
Disallow /projects/gtr/gt
Disallow /Structure/flink/flink.cgi
Disallow /Structure/vast/vastsrv.cgi
Disallow /viewvc
Disallow /sars-cov-2

yacybot

Rule Path
Disallow *

browsershots

Rule Path
Disallow
Disallow /sars-cov-2

slurp

Rule Path
Disallow /cgi-bin
Disallow /entrez
Disallow /stat
Disallow /COG
Disallow /Entrez
Disallow /myncbi
Disallow /mailman
Disallow /sutils
Disallow /Taxonomy/Selector
Disallow /Taxonomy/CommonTree
Disallow /entrez/sutils
Disallow /mapview
Disallow /blast/BlastAlign.cgi
Disallow /blast/bl2seq/wblast2.cgi
Disallow /SAGE
Disallow /SNP
Disallow /snp
Disallow /portal
Disallow /pmc/utilities/
Disallow /pmc/*/bin/
Disallow /pmc/*/pdf/
Disallow /pmc/*/equ/
Disallow /pmc/*.cgi
Disallow /pmc/*report%3D
Disallow /pmc/*page%3D
Disallow /pmc/articles/*/citedby/
Disallow /pmc/publisherportal/api/
Disallow /pmc/publisherportal/application/
Disallow /pmc/publisherportal/download
Disallow /pmc/publisherportal/journalmanager/
Disallow /labs/pmc/
Disallow /geo/tools
Disallow /projects/geo/tools
Disallow /geo/download
Disallow /projects/geo/download
Disallow /projects/gtr/gt
Disallow /Structure/flink/flink.cgi
Disallow /Structure/vast/vastsrv.cgi
Disallow /viewvc
Disallow /sars-cov-2

Other Records

Field Value
crawl-delay 1

twitterbot

Rule Path
Disallow *
Allow /pmc/articles
Allow /books/NBK*

exabot

Rule Path
Disallow /Structure
Disallow /mailman
Disallow /stat
Disallow /sars-cov-2

bingbot

Rule Path
Disallow /cgi-bin
Disallow /entrez
Disallow /stat
Disallow /COG
Disallow /Entrez
Disallow /myncbi
Disallow /mailman
Disallow /sutils
Disallow /Taxonomy/Selector
Disallow /Taxonomy/CommonTree
Disallow /entrez/sutils
Disallow /mapview
Disallow /SAGE
Disallow /SNP
Disallow /snp
Disallow /blast/BlastAlign.cgi
Disallow /blast/bl2seq/wblast2.cgi
Disallow /portal
Disallow /pmc/utilities/
Disallow /pmc/issues/
Disallow /pmc/*/bin/
Disallow /pmc/*/pdf/
Disallow /pmc/*/equ/
Disallow /pmc/*.cgi
Disallow /pmc/*report%3D
Disallow /pmc/*page%3D
Disallow /pmc/articles/*/citedby/
Disallow /pmc/publisherportal/api/
Disallow /pmc/publisherportal/application/
Disallow /pmc/publisherportal/download
Disallow /pmc/publisherportal/journalmanager/
Disallow /labs/pmc/
Disallow /geo/tools
Disallow /projects/geo/tools
Disallow /geo/download
Disallow /projects/geo/download
Disallow /projects/gtr/gt
Disallow /Structure/flink/flink.cgi
Disallow /Structure/vast/vastsrv.cgi
Disallow /viewvc
Disallow /clone/library/genomic/organisms
Disallow /sars-cov-2
Allow /portal/portal3rc.fcgi/*/*.js
Allow /portal/portal3rc.fcgi/*/*.css

*

Rule Path
Disallow /cgi-bin
Disallow /stat
Disallow /entrez
Disallow /COG
Disallow /Entrez
Disallow /myncbi
Disallow /mailman
Disallow /sutils
Disallow /Taxonomy/Selector
Disallow /Taxonomy/CommonTree
Disallow /entrez/sutils
Disallow /mapview
Disallow /SAGE
Disallow /SNP
Disallow /snp
Disallow /blast/BlastAlign.cgi
Disallow /blast/bl2seq/wblast2.cgi
Disallow /portal
Disallow /pmc/utilities/
Disallow /pmc/issues/
Disallow /pmc/articles/
Disallow /pmc/ivip/
Disallow /pmc/articlerender.fcgi
Disallow /pmc/pagerender.fcgi
Disallow /pmc/picrender.fcgi
Disallow /pmc/tocrender.fcgi
Disallow /pmc/*report%3D
Disallow /pmc/*page%3D
Disallow /pmc/articles/*/citedby/
Disallow /pmc/publisherportal/api/
Disallow /pmc/publisherportal/application/
Disallow /pmc/publisherportal/download
Disallow /pmc/publisherportal/journalmanager/
Disallow /labs/pmc/
Disallow /geo/tools
Disallow /projects/geo/tools
Disallow /geo/download
Disallow /projects/geo/download
Disallow /projects/gtr/gt
Disallow /Structure/flink/flink.cgi
Disallow /Structure/vast/vastsrv.cgi
Disallow /viewvc
Disallow /clone/library/genomic/organisms
Disallow /sars-cov-2
Allow /portal/portal3rc.fcgi/*/*.js
Allow /portal/portal3rc.fcgi/*/*.css

Other Records

Field Value
crawl-delay 5
Back to top

Other Records

Field Value
sitemap https://www.ncbi.nlm.nih.gov/Structure/sitemaps/cdd/sitemap_index.xml
sitemap https://www.ncbi.nlm.nih.gov/Structure/sitemaps/genome/sitemap_index.xml
sitemap https://www.ncbi.nlm.nih.gov/Structure/sitemaps/mmdb/sitemap_index.xml
sitemap https://www.ncbi.nlm.nih.gov/Structure/sitemaps/sparcle/sitemap.xml
sitemap https://www.ncbi.nlm.nih.gov/Traces/sitemap_index.xml
sitemap https://www.ncbi.nlm.nih.gov/core/assets/osiris/files/sitemap.xml
sitemap https://www.ncbi.nlm.nih.gov/escan?p=assembly/sitemap_index.xml
sitemap https://www.ncbi.nlm.nih.gov/escan?p=biosystems/sitemap_index.xml
sitemap https://www.ncbi.nlm.nih.gov/escan?p=books/sitemap_index.xml
sitemap https://www.ncbi.nlm.nih.gov/escan?p=cdd/sitemap_index.xml
sitemap https://www.ncbi.nlm.nih.gov/escan?p=clinvar/sitemap_index.xml
sitemap https://www.ncbi.nlm.nih.gov/escan?p=gene/sitemap_index.xml
sitemap https://www.ncbi.nlm.nih.gov/escan?p=gtr/sitemap_index.xml
sitemap https://www.ncbi.nlm.nih.gov/escan?p=medgen/sitemap_index.xml
sitemap https://www.ncbi.nlm.nih.gov/escan?p=mesh/sitemap_index.xml
sitemap https://www.ncbi.nlm.nih.gov/escan?p=ncbi/sitemap.xml
sitemap https://www.ncbi.nlm.nih.gov/escan?p=nlmcatalog/sitemap_index.xml
sitemap https://www.ncbi.nlm.nih.gov/escan?p=orgtrack/sitemap_index.xml
sitemap https://www.ncbi.nlm.nih.gov/escan?p=pmc/sitemap_index.xml
sitemap https://www.ncbi.nlm.nih.gov/escan?p=snp/sitemap_index.xml
sitemap https://www.ncbi.nlm.nih.gov/escan?p=sra/sitemap_index.xml
Back to top

Comments

  • robots.txt - robot exclusion file - back-end server version - no robots!
  • ========================================================================
Back to top