Critical Assessment of Metagenome Interpretation - A benchmark of metagenomics software

Sczyrba, A; Hofmann, P; Belmann, P; Koslicki, D; Janssen, S; Dröge, J; Gregor, I; Majda, S; Fiedler, J; Dahms, E; Bremges, A; Fritz, A; Garrido-Oter, R; Jørgensen, TS; Shapiro, N; Blood, PD; Gurevich, A; Bai, Y; Turaev, D; Demaere, MZ; Chikhi, R; Nagarajan, N; Quince, C; Meyer, F; Balvočiutė, M; Hansen, LH; Sørensen, SJ; Chia, BKH; Denis, B; Froula, JL; Wang, Z; Egan, R; Don Kang, D; Cook, JJ; Deltel, C; Beckstette, M; Lemaitre, C; Peterlongo, P; Rizk, G; Lavenier, D; Wu, YW; Singer, SW; Jain, C; Strous, M; Klingenberg, H; Meinicke, P; Barton, MD; Lingner, T; Lin, HH; Liao, YC; Silva, GGZ; Cuevas, DA; Edwards, RA; Saha, S; Piro, VC; Renard, BY; Pop, M; Klenk, HP; Göker, M; Kyrpides, NC; Woyke, T; Vorholt, JA; Schulze-Lefert, P; Rubin, EM; Darling, AE; Rattei, T; McHardy, AC

Critical Assessment of Metagenome Interpretation - A benchmark of metagenomics software

Sczyrba, A Hofmann, P Belmann, P Koslicki, D Janssen, S Dröge, J Gregor, I Majda, S Fiedler, J Dahms, E Bremges, A Fritz, A Garrido-Oter, R Jørgensen, TS Shapiro, N Blood, PD Gurevich, A Bai, Y Turaev, D Demaere, MZ

Chikhi, R Nagarajan, N Quince, C Meyer, F Balvočiutė, M Hansen, LH Sørensen, SJ Chia, BKH Denis, B Froula, JL Wang, Z Egan, R Don Kang, D Cook, JJ Deltel, C Beckstette, M Lemaitre, C Peterlongo, P Rizk, G Lavenier, D Wu, YW Singer, SW Jain, C Strous, M Klingenberg, H Meinicke, P Barton, MD Lingner, T Lin, HH Liao, YC Silva, GGZ Cuevas, DA Edwards, RA Saha, S Piro, VC Renard, BY Pop, M Klenk, HP Göker, M Kyrpides, NC Woyke, T Vorholt, JA Schulze-Lefert, P Rubin, EM Darling, AE

Rattei, T McHardy, AC

Permalink

Publication Type:: Journal Article
Citation:: Nature Methods, 2017, 14 (11), pp. 1063 - 1071
Issue Date:: 2017-10-31

Open Access

Copyright Clearance Process

Recently Added
In Progress
Open Access

This item is open access.

Adobe PDF

Download Accepted Manuscript VersionAdobe PDF (5.82 MB)

Adobe PDF

Download Accepted Manuscript VersionAdobe PDF (1.54 MB)

View on publisher's site

View statistics

Full metadata record

Field	Value	Language
dc.contributor.author	Sczyrba, A	en_US
dc.contributor.author	Hofmann, P	en_US
dc.contributor.author	Belmann, P	en_US
dc.contributor.author	Koslicki, D	en_US
dc.contributor.author	Janssen, S	en_US
dc.contributor.author	Dröge, J	en_US
dc.contributor.author	Gregor, I	en_US
dc.contributor.author	Majda, S	en_US
dc.contributor.author	Fiedler, J	en_US
dc.contributor.author	Dahms, E	en_US
dc.contributor.author	Bremges, A	en_US
dc.contributor.author	Fritz, A	en_US
dc.contributor.author	Garrido-Oter, R	en_US
dc.contributor.author	Jørgensen, TS	en_US
dc.contributor.author	Shapiro, N	en_US
dc.contributor.author	Blood, PD	en_US
dc.contributor.author	Gurevich, A	en_US
dc.contributor.author	Bai, Y	en_US
dc.contributor.author	Turaev, D	en_US
dc.contributor.author	Demaere, MZ https://orcid.org/0000-0002-7601-5108	en_US
dc.contributor.author	Chikhi, R	en_US
dc.contributor.author	Nagarajan, N	en_US
dc.contributor.author	Quince, C	en_US
dc.contributor.author	Meyer, F	en_US
dc.contributor.author	Balvočiutė, M	en_US
dc.contributor.author	Hansen, LH	en_US
dc.contributor.author	Sørensen, SJ	en_US
dc.contributor.author	Chia, BKH	en_US
dc.contributor.author	Denis, B	en_US
dc.contributor.author	Froula, JL	en_US
dc.contributor.author	Wang, Z	en_US
dc.contributor.author	Egan, R	en_US
dc.contributor.author	Don Kang, D	en_US
dc.contributor.author	Cook, JJ	en_US
dc.contributor.author	Deltel, C	en_US
dc.contributor.author	Beckstette, M	en_US
dc.contributor.author	Lemaitre, C	en_US
dc.contributor.author	Peterlongo, P	en_US
dc.contributor.author	Rizk, G	en_US
dc.contributor.author	Lavenier, D	en_US
dc.contributor.author	Wu, YW	en_US
dc.contributor.author	Singer, SW	en_US
dc.contributor.author	Jain, C	en_US
dc.contributor.author	Strous, M	en_US
dc.contributor.author	Klingenberg, H	en_US
dc.contributor.author	Meinicke, P	en_US
dc.contributor.author	Barton, MD	en_US
dc.contributor.author	Lingner, T	en_US
dc.contributor.author	Lin, HH	en_US
dc.contributor.author	Liao, YC	en_US
dc.contributor.author	Silva, GGZ	en_US
dc.contributor.author	Cuevas, DA	en_US
dc.contributor.author	Edwards, RA	en_US
dc.contributor.author	Saha, S	en_US
dc.contributor.author	Piro, VC	en_US
dc.contributor.author	Renard, BY	en_US
dc.contributor.author	Pop, M	en_US
dc.contributor.author	Klenk, HP	en_US
dc.contributor.author	Göker, M	en_US
dc.contributor.author	Kyrpides, NC	en_US
dc.contributor.author	Woyke, T	en_US
dc.contributor.author	Vorholt, JA	en_US
dc.contributor.author	Schulze-Lefert, P	en_US
dc.contributor.author	Rubin, EM	en_US
dc.contributor.author	Darling, AE https://orcid.org/0000-0003-2397-7925	en_US
dc.contributor.author	Rattei, T	en_US
dc.contributor.author	McHardy, AC	en_US
dc.date.available	2017-08-25	en_US
dc.date.issued	2017-10-31	en_US
dc.identifier.citation	Nature Methods, 2017, 14 (11), pp. 1063 - 1071	en_US
dc.identifier.issn	1548-7091	en_US
dc.identifier.uri	http://hdl.handle.net/10453/124093
dc.description.abstract	Methods for assembly, taxonomic profiling and binning are key to interpreting metagenome data, but a lack of consensus about benchmarking complicates performance assessment. The Critical Assessment of Metagenome Interpretation (CAMI) challenge has engaged the global developer community to benchmark their programs on highly complex and realistic data sets, generated from ∼700 newly sequenced microorganisms and ∼600 novel viruses and plasmids and representing common experimental setups. Assembly and genome binning programs performed well for species represented by individual genomes but were substantially affected by the presence of related strains. Taxonomic profiling and binning programs were proficient at high taxonomic ranks, with a notable performance decrease below family level. Parameter settings markedly affected performance, underscoring their importance for program reproducibility. The CAMI results highlight current challenges but also provide a roadmap for software selection to answer specific research questions.	en_US
dc.relation	http://purl.org/au-research/grants/arc/LP150100912
dc.relation.ispartof	Nature Methods	en_US
dc.relation.isbasedon	10.1038/nmeth.4458	en_US
dc.subject.classification	Developmental Biology	en_US
dc.subject.mesh	Sequence Analysis, DNA	en_US
dc.subject.mesh	Algorithms	en_US
dc.subject.mesh	Software	en_US
dc.subject.mesh	Benchmarking	en_US
dc.subject.mesh	Metagenomics	en_US
dc.title	Critical Assessment of Metagenome Interpretation - A benchmark of metagenomics software	en_US
dc.type	Journal Article
utslib.citation.volume	11	en_US
utslib.citation.volume	14	en_US
utslib.for	0604 Genetics	en_US
utslib.for	1004 Medical Biotechnology	en_US
utslib.for	1101 Medical Biochemistry and Metabolomics	en_US
utslib.for	06 Biological Sciences	en_US
utslib.for	10 Technology	en_US
utslib.for	11 Medical and Health Sciences	en_US
pubs.embargo.period	Not known	en_US
pubs.organisational-group	/University of Technology Sydney
pubs.organisational-group	/University of Technology Sydney/Faculty of Science
pubs.organisational-group	/University of Technology Sydney/Strength - ithree - Institute of Infection, Immunity and Innovation
utslib.copyright.status	open_access
pubs.issue	11	en_US
pubs.publication-status	Published	en_US
pubs.volume	14	en_US

Abstract:

Methods for assembly, taxonomic profiling and binning are key to interpreting metagenome data, but a lack of consensus about benchmarking complicates performance assessment. The Critical Assessment of Metagenome Interpretation (CAMI) challenge has engaged the global developer community to benchmark their programs on highly complex and realistic data sets, generated from ∼700 newly sequenced microorganisms and ∼600 novel viruses and plasmids and representing common experimental setups. Assembly and genome binning programs performed well for species represented by individual genomes but were substantially affected by the presence of related strains. Taxonomic profiling and binning programs were proficient at high taxonomic ranks, with a notable performance decrease below family level. Parameter settings markedly affected performance, underscoring their importance for program reproducibility. The CAMI results highlight current challenges but also provide a roadmap for software selection to answer specific research questions.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/121935