Enhancing Biomedical Text Summarization and Question-Answering: On the Utility of Domain-Specific Pre-Training University of Technology Sydney participation in BioASQ Task 11b Phase B

Galat, D; Rizoiu, MA

Enhancing Biomedical Text Summarization and Question-Answering: On the Utility of Domain-Specific Pre-Training University of Technology Sydney participation in BioASQ Task 11b Phase B

Galat, D

Rizoiu, MA

Permalink

Publication Type:: Conference Proceeding
Citation:: CEUR Workshop Proceedings, 2023, 3497, pp. 102-113
Issue Date:: 2023-01-01

Open Access

Copyright Clearance Process

Recently Added
In Progress
Open Access

This item is open access.

Adobe PDF

Download Published VersionAdobe PDF (1.36 MB)

View statistics

Full metadata record

Field	Value	Language
dc.contributor.author	Galat, D https://orcid.org/0000-0003-3825-2142
dc.contributor.author	Rizoiu, MA
dc.date.accessioned	2023-11-16T14:55:06Z
dc.date.available	2023-11-16T14:55:06Z
dc.date.issued	2023-01-01
dc.identifier.citation	CEUR Workshop Proceedings, 2023, 3497, pp. 102-113
dc.identifier.issn	1613-0073
dc.identifier.uri	http://hdl.handle.net/10453/173439
dc.description.abstract	Biomedical summarization requires large datasets to train for text generation. We show that while transfer learning offers a viable option for addressing this challenge, an in-domain pre-training does not always offer advantages in a BioASQ summarization task. We identify a suitable model architecture and use it to show a benefit of a general-domain pre-training followed by a task-specific fine-tuning in the context of a BioASQ summarization task, leading to a novel three-step fine-tuning approach that works with only a thousand in-domain examples. Our results indicate that a Large Language Model without domain-specific pre-training can have a significant edge in some domain-specific biomedical text generation tasks.
dc.language	en
dc.relation.ispartof	CEUR Workshop Proceedings
dc.rights	info:eu-repo/semantics/restrictedAccess
dc.rights	© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
dc.subject.classification	4609 Information systems
dc.title	Enhancing Biomedical Text Summarization and Question-Answering: On the Utility of Domain-Specific Pre-Training University of Technology Sydney participation in BioASQ Task 11b Phase B
dc.type	Conference Proceeding
utslib.citation.volume	3497
pubs.organisational-group	/University of Technology Sydney
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology/School of Computer Science
utslib.copyright.status	open_access	*
dc.date.updated	2023-11-16T14:54:57Z
pubs.publication-status	Published
pubs.volume	3497

Abstract:

Biomedical summarization requires large datasets to train for text generation. We show that while transfer learning offers a viable option for addressing this challenge, an in-domain pre-training does not always offer advantages in a BioASQ summarization task. We identify a suitable model architecture and use it to show a benefit of a general-domain pre-training followed by a task-specific fine-tuning in the context of a BioASQ summarization task, leading to a novel three-step fine-tuning approach that works with only a thousand in-domain examples. Our results indicate that a Large Language Model without domain-specific pre-training can have a significant edge in some domain-specific biomedical text generation tasks.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/173439