ChatGPT versus engineering education assessment: a multidisciplinary and multi-institutional benchmarking and analysis of this generative artificial intelligence tool to investigate assessment integrity

Nikolic, S; Daniel, S; Haque, R; Belkina, M; Hassan, GM; Grundy, S; Lyden, S; Neal, P; Sandison, C

ChatGPT versus engineering education assessment: a multidisciplinary and multi-institutional benchmarking and analysis of this generative artificial intelligence tool to investigate assessment integrity

Nikolic, S Daniel, S

Haque, R Belkina, M Hassan, GM Grundy, S Lyden, S Neal, P Sandison, C

Permalink

Publisher:: TAYLOR & FRANCIS LTD
Publication Type:: Journal Article
Citation:: European Journal of Engineering Education, 2023, 48, (4), pp. 559-614
Issue Date:: 2023-01-01

Open Access

Copyright Clearance Process

Recently Added
In Progress
Open Access

This item is open access.

Adobe PDF

Download Published versionAdobe PDF (4.83 MB)

View on publisher's site

View statistics

Full metadata record

Field	Value	Language
dc.contributor.author	Nikolic, S
dc.contributor.author	Daniel, S https://orcid.org/0000-0002-7528-9713
dc.contributor.author	Haque, R
dc.contributor.author	Belkina, M
dc.contributor.author	Hassan, GM
dc.contributor.author	Grundy, S
dc.contributor.author	Lyden, S
dc.contributor.author	Neal, P
dc.contributor.author	Sandison, C
dc.date.accessioned	2024-03-12T05:30:17Z
dc.date.available	2024-03-12T05:30:17Z
dc.date.issued	2023-01-01
dc.identifier.citation	European Journal of Engineering Education, 2023, 48, (4), pp. 559-614
dc.identifier.issn	0304-3797
dc.identifier.issn	1469-5898
dc.identifier.uri	http://hdl.handle.net/10453/176575
dc.description.abstract	ChatGPT, a sophisticated online chatbot, sent shockwaves through many sectors once reports filtered through that it could pass exams. In higher education, it has raised many questions about the authenticity of assessment and challenges in detecting plagiarism. Amongst the resulting frenetic hubbub, hints of potential opportunities in how ChatGPT could support learning and the development of critical thinking have also emerged. In this paper, we examine how ChatGPT may affect assessment in engineering education by exploring ChatGPT responses to existing assessment prompts from ten subjects across seven Australian universities. We explore the strengths and weaknesses of current assessment practice and discuss opportunities on how ChatGPT can be used to facilitate learning. As artificial intelligence is rapidly improving, this analysis sets a benchmark for ChatGPT’s performance as of early 2023 in responding to engineering education assessment prompts. ChatGPT did pass some subjects and excelled with some assessment types. Findings suggest that changes in current practice are needed, as typically with little modification to the input prompts, ChatGPT could generate passable responses to many of the assessments, and it is only going to get better as future versions are trained on larger data sets.
dc.language	English
dc.publisher	TAYLOR & FRANCIS LTD
dc.relation.ispartof	European Journal of Engineering Education
dc.relation.isbasedon	10.1080/03043797.2023.2213169
dc.rights	info:eu-repo/semantics/openAccess
dc.subject	09 Engineering, 13 Education
dc.subject.classification	39 Education
dc.subject.classification	40 Engineering
dc.title	ChatGPT versus engineering education assessment: a multidisciplinary and multi-institutional benchmarking and analysis of this generative artificial intelligence tool to investigate assessment integrity
dc.type	Journal Article
utslib.citation.volume	48
utslib.for	09 Engineering
utslib.for	13 Education
pubs.organisational-group	University of Technology Sydney
pubs.organisational-group	University of Technology Sydney/Faculty of Engineering and Information Technology
pubs.organisational-group	University of Technology Sydney/Faculty of Engineering and Information Technology/School of Professional Practice and Leadership
utslib.copyright.status	open_access	*
dc.date.updated	2024-03-12T05:30:16Z
pubs.issue	4
pubs.publication-status	Published
pubs.volume	48
utslib.citation.issue	4

Abstract:

ChatGPT, a sophisticated online chatbot, sent shockwaves through many sectors once reports filtered through that it could pass exams. In higher education, it has raised many questions about the authenticity of assessment and challenges in detecting plagiarism. Amongst the resulting frenetic hubbub, hints of potential opportunities in how ChatGPT could support learning and the development of critical thinking have also emerged. In this paper, we examine how ChatGPT may affect assessment in engineering education by exploring ChatGPT responses to existing assessment prompts from ten subjects across seven Australian universities. We explore the strengths and weaknesses of current assessment practice and discuss opportunities on how ChatGPT can be used to facilitate learning. As artificial intelligence is rapidly improving, this analysis sets a benchmark for ChatGPT’s performance as of early 2023 in responding to engineering education assessment prompts. ChatGPT did pass some subjects and excelled with some assessment types. Findings suggest that changes in current practice are needed, as typically with little modification to the input prompts, ChatGPT could generate passable responses to many of the assessments, and it is only going to get better as future versions are trained on larger data sets.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/176575