Generating Handwritten Mathematical Expressions from Symbol Graphs: An End-to-End Pipeline

Chen, Y; Gao, F; Zhang, Y; Qiao, M; Wang, N

Generating Handwritten Mathematical Expressions from Symbol Graphs: An End-to-End Pipeline

Chen, Y Gao, F Zhang, Y Qiao, M

Wang, N

Permalink

Publisher:: Institute of Electrical and Electronics Engineers (IEEE)
Publication Type:: Conference Proceeding
Citation:: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2024, 00, pp. 15675-15685
Issue Date:: 2024-01-01

Recently Added

	Filename	Description	Size
	Chen_Generating_Handwritten_Mathematical_Expressions_From_Symbol_Graphs_An_End-to-End_Pipeline_CVPR_2024_paper.pdf	Published version	4.01 MB	Adobe PDF	View/Open

Copyright Clearance Process

Recently Added
In Progress
Open Access

This item is new to OPUS and is not currently available.

Full metadata record

Field	Value	Language
dc.contributor.author	Chen, Y
dc.contributor.author	Gao, F
dc.contributor.author	Zhang, Y
dc.contributor.author	Qiao, M https://orcid.org/0000-0002-0990-5506
dc.contributor.author	Wang, N
dc.date	2024-06-16
dc.date.accessioned	2025-04-05T05:58:07Z
dc.date.available	2025-04-05T05:58:07Z
dc.date.issued	2024-01-01
dc.identifier.citation	Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2024, 00, pp. 15675-15685
dc.identifier.issn	1063-6919
dc.identifier.uri	http://hdl.handle.net/10453/186651
dc.description.abstract	In this paper, we explore a novel challenging generation task, i.e. Handwritten Mathematical Expression Generation (HMEG) from symbolic sequences. Since symbolic sequences are naturally graph-structured data, we formulate HMEG as a graph-to-image (G2I) generation problem. Unlike the generation of natural images, HMEG requires critic layout clarity for synthesizing correct and recognizable formulas, but has no real masks available to supervise the learning process. To alleviate this challenge, we propose a novel end-to-end G2I generation pipeline (i.e. graph → layout →mask →image), which requires no real masks or nondifferentiable alignment between layouts and masks. Technically, to boost the capacity of predicting detailed relations among adjacent symbols, we propose a Less-is-More (LiM) learning strategy. In addition, we design a differentiable layout refinement module, which maps bounding boxes to pixel-level soft masks, so as to further alleviate ambiguous layout areas. Our whole model, including layout prediction, mask refinement, and image generation, can be jointly optimized in an end-to-end manner. Experimental results show that, our model can generate highquality HME images, and outperforms previous generative methods. Besides, a series of ablations study demonstrate effectiveness of the proposed techniques. Finally, we validate that our generated images promisingly boosts the performance of HME recognition models, through data augmentation. Our code and results are available at: https://github.com/AiArt-HDU/HMEG.
dc.language	en
dc.publisher	Institute of Electrical and Electronics Engineers (IEEE)
dc.relation.ispartof	Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
dc.relation.ispartof	2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
dc.relation.isbasedon	10.1109/CVPR52733.2024.01484
dc.rights	info:eu-repo/semantics/restrictedAccess
dc.title	Generating Handwritten Mathematical Expressions from Symbol Graphs: An End-to-End Pipeline
dc.type	Conference Proceeding
utslib.citation.volume	00
pubs.organisational-group	University of Technology Sydney
pubs.organisational-group	University of Technology Sydney/Faculty of Engineering and Information Technology
pubs.organisational-group	University of Technology Sydney/Faculty of Engineering and Information Technology/School of Computer Science
pubs.organisational-group	University of Technology Sydney/UTS Groups
pubs.organisational-group	University of Technology Sydney/UTS Groups/Australian Artificial Intelligence Institute (AAII)
utslib.copyright.status	recently_added	*
dc.date.updated	2025-04-05T05:58:05Z
pubs.finish-date	2024-06-22
pubs.publication-status	Published
pubs.start-date	2024-06-16
pubs.volume	00

Abstract:

In this paper, we explore a novel challenging generation task, i.e. Handwritten Mathematical Expression Generation (HMEG) from symbolic sequences. Since symbolic sequences are naturally graph-structured data, we formulate HMEG as a graph-to-image (G2I) generation problem. Unlike the generation of natural images, HMEG requires critic layout clarity for synthesizing correct and recognizable formulas, but has no real masks available to supervise the learning process. To alleviate this challenge, we propose a novel end-to-end G2I generation pipeline (i.e. graph → layout →mask →image), which requires no real masks or nondifferentiable alignment between layouts and masks. Technically, to boost the capacity of predicting detailed relations among adjacent symbols, we propose a Less-is-More (LiM) learning strategy. In addition, we design a differentiable layout refinement module, which maps bounding boxes to pixel-level soft masks, so as to further alleviate ambiguous layout areas. Our whole model, including layout prediction, mask refinement, and image generation, can be jointly optimized in an end-to-end manner. Experimental results show that, our model can generate highquality HME images, and outperforms previous generative methods. Besides, a series of ablations study demonstrate effectiveness of the proposed techniques. Finally, we validate that our generated images promisingly boosts the performance of HME recognition models, through data augmentation. Our code and results are available at: https://github.com/AiArt-HDU/HMEG.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/186651