Training Neural Networks in Single vs. Double Precision
Journal
Proceedings of the 14th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - KDIR
ISSN
2184-3228
ISSN-Digital
2184-3228
Type
conference paper
Date Issued
2022-10
Author(s)
Research Team
Data Science and Natural Language Processing
Abstract
The commitment to single-precision floating-point arithmetic is widespread in the deep learning community. To evaluate whether this commitment is justified, the influence of computing precision (single and double precision) on the optimization performance of the Conjugate Gradient (CG) method (a second-order optimization algorithm) and Root Mean Square Propagation (RMSprop) (a first-order algorithm) has been investigated. Tests of neural networks with one to five fully connected hidden layers and moderate or strong nonlinearity with up to 4 million network parameters have been optimized for Mean Square Error (MSE). The training tasks have been set up so that their MSE minimum was known to be zero. Computing experiments have dis-closed that single-precision can keep up (with superlinear convergence) with double-precision as long as line search finds an improvement. First-order methods such as RMSprop do not benefit from double precision. However, for moderately nonlinear tasks, CG is clearly superior. For strongly nonlinear tasks, both algorithm classes find only solutions fairly poor in terms of mean square error as related to the output variance. CG with double floating-point precision is superior whenever the solutions have the potential to be useful for the application goal.
Language
English
HSG Classification
contribution to scientific community
HSG Profile Area
None
Refereed
Yes
Publisher
SciTePress
Start page
307
End page
314
Subject(s)
Division(s)
Contact Email Address
bernhard.bermeitinger@unisg.ch
Eprints ID
267727
File(s)![Thumbnail Image]()
![Thumbnail Image]()
restricted
Name
115779.pdf
Size
453.94 KB
Format
Adobe PDF
Checksum (MD5)
c5139404431c5976a045bbd329c9d63d
open.access
Name
TrainingNeuralNetworksinSinglevs.DoublePrecisionKDIR2022Valetta.pdf
Size
944.02 KB
Format
Adobe PDF
Checksum (MD5)
a44efeec2ec903af833b096a9ba7f3ef