Subset Pretraining for Enhancing Neural Network Training Efficiency

Journal
Communications in Computer and Information Science
ISSN
1865-0929
ISSN-Digital
1865-0937
ISBN
978-3-032-06878-1
Type
conference paper
Date Issued
2025-10-24
Author(s)
;
;
;
DOI
Abstract
We propose a novel alternative to traditional randomly sampled mini-batches for gradient computation: using a fixed subset for complete pretraining of a neural network model. This approach enables deterministic convergence instead of a merely probabilistic one, as proven by the stochastic approximation theory, whose prerequisites are frequently violated by popular optimization algorithms. The approach is justified by the hypothesis that the loss minimum of the training set can be expected to be well-approximated by the minima of its subsets. Such subset minima can be computed in a fraction of the time necessary for optimizing with the whole training set. They are also compatible with efficient second-order optimization methods, such as the conjugate gradient optimizer. These methods are particularly efficient in the convex environment of the loss minimum. The image classification datasets MNIST, CIFAR-10, and CIFAR-100, (optionally extended by augmentation of training data) test this hypothesis. The experiments confirm that the models achieve performance equivalent to that when trained with the conventional training scheme. In conclusion, if the overdetermination ratio for the given model and dataset sufficiently exceed unity, even small subsets are representative. This results in a possible reduction of the computing expense to a tenth or less.

This paper is an extended version of Spörer et al. [13].
Language
English
Refereed
Yes
Publisher
Springer
Publisher place
Cham
Volume
2703
Start page
22
End page
36
Pages
8
Event Title
International Joint Conference on Knowledge Discovery, Knowledge Engineering, and Knowledge Management
Event Location
Porto, Portugal
Event Date
17-19 November 2024
Official URL
URL
Contact Email Address
bernhard.bermeitinger@unisg.ch
Additional Information
Postpublication