Efficient Variants of the Conjugate Gradient Algorithm for Neural Networks
Journal
Proceedings of the 18th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management
ISSN
2184-3228
ISSN-Digital
2184-3228
ISBN
978-989-758-868-6
Type
conference paper
Date Issued
2026-10
Author(s)
Abstract
The use of second-order optimization methods has been limited by their high computational expense. While recent variants focus on reducing this cost, such simplifications often risk premature termination, a more severe issue than expense itself. We identify primary causes of premature causes: simplified line search procedures and non-descending conjugate search directions. Computing experiments on a reduced GPT-2 model and Resnet-18 show the superiority of (1) a safe Golden Section line search over the Wolfe method, and (2) the Dai-Yuan beta method of the search direction, which generalizes convergence guarantees to standard Wolfe conditions.
Language
English
HSG Classification
contribution to scientific community
Refereed
Yes
Publisher
SCITEPRESS - Science and Technology Publications
Publisher place
Portugal
Start page
267
End page
274
Pages
8
Event Location
Angers, France
Subject(s)
File(s)![Thumbnail Image]()
Name
153417.pdf
Type
Main Article
Size
1.46 MB
Format
Adobe PDF
Checksum (MD5)
0203016db9ba43228445eb3fc86a2fe4