Tomas Hrycej

Title

Dr.

Last Name

Hrycej

First name

Tomas

tomas.hrycej@unisg.ch

Now showing 1 - 4 of 4

Some of the metrics are blocked by your
consent settings
Reducing the Transformer Architecture to a Minimum
(SCITEPRESS - Science and Technology Publications, 2024-11)
Bernhard Bermeitinger
;
Tomas Hrycej
;
Massimo Pavone
;
Julianus Kath
;
Siegfried Handschuh
Transformers are a widespread and successful model architecture, particularly in Natural Language Processing (NLP) and Computer Vision (CV). The essential innovation of this architecture is the Attention Mechanism, which solves the problem of extracting relevant context information from long sequences in NLP and realistic scenes in CV. A classical neural network component, a Multi-Layer Perceptron (MLP), complements the attention mechanism. Its necessity is frequently justified by its capability of modeling nonlinear relationships. However, the attention mechanism itself is nonlinear through its internal use of similarity measures. A possible hypothesis is that this nonlinearity is sufficient for modeling typical application problems. As the MLPs usually contain the most trainable parameters of the whole model, their omission would substantially reduce the parameter set size. Further components can also be reorganized to reduce the number of parameters. Under some conditions, query and key matrices can be collapsed into a single matrix of the same size. The same is true about value and projection matrices, which can also be omitted without eliminating the substance of the attention mechanism. Initially, the similarity measure was defined asymmetrically, with peculiar properties such as that a token is possibly dissimilar to itself. A possible symmetric definition requires only half of the parameters. All these parameter savings make sense only if the representational performance of the architecture is not significantly reduced. A comprehensive empirical proof for all important domains would be a huge task. We have laid the groundwork by testing widespread CV benchmarks: MNIST, CIFAR-10, and, with restrictions, ImageNet. The tests have shown that simplified transformer architectures (a) without MLP, (b) with collapsed matrices, and (c) symmetric similarity matrices exhibit similar performance as the original architecture, saving up to 90 % of parameters without hurting the classification performance.
Type:conference paper
Journal:Proceedings of the 16th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management
URL:https://www.scitepress.org/PublicationsDetail.aspx?ID=REzv8Neq7FY=&t=1
DOI:10.5220/0012891000003838
URI:https://www.alexandria.unisg.ch/handle/20.500.14171/121344
Scopus© Citations 2
Some of the metrics are blocked by your
consent settings
Efficient Neural Network Training via Subset Pretraining
(SCITEPRESS - Science and Technology Publications, 2024-11-22)
Jan Spörer
;
Bernhard Bermeitinger
;
Tomas Hrycej
;
Limacher, Niklas
;
Siegfried Handschuh
In training neural networks, it is common practice to use partial gradients computed over batches, mostly very small subsets of the training set. This approach is motivated by the argument that such a partial gradient is close to the true one, with precision growing only with the square root of the batch size. A theoretical justification is with the help of stochastic approximation theory. However, the conditions for the validity of this theory are not satisfied in the usual learning rate schedules. Batch processing is also difficult to combine with efficient second-order optimization methods. This proposal is based on another hypothesis: the loss minimum of the training set can be expected to be well-approximated by the minima of its subsets. Such subset minima can be computed in a fraction of the time necessary for optimizing over the whole training set. This hypothesis has been tested with the help of the MNIST, CIFAR-10, and CIFAR-100 image classification benchmarks, optionally extended by training data augmentation. The experiments have confirmed that results equivalent to conventional training can be reached. In summary, even small subsets are representative if the overdetermination ratio for the given model parameter set sufficiently exceeds unity. The computing expense can be reduced to a tenth or less.
Type:conference paper
Journal:Proceedings of the 16th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management
URL:https://www.scitepress.org/PublicationsDetail.aspx?ID=EvHJGVpxuVs=
DOI:10.5220/0012893600003838
URI:https://www.alexandria.unisg.ch/handle/20.500.14171/121343
Scopus© Citations 3
Some of the metrics are blocked by your
consent settings
Efficient Neural Network Training via Subset Pretraining
(SciTePress, 2024-11)
Jan Spörer
;
Bernhard Bermeitinger
;
Tomas Hrycej
;
Niklas Limacher
;
Siegfried Handschuh
In training neural networks, it is common practice to use partial gradients computed over batches, mostly very small subsets of the training set. This approach is motivated by the argument that such a partial gradient is close to the true one, with precision growing only with the square root of the batch size. A theoretical justification is with the help of stochastic approximation theory. However, the conditions for the validity of this theory are not satisfied in the usual learning rate schedules. Batch processing is also difficult to combine with efficient second-order optimization methods. This proposal is based on another hypothesis: the loss minimum of the training set can be expected to be well-approximated by the minima of its subsets. Such subset minima can be computed in a fraction of the time necessary for optimizing over the whole training set. This hypothesis has been tested with the help of the MNIST, CIFAR-10, and CIFAR-100 image classification benchmarks, optionally extended by training data augmentation. The experiments have confirmed that results equivalent to conventional training can be reached. In summary, even small subsets are representative if the overdetermination ratio for the given model parameter set sufficiently exceeds unity. The computing expense can be reduced to a tenth or less.
Type:conference paper
Journal:Proceedings of the 16th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management
URL:https://www.scitepress.org/Link.aspx?doi=10.5220%2f0012893600003838
DOI:10.5220/0012893600003838
URI:https://www.alexandria.unisg.ch/handle/20.500.14171/123208
Scopus© Citations 3
Some of the metrics are blocked by your
consent settings
Make Deep Networks Shallow Again
(SciTePress, 2023-11-15)
Bernhard Bermeitinger
;
Tomas Hrycej
;
Siegfried Handschuh
Deep neural networks have a good success record and are thus viewed as the best architecture choice for complex applications. Their main shortcoming has been, for a long time, the vanishing gradient which prevented the numerical optimization algorithms from acceptable convergence. An important special case of network architecture, frequently used in computer vision applications, consists of using a stack of layers of the same dimension. For this architecture, a breakthrough has been achieved by the concept of residual connections-an identity mapping parallel to a conventional layer. This concept substantially alleviates the vanishing gradient problem and is thus widely used. The focus of this paper is to show the possibility of substituting the deep stack of residual layers with a shallow architecture with comparable expressive power and similarly good convergence properties. A stack of residual layers can be expressed as an expansion of terms similar to the Taylor expansion. This expansion suggests the possibility of truncating the higher-order terms and receiving an architecture consisting of a single broad layer composed of all initially stacked layers in parallel. In other words, a sequential deep architecture is substituted by a parallel shallow one. Prompted by this theory, we investigated the performance capabilities of the parallel architecture in comparison to the sequential one. The computer vision datasets MNIST and CIFAR10 were used to train both architectures for a total of 6,912 combinations of varying numbers of convolutional layers, numbers of filters, kernel sizes, and other meta parameters. Our findings demonstrate a surprising equivalence between the deep (sequential) and shallow (parallel) architectures. Both layouts produced similar results in terms of training and validation set loss. This discovery implies that a wide, shallow architecture can potentially replace a deep network without sacrificing performance. Such substitution has the potential to simplify network architectures, improve optimization efficiency, and accelerate the training process.
Type:conference paper
Journal:Proceedings of the 15th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management
URL:https://www.scitepress.org/PublicationsDetail.aspx?ID=njj8erFelyk%3d&t=1
DOI:10.5220/0012203800003598
URI:https://www.alexandria.unisg.ch/handle/20.500.14171/118680
Scopus© Citations 1

Tomas Hrycej

Filters

Settings

Sort By

Results per page