Konstantin Schürholt
Thumbnail Image
Last Name
Schürholt
First name
Konstantin
Email
konstantin.schuerholt@unisg.ch
Phone
+41 71 224 34 54

Filters

5
Reset filters

Settings
Now showing 1 - 5 of 5
  • Publication
    Towards Scalable and Versatile Weight Space Learning
    (PMLR 235, 2024, 2024-07-23)
    Konstantin Schürholt
    ;
    Michael W. Mahoney
    ;
    Damian Borth
    Learning representations of well-trained neural network models holds the promise to provide an understanding of the inner workings of those models. However, previous work has either faced limitations when processing larger networks or was task-specific to either discriminative or generative tasks. This paper introduces the SANE approach to weight-space learning. SANE overcomes previous limitations by learning task-agnostic representations of neural networks that are scalable to larger models of varying architectures and that show capabilities beyond a single task. Our method extends the idea of hyper-representations towards sequential processing of subsets of neural network weights, thus allowing one to embed larger neural networks as a set of tokens into the learned representation space. SANE reveals global model information from layer-wise embeddings, and it can sequentially generate unseen neural network models, which was unattainable with previous hyper-representation learning methods. Extensive empirical evaluation demonstrates that SANE matches or exceeds state-of-the art performance on several weight representation learning benchmarks, particularly in initialization for new tasks and larger ResNet architectures.
    Type: conference paper
    Volume: Proceedings of the 41st International Conference on Machine Learning, Vienna, Austria.
    URL: https://icml.cc/virtual/2024/poster/32815
    URI: https://www.alexandria.unisg.ch/handle/20.500.14171/120512
  • Publication
    Sparsified Model Zoo Twins: Investigating Populations of Sparsified Neural Network Models
    ( 2023)
    Honegger, Dominik
    ;
    Konstantin Schürholt
    ;
    Damian Borth
    With growing size of Neural Networks (NNs), model sparsification to reduce the computational cost and memory demand for model inference has become of vital interest for both research and production. While many sparsification methods have been proposed and successfully applied on individual models, to the best of our knowledge their behavior and robustness has not yet been studied on large populations of models. With this paper, we address that gap by applying two popular sparsification methods on populations of models (so called model zoos) to create sparsified versions of the original zoos. We investigate the performance of these two methods for each zoo, compare sparsification layer-wise, and analyse agreement between original and sparsified populations. We find both methods to be very robust with magnitude pruning able outperform variational dropout with the exception of high sparsification ratios above 80%. Further, we find sparsified models agree to a high degree with their original non-sparsified counterpart, and that the performance of original and sparsified model is highly correlated. Finally, all models of the model zoos and their sparsified model twins are publicly available: modelzoos.cc.
    Type: conference paper
    Journal: ICLR 2023 Workshop on Sparsity in Neural Networks
    URI: https://www.alexandria.unisg.ch/handle/20.500.14171/118949
  • Publication
    Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights
    (Curran Associates, Inc., 2022-11)
    Schürholt, Konstantin
    ;
    Knyazev, Boris
    ;
    Giro-i-Nieto, Xavier
    ;
    Borth, Damian
    Learning representations of neural network weights given a model zoo is an emerg- ing and challenging area with many potential applications from model inspection, to neural architecture search or knowledge distillation. Recently, an autoencoder trained on a model zoo was able to learn a hyper-representation, which captures intrinsic and extrinsic properties of the models in the zoo. In this work, we ex- tend hyper-representations for generative use to sample new model weights. We propose layer-wise loss normalization which we demonstrate is key to generate high-performing models and several sampling methods based on the topology of hyper-representations. The models generated using our methods are diverse, per- formant and capable to outperform strong baselines as evaluated on several down- stream tasks: initialization, ensemble sampling and transfer learning. Our results indicate the potential of knowledge aggregation from model zoos to new models via hyper-representations thereby paving the avenue for novel research directions.
    Type: conference paper
    Volume: 35
    URI: https://www.alexandria.unisg.ch/handle/20.500.14171/108129
  • Publication
    Model Zoos: A Dataset of Diverse Populations of Neural Network Models
    (36th Conference on Neural Information Processing Systems (NeurIPS 2022) Track on Datasets and Benchmarks., 2022-11)
    Schürholt, Konstantin
    ;
    Taskiran, Diyar
    ;
    Knyazev, Boris
    ;
    Giro-i-Nieto, Xavier
    ;
    Borth, Damian
    In the last years, neural networks (NN) have evolved from laboratory environments to the state-of-the-art for many real-world problems. It was shown that NN models (i.e., their weights and biases) evolve on unique trajectories in weight space during training. Following, a population of such neural network models (referred to as model zoo) would form structures in weight space. We think that the geometry, curvature and smoothness of these structures contain information about the state of training and can reveal latent properties of individual models. With such model zoos, one could investigate novel approaches for (i) model analysis, (ii) discover unknown learning dynamics, (iii) learn rich representations of such populations, or (iv) exploit the model zoos for generative modelling of NN weights and biases. Unfortunately, the lack of standardized model zoos and available benchmarks significantly increases the friction for further research about populations of NNs. With this work, we publish a novel dataset of model zoos containing systematically generated and diverse populations of NN models for further research. In total the proposed model zoo dataset is based on eight image datasets, consists of 27 model zoos trained with varying hyperparameter combinations and includes 50’360 unique NN models as well as their sparsified twins, resulting in over 3’844’360 collected model states. Additionally, to the model zoo data we provide an in-depth analysis of the zoos and provide benchmarks for multiple downstream tasks. The dataset can be found at www.modelzoos.cc.
    Type: conference paper
    URI: https://www.alexandria.unisg.ch/handle/20.500.14171/108130
  • Publication
    Self-Supervised Representation Learning on Neural Network Weights for Model Characteristic Prediction
    (Neural Information Processing Systems (NeurIPS), 2021-11-09)
    Schürholt, Konstantin
    ;
    Kostadinov, Dimche
    ;
    Borth, Damian
    Self-Supervised Learning (SSL) has been shown to learn useful and information- preserving representations. Neural Networks (NNs) are widely applied, yet their weight space is still not fully understood. Therefore, we propose to use SSL to learn neural representations of the weights of populations of NNs. To that end, we introduce domain specific data augmentations and an adapted attention architecture. Our empirical evaluation demonstrates that self-supervised representation learning in this domain is able to recover diverse NN model characteristics. Further, we show that the proposed learned representations outperform prior work for predicting hyper-parameters, test accuracy, and generalization gap as well as transfer to out-of-distribution settings.
    Type: conference paper
    Volume: 35
    URI: https://www.alexandria.unisg.ch/handle/20.500.14171/109739