The Impact of Model Zoo Size and Composition on Weight Space Learning
Journal
ICLR Workshop on Neural Network Weights as a New Data Modality
Type
conference contribution
Date Issued
2025-04-14
Author(s)
Abstract
Re-using trained neural network models is a common strategy to reduce training cost and transfer knowledge. Weight space learning - using the weights of trained models as data modality - is a promising new field to re-use populations of pre-trained models for future tasks. Approaches in this field have demonstrated high performance both on model analysis and weight generation
tasks. However, until now their learning setup requires homogeneous model zoos where all models share the same exact architecture, limiting their capability to generalize beyond the population of models they saw during training. In this work, we remove this constraint and propose a modification to a common weight space learning method to accommodate training on heterogeneous populations of models. We further investigate the resulting impact of model diversity on generating unseen neural network model weights for zero-shot knowledge
transfer. Our extensive experimental evaluation shows that including models with varying underlying image datasets has a high impact on performance and generalization, for both in- and out of-distribution settings. Code is available on github.com/HSG-AIML/MultiZoo-SANE.
tasks. However, until now their learning setup requires homogeneous model zoos where all models share the same exact architecture, limiting their capability to generalize beyond the population of models they saw during training. In this work, we remove this constraint and propose a modification to a common weight space learning method to accommodate training on heterogeneous populations of models. We further investigate the resulting impact of model diversity on generating unseen neural network model weights for zero-shot knowledge
transfer. Our extensive experimental evaluation shows that including models with varying underlying image datasets has a high impact on performance and generalization, for both in- and out of-distribution settings. Code is available on github.com/HSG-AIML/MultiZoo-SANE.
Language
English
Keywords
machine learning
deep learning
weight space learning
representation learning
Official URL
File(s)![Thumbnail Image]()
Name
2504.10141v1.pdf
Size
620.42 KB
Format
Adobe PDF
Checksum (MD5)
a9fcae59c41a45704564f300e697429a