Hyper-Representations for Pre-Training and Transfer Learning
Type
conference paper
Date Issued
2022
Author(s)
Abstract (De)
Learning representations of neural network weights given a model zoo is an emerging and challenging area with many potential applications from model inspection, to neural architecture search or knowledge distillation. Recently, an autoencoder trained on a model zoo was able to learn a hyper-representation, which captures intrinsic and extrinsic properties of the models in the zoo. In this work, we extend hyperrepresentations for generative use to sample new model weights as pre-training. We propose layerwise loss normalization which we demonstrate is key to generate high-performing models and a sampling method based on the empirical density of hyper-representations. The models generated using our methods are diverse, performant and capable to outperform conventional baselines for transfer learning. Our results indicate the potential of knowledge aggregation from model zoos to new models via hyper-representations thereby paving the avenue for novel research directions.
Language
English
Keywords
Computer Science - Machine Learning
HSG Classification
contribution to scientific community
Event Title
First Workshop of Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML 2022
Subject(s)
Eprints ID
267865
File(s)![Thumbnail Image]()
open.access
Name
ICML_HyperRepresentations_Generating_Neural_Network_Weights_final.pdf
Size
625.68 KB
Format
Adobe PDF
Checksum (MD5)
b525e547c347255ae0471c84f9bbd3a1