Linus Mathias Scheibenreif
Last Name
Scheibenreif
First name
Linus Mathias
Email
linus.scheibenreif@unisg.ch
Phone
+41 71 224 26 98
Now showing 1 - 3 of 3
- Some of the metrics are blocked by yourconsent settings
Item type:Publication, Masked Vision Transformers for Hyperspectral Image Classification(2023-06-18); ; Transformer architectures have become state-of-the-art models in computer vision and natural language processing. To a significant degree, their success can be attributed to self-supervised pre-training on large scale unlabeled datasets. This work investigates the use of self-supervised masked image reconstruction to advance transformer models for hyperspectral remote sensing imagery. To facilitate self-supervised pre-training, we build a large dataset of unlabeled hyperspectral observations from the EnMAP satellite and systematically investigate modifications of the vision transformer architecture to optimally leverage the characteristics of hyperspectral data. We find significant improvements in accuracy on different land cover classification tasks over both standard vision and sequence transformers using (i) blockwise patch embeddings, (ii) spatialspectral self-attention, (iii) spectral positional embeddings and (iv) masked self-supervised pre-training 1. The resulting model outperforms standard transformer architectures by +5% accuracy on a labeled subset of our EnMAP data and by +15% on Houston2018 hyperspectral dataset, making it competitive with a strong 3D convolutional neural network baseline. In an ablation study on label-efficiency based on the Houston2018 dataset, self-supervised pretraining significantly improves transformer accuracy when little labeled training data is available. The self-supervised model outperforms randomly initialized transformers and the 3D convolutional neural network by +7-8% when only 0.1-10% of the training labels are available.Type:conference paper - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Ben-ge: Extending BigEarthNet with Geographical and Environmental Data(2023-07-04T14:17:54Z); ; ; ; Deep learning methods have proven to be a powerful tool in the analysis of large amounts of complex Earth observation data. However, while Earth observation data are multi-modal in most cases, only single or few modalities are typically considered. In this work, we present the ben-ge dataset, which supplements the BigEarthNet-MM dataset by compiling freely and globally available geographical and environmental data. Based on this dataset, we showcase the value of combining different data modalities for the downstream tasks of patch-based land-use/land-cover classification and land-use/land-cover segmentation. ben-ge is freely available and expected to serve as a test bed for fully supervised and self-supervised Earth observation applications.Type:conference contribution - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Efficient Learning for Earth Observation Applications with Self-Supervised Learning(2023-03-22); ; ; Earth observation methods are utilized across a wide range of research disciplines and therefore play an increasingly important role. This importance is fueled in part by the ever-increasing amount of (freely) available Earth observation data. The systematic analysis of these vast amounts of data requires scalable methods that are tailored to the complexity of the data; Deep Learning meets these requirements and has therefore proven to be a useful tool. However, the supervised learning process that is typically used to train these models to perform specific tasks requires the annotation of large amounts of training data, a process that is both expensive and laborious. Recently, self-supervised learning paradigms that do not rely on annotated data have emerged and gained traction in the field of computer vision. Such methods enable the pre-training of Deep Learning models on large amounts of annotation-free data and the subsequent supervised fine-tuning of the pretrained models on specific tasks. As a result of the pretraining, this fine-tuning process requires much smaller amounts of annotated data and therefore enables a much more efficient learning process. Thus, self-supervised learning represents a worthwhile path for Earth observation applications as it is able to leverage the vast amounts of freely available Earth observation data to enable the efficient learning of related tasks. In this work, we employ self-supervised learning methods on multi-modal Earth observation data to quantify its effects on the learning efficiency. We pre-train different Deep Learning model backbones on the task of identifying matching scenes from Sentinel-1 SAR and Sentinel-2 multi-band data in a contrastive learning setup. Our results support the notion that self-supervised learning is useful for efficient learning of Earth observation tasks. We find that the pre-training process results in the successful learning of rich latent representations: the supervised fine-tuning of pre-trained models results in (i) a better performance than models that were trained in a fully supervised way while at the same time (ii) requiring only 10-20% of the annotated data. Furthermore, (iii) this effect can be observed across different downstream tasks such as patch-based classification or image segmentation, indicating that the learned latent representations are task agnostic. Based on these findings, we expect self-supervised pre-training to enable more efficient Deep Learning for Earth observation applications and therefore to boost scientific output across a range of Earth observation-related research disciplines.Type:conference contribution