Is More Data Worth the Cost? Dataset Scaling Laws in a Tiny Attention-Only Decoder
Journal
2026 IEEE Swiss Conference on Data Science and AI (SDS)
ISSN
2835-3412
ISSN-Digital
2835-3420
ISBN
979-8-3195-0600-9
Type
conference paper
Date Issued
2026-05-06
Author(s)
;
;
Städeli, Rico
;
;
;
Research Team
Quantitative Data Science
Abstract
Training Transformer language models is expensive, as performance typically improves with increasing dataset size and computational budget. Although scaling laws describe this trend at large scale, their implications in controlled, smaller-scale settings remain less explored. In this work, we isolate dataset-size effects using a strongly reduced attention-only decoder architecture. By training on progressively larger power-of-two subsets, we observe smooth performance improvements accompanied by clear diminishing returns, consistent with scaling-law behavior. Using only about 30 % of the training data is sufficient to reach approximately 90 % of the full-data validation token-level accuracy. These results provide actionable insights into dataset scaling in a controlled, component-isolated setting and offer practical guidance for balancing dataset size and computational cost in compute- and data-restricted environments, such as small research labs and exploratory model development.
Language
English
HSG Classification
contribution to education
Refereed
Yes
Book title
2026 IEEE Swiss Conference on Data Science and AI (SDS)
Publisher
IEEE
Publisher place
Zurich, Switzerland
Start page
83
End page
90
Pages
8
Event Title
2026 IEEE Swiss Conference on Data Science and AI (SDS)
Event Location
Zurich, Switzerland
Event Date
6-7 May 2026
Official URL
Subject(s)
File(s)![Thumbnail Image]()
Name
Is_More_Data_Worth_the_Cost_Dataset_Scaling_Laws_in_a_Tiny_Attention-Only_Decoder.pdf
Type
Main Article
Size
1.44 MB
Format
Adobe PDF
Checksum (MD5)
fb6771d442f047dc7bcb0833b9e0de96