Beyond accuracy: Evaluating human forecast adjustments in a multidimensional forecast quality framework
Type
conference paper
Date Issued
2026
Author(s)
Abstract
Forecast evaluation has traditionally focused on accuracy as the primary performance criterion. Yet accuracy alone fails to capture all quality dimensions that matter in practice. In many settings, small errors carry limited operational consequences, whereas large errors are disproportionately costly, making reliability, the absence of extreme forecast errors, a critical concern. Likewise, excessive variation across forecasts issued at different horizons for the same target period can destabilize production and procurement plans, rendering stability an equally important consideration. This paper proposes a multidimensional framework that integrates accuracy, reliability, and stability into a unified assessment of forecast quality. To identify Pareto-efficient trade-offs across these dimensions, we develop a Data Envelopment Analysis (DEA) based approach, treating each customer–SKU time series as a decision-making unit and evaluating both system-generated and human-adjusted forecasts relative to the efficient frontier. This approach enables a nuanced assessment of whether judgmental adjustments improve forecast quality. We apply the framework to a real-world demand planning dataset from a food manufacturing company, comprising over 3,000 customer–SKU monthly time series observed over three years. Complementary qualitative interviews with demand planners confirm the practical relevance of the proposed quality dimensions and reveal inherent tensions among them. Our preliminary results indicate that human adjustments improve accuracy for a majority of time series but tend to reduce reliability and stability, which deteriorate in more than half of the cases. Notably, simultaneous improvement across all three dimensions is very uncommon, while the most frequent pattern among accuracy-improving adjustments is a concurrent worsening of both reliability and stability. The study contributes to the forecasting literature in two ways. First, it demonstrates that accuracy-focused evaluation gives an incomplete and potentially misleading picture of the value added by judgmental intervention, as improvements in one quality dimension may come at the expense of others. Second, it offers a practical, transferable methodology for multidimensional forecast evaluation, providing a foundation for more comprehensive comparisons of forecasting approaches.
Language
English
HSG Classification
contribution to scientific community
Refereed
No
Event Title
International Symposium on Forecasting
Event Location
Montreal, Canada
Event Date
28.06.2026 - 01.07.2026
Subject(s)
Division(s)