Browsing by Person "Puke, Marius"

Now showing 1 - 1 of 1

Robust evaluation of point and probability predictions in economics and finance
(2025) Puke, Marius; Jung, Robert
In economics and finance, forecasts guide many critical decisions, making their evaluation indispensable. Evaluation can be absolute, focusing on the calibration of an individual sequence of forecasts, or relative, focusing on the ranking of rival forecast sequences based on strictly consistent scoring functions. This cumulative doctoral thesis consists of four independent research papers, each contributing new methods for absolute or relative forecast evaluation, with the common aim of resolving robustness issues in current practice. These issues manifest themselves in two ways: first, sensitivity to tuning parameters that allow researchers to influence results by choosing favorable settings, and second, the absence of widely accepted procedures to quantify sampling uncertainty in commonly used evaluation tools. The papers in Chapters 2 and 3 propose robust calibration assessment methods for probability forecasts of binary events. From a theoretical perspective, the methods in both papers build on the natural monotonicity assumption between forecasts and event rates. In fact, this condition expresses the core intuition that higher probability forecasts, for example 70%, imply more frequent event realizations than lower ones, such as 30%, and constitutes a minimal requirement for calibration assessment since violations justify discarding the forecasts without further analysis. Importantly, Chapter 3 also responds to a longstanding request in the literature by introducing inverted calibration tests, i.e., tests in which the calibration property is formulated as part of the alternative. In simulation studies, as well as empirical applications to binary events, such as credit default and low-birth-weight forecasting, the proposed methods demonstrate their ability to resolve robustness issues of existing approaches. The papers in Chapters 4 and 5 concern the evaluation of point forecasts for real-valued outcomes. Specifically, these papers consider the decomposition of a scoring function (e.g., the omnipresent mean squared error) into non-negative numerical measures of miscalibration and discrimination and provide general results for quantifying the sampling uncertainty of these components. From a conceptual perspective, this decomposition provides deeper insights into forecast performance than the overall score and allows establishing formal connections between absolute and relative evaluation procedures. From a theoretical perspective, Chapter 4 lays the methodological foundation by establishing three separate asymptotic approximations for the decomposition components, while Chapter 5 combines these results to propose tests for the null hypotheses of equal miscalibration and equal discrimination between a pair of rival forecasts. These tests supplement classical procedures for testing equal predictive ability and provide complementary insights into forecast performance, as demonstrated in simulation studies and empirical applications to volatility and inflation forecasting. Collectively, the four independent projects contribute to a coherent and unified view of forecast evaluation by placing the use of calibration tests and score decompositions on statistically solid footing. The resulting methods extend well beyond the specific applications studied here and are broadly applicable across diverse forecasting domains in economics, finance, and related fields.