Calibration training for improving probabilistic judgments using an interactive app

No Thumbnail Available
Issue Date
2023-12
Embargo End Date
Authors
Gruetzemacher, Ross
Lee, Kang Bok
Paradice, David
Advisor
Citation

Gruetzemacher, R., Lee, K.B., & Paradice, D. (2023). Calibration training for improving probabilistic judgments using an interactive app. Futures and Foresight Science. https://doi.org/10.1002/ffo2.177

Abstract

We describe an exploratory study examining the effectiveness of an interactive app and a novel training process for improving calibration and reducing overconfidence in probabilistic judgments. We evaluated the training used in the app by conducting an American college football forecasting tournament involving 153 business school students making 52 forecasts over 11 weeks. A coarsened exact matching analysis found statistical evidence that, in under 30?min, the more challenging training was able to modestly reduce overconfidence, improve calibration and improve the accuracy of probabilistic judgments (measured by the Brier score). The experimental results also suggest that the generic training can generalize across domains and that effective calibration training is possible without expert facilitators or pedagogical training materials. Although no previous studies have reported similar results, due to the modest effect, we conclude that these results should only be interpreted as a proof of concept and that further evaluation and validation of mechanisms of the app's effect is necessary.

Table of Content
Description
Click on the DOI link to access this article (may not be free).
publication.page.dc.relation.uri
DOI