edu.ucar.opensky::articles:27339 UCAR/NCAR - Library OpenSky Support PO Box 3000 Boulder CO 80307-3000 opensky@ucar.edu http://opensky.ucar.edu/ homepage pointOfContact 2025-12-24T18:11:34.691904 ISO 19115-2 Geographic Information - Metadata - Part 2: Extensions for Imagery and Gridded Data ISO 19115-2:2009(E) https://n2t.org/ark:/85065/d7dv1q3w https://doi.org/10.4230/OASICS.LDK.2019.12 https Related ConferencePaper #1 Validation Methodology for Expert-Annotated Datasets: Event Annotation Case Study https://doi.org/10.5281/zenodo.8370665 https Related Dataset #1 Quantitative Content Analysis Data for Hand Labeling Road Surface Conditions in New York State Department of Transportation Camera Images https://doi.org/10.5281/zenodo.7113671 https Related Other #1 Trust and trustworthiness codebook for content analysis: An example from NWS Forecaster interviews about AI Increasing the reproducibility and replicability of supervised AI/ML in the Earth systems science by leveraging social science methods 2024-07-01T00:00:00Z publication https://n2t.org/ark:/85065/d7dv1q3w https://doi.org/10.1029/2023EA003364 Wirz, Christopher D. author Sutter, C. author Demuth, Julie L. author Mayer, Kirsten author Chapman, William author Cains, Mariana author Radford, Jacob author Przybylo, V. author Evans, A. author Martin, T. author Gaudet, L. C. author Sulia, K. author Bostrom, A. author Gagne, David John author Bassill, N. author Schumacher, Andrea author Thorncroft, C. author UCAR/NCAR - Library publisher Artificial intelligence (AI) and machine learning (ML) pose a challenge for achieving science that is both reproducible and replicable. The challenge is compounded in supervised models that depend on manually labeled training data, as they introduce additional decision-making and processes that require thorough documentation and reporting. We address these limitations by providing an approach to hand labeling training data for supervised ML that integrates quantitative content analysis (QCA)-a method from social science research. The QCA approach provides a rigorous and well-documented hand labeling procedure to improve the replicability and reproducibility of supervised ML applications in Earth systems science (ESS), as well as the ability to evaluate them. Specifically, the approach requires (a) the articulation and documentation of the exact decision-making process used for assigning hand labels in a "codebook" and (b) an empirical evaluation of the reliability" of the hand labelers. In this paper, we outline the contributions of QCA to the field, along with an overview of the general approach. We then provide a case study to further demonstrate how this framework has and can be applied when developing supervised ML models for applications in ESS. With this approach, we provide an actionable path forward for addressing ethical considerations and goals outlined by recent AGU work on ML ethics in ESS. UCAR/NCAR - Library OpenSky Support PO Box 3000 Boulder CO 80307-3000 opensky@ucar.edu http://opensky.ucar.edu/ homepage pointOfContact PDF Text theme Resource Type 2016-01-01T00:00:00Z publication Copyright author(s). This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. None eng geoscientificInformation