A Statistical Investigation of the CESM Ensemble Consistency Testing Framework

Constant evolution and improvement of large scale climate simulation codes such as the Community Earth System Model (CESM) necessitate quality checks to verify the constitution of new climate simulations. The CESM Ensemble Consistency Test (CESM-ECT) was developed as a flexible but objective method for checking the statistical consistency between an accepted ensemble of climate simulations and a new simulation created with updated code or within a new computational infrastructure. CESM-ECT utilizes a testing framework based on the popular technique of Principal Component Analysis (PCA) to determine whether a set of new simulations is statistically distinguishable from the established ensemble of climate simulations. One shortcoming of PCA is that accurate estimation requires the computational expense of a large ensemble. In this work we begin with an exploratory analysis of the CESM climate simulation data as well as an overview of PCA. Then we develop certain diagnostic tools to evaluate the estimation within PCA with respect to the size of the ensemble. Finally we demonstrate the limitations of PCA for smaller ensemble sizes by showing poor performance according to our diagnostics as well as an increased false positive rate for both empirical and simulated data.