Breakout Summary Report

This session aims to explore how Observing System Simulation Experiments (OSSEs) and related methods, such as Bayesian uncertainty quantification, can be operationalized to enhance the integration of ARM observations into model development. By cultivating a community of practice, we seek to maximize the impact of ARM data in quantifying and reducing uncertainties in key atmospheric processes and their regional and global impacts. Building on outcomes from the CLIVAR Micro2Macro workshop (https://usclivar.org/meetings/micro2macro) and insights from the ARM UEC modeling outreach subgroup survey (https://surveymonkey.com/r/armmodelingsurvey), we will address the critical question:

How can ARM and the ASR community most effectively leverage OSSEs and related methods, such as Bayesian uncertainty quantification, to maximize the impact of ARM data on understanding and reducing key uncertainties in atmospheric processes?

Additionally, we will discuss community needs for software and algorithmic frameworks that enable widespread use of advanced model-observation integration methodologies.

Potential applications span the full range of scales, including:

• Large-Eddy Simulation (LES) OSSEs for optimizing field campaign design to capture critical atmospheric phenomena effectively.

• Climate OSSEs for identifying observations and process improvements most likely to reduce uncertainties in future climate projections.

• Digital Twin OSSEs to support the design of complex laboratory experiments, such as the proposed DRACO cloud chamber.

• What do we need from OSSEs/UQ for hypothesis-driven science?

• How do we leverage OSSEs to interpret observations?

• How do we leverage OSSEs to make ARM deployments more impactful?

This breakout session focused on how Uncertainty Quantification (UQ) and Observing System Simulation Experiments (OSSEs) can support hypothesis-driven science, inform more efficient observational deployments, and improve the interpretation of field and laboratory data within the Atmospheric Radiation Measurement (ARM) and Atmospheric System Research (ASR) ecosystems.

As atmospheric science research matures from exploratory and frequentist statistics to hypothesis driven science, new methods are needed to test hypotheses in the presence of multiple confounding sources of variability and uncertainty. OSSEs provide a testbed that allows a Bayesian framework to advance ASR science goals. OSSEs give us a way to test hypotheses by telling us whether new observations are inconsistent with our underlying process model given the uncertainty in every other process. The argument was made that while AI models can be useful in developing these testbeds, it is still the purview of DOE to provide process understanding to enable medium/long-range predictability.

Presentations were given by three speakers addressing: using OSSEs to propose future field deployments [Jiang]; using OSSEs to evaluate observational information content in terms of processes and societally-relevant Earth system projections [Elsaesser]; and to mediate between observables in the laboratory environment and the actual processes being studied [Shaw]. A common theme of these talks was that we already possess many of the computational tools needed to do this, and the main impediment is connecting the global-scale model tools that exist to observations through simulators and dealing with spatial heterogeneity. Within the laboratory space, this scale gap was less of an issue and groups are already using more complex models to build up unobservable quantities from observables.

An important point brought up by an audience member was whether OSSEs, which are usually built on sampling underlying physical model uncertainty through something like a perturbed parameter/physics/process ensemble (PPE), are useful since they will almost surely experience some degree of structural error. It was discussed that any underlying physical model of a process is going to be, by nature, approximate and structurally incomplete and this problem becomes more difficult at larger scale gaps. However, this is more of a feature than a bug since in the case where your physical model is structurally wrong, it would allow identification of that error.

Another issue that was raised was how we account for observational systematic uncertainty. It was discussed that this is exceedingly difficult to tackle in observations, but one approach might be to offer observation ‘bundles’ where multiple retrievals/observations of the same variable are grouped together and their difference provides an estimate of systematic errors.

It was discussed how OSSEs might have been useful in planning the Bankhead National Forest (BNF) site. While the was heavily inspired by OSSEs as used for satellite missions (e.g. INvestigation of Convective UpdraftS (INCUS)) there should be some discussion in the future about identifying key differences and highlighting the differing mission as it relates to ASR/ARM science (e.g., implementing and applying OSSE frameworks given the significantly shorter ARM and ASR deployment award cycles, respectively).

Based on an audience suggestion, we provide a concrete example of how OSSEs allow us to address ARM/ASR science goals. We can imagine a closure of some observable based on a physical model of a process (Figure 1). This might be observations of cloud droplet number and an investigator trying to develop a process model of coalescence. The investigator develops their model using past observations (blue squares and the single grey line). The investigator proposes a new deployment to answer the question of whether their process model of coalescence is correct for previously unprobed regime. Figure 1 shows three hypothetical measurements that might be made during the deployment: #1: very far from the predictions of the process model; #2: nearer to the process model but outside of the observational uncertainty; and #3 in agreement within observational uncertainty with the process model. If the investigator doesn’t consider uncertainty from all the other processes that affect cloud droplet number, they will incorrectly reject their process model unless the observations in this new regime look like #3. One can envision using an OSSE combined with a PPE and propagating uncertainty from all other processes (i.e. nucleation, etc.) to generate a process uncertainty envelope (shaded region in Figure 1). This also highlights the necessity to provide systematic uncertainty for observations because the error bars would be effectively infinite and no process model could be rejected.

• The tools to build OSSEs already exist at the global scale (the existing PPE creation/sampling approach) and at the laboratory scale. These tools are advancing to bridge the gap with observations.

• New parameter estimation frameworks provide a digital testbed that enables us to flip from frequentist hypothesis testing (the probability of a set of observations with a theory) to Bayesian testing (the probability of a theory being true given a set of observations).

• Lessons learned from BNF point towards the utility of pre-campaign/deployment planning.

• Systematic uncertainty in observations remains a key issue in tackling OSSE/hypothesis workflow, but the possibility exists to simply bundle observations as a first-order solution.

There is an awkward middle ground between very small-domain/high-resolution environments like the PiChamber and global models with O-100km grids where OSSE tools already exist at both scales. In general, application of OSSEs to proposed field campaigns requires the ability to simulate the conditions likely to be observed and the observations to be collected. Additional work is needed to bridge across scales; specifically, bridging across ARM-relevant scales for deployment planning purposes.

N/A

N/A

• Created a mailing list at: https://forms.gle/H7QZsZJY9DZacWdH8.

• Breakout to gauge progress at next PI meeting.

• AMS special collection discussed.

• BAMS article by Fridlind et al. discussing OSSEs planned to be published in next few months.

N/A