ARM Priorities

After getting approval of the IMB, AAF is embarking on routine flights over the SGP site on FY23. It is expected that we will deploy 3 times during FY23 namely in March, June and August 2023.

This ENG will continue to capture and finalize the proposed ARM Data Workbench (ADW) concept requirements. Based on the initial needs gathered from a set of users and stakeholders, ARM data service staff developed a data workbench concept and presented the idea during the 2019 and 2020 ARM PI meetings (https://asr.science.energy.gov/meetings/stm/posters/abstract/2509). The ARM Data Workbench (ADW) is a revolutionary way to interact with the vast amount of data ARM offers. Utilizing parallel processing frameworks, data stores, and NoSQL technologies, ADC developed an initial version of backend workbench infrastructure and successfully used it in the LASSO bundle browser and a few other use cases. As part of this proof-of-concept, we now have access to over 50 select datastreams readily available on this platform used in test cases.

Once developed, the ADW architecture will provide a set of tools for users to select data, retrieve measurement values, visualize, perform data analysis and even create their own data bundle. Some of the initial requirements gathered include the ability to filter data, apply user-developed and on-the-fly equations for conditional querying that will allow users to make a unique data product for their needs. Other ARM initiatives such as data epoch, data tagging are expected to leverage this architecture.

Relevant background details, posters, and abstracts are attached in this ENG. This concept is also included in the latest ARM decadal vision, and this ENG will be the first step addressing this activity.

It is important to note that the architecture will have to be built with a goal of leveraging relevant tools and software that ARM staff are currently building. For example, the workbench is expected to utilize the parallel processing frameworks, NOSQL data stores, visualization and analysis software published on the ACT. This architecture will also integrate the current ARM Computing Environment (ACE). The workbench will be made available on both the ADC resources and ARM HPC clusters.

The primary scope of this ENG is to identify the potential users and stakeholders, and collect the requirements.

This ENG will have the following tasks: - Identify remaining user groups and stakeholders and gather requirements - Review and finalize the requirements - Evaluate current capabilities and develop preliminary wireframes/prototypes for demonstrating the concept - Determine the effort needed to leverage ACT, ACE, and other related tools as part of the overall ADW architecture

The implementation and design reviews will be tracked in a separate ENG with the actual implementation is expected to begin in early FY22.

Purpose of this ENG is to design, and deploy a new Cumulus-2 cluster within the ARM Computing Environment. Cumulus-1 has served the needs of ARM's LASSO modeling, and other research and data processing efforts over the last few years. As Cumulus-1 comes close to end of life, a replacement cluster needs to be designed and deployed to ensure uninterrupted availability of computational capacity for LASSO model simulations and operational data processing. This overarching ENG would capture all stages of requirements gathering, design and deployment thru child ENG/EWO/TSKs.

Target user groups: - - LASSO development and operations team. -- VAP developers. -- Operational data processing team. -- ARM/ASR Research community.

The current calibration database is housed in OSS which is being replaced by the asset management tool. OSS will remain until we have a new calibration system. Requirements have been gathered through 3 feedback sessions with ARM mentors, site operations, and infrastructure in December along with information solicited from science users through email and discussions with a UEC subgroup. The detailed requirements are housed in the google doc with the higher level requirements mapping an priorities in the google sheet for broader review and approval

Doc: https://docs.google.com/document/d/1a6-O8O0tGsI6nJz5YKMhij2GJGrcyMWK80XC7zfx5fk/edit?usp=sharing Sheet: https://docs.google.com/spreadsheets/d/1AylvpmP_wvOzJvDre0CFflX4dShRH-Ty72fPbq8lqOY/edit?usp=sharing

In talking with the science users, it was clear that there was a large array of needs when it comes to the dissemination of calibration information. There are three main schools of though which I think we need to account for, 1.) those that want to see the calibration records and have them delivered with their data, 2.) those that want to be able to view the records in a web-base viewer, and 3.) those that just want to know that their data are calibrated through a mechanism in the data files. 1 and 2 will be addressed in phase 1 and 3 will require a larger effort that includes modifications to the ingest processes in phase 2.

Introduction To obtain retrievals of fractional sky cover over its atmospheric observatories, the ARM user facility uses Total Sky Imagers (TSI), which provide real-time processing of hemispheric visible images of daytime sky conditions. For a continuous representation of cloud life-cycles, an Infrared Sky Imager is also operating at the Southern Great Plains facility that captures infrared images of the sky, during both the day and night.

Background TSIs were initially installed at the ARM sites from 2000 to 2003. The instruments were re-engineered to increase their performance in the harsh environments of the remote sites (ENG0000378) and later modified to increase imager resolution (ENG0000674), provide site-specific imager calibration (ENG0000644), and improve autonomous control (ENG0003147). However, the TSIs are aging and many of the parts have become unserviceable and/or are no-longer in production.

Objective The purpose of this request is to consider options for the replacement of the ARM TSIs with newer, commercially-available, technology. I will present the current status of the TSIs, results of the ARM Sky Imager Measurement Needs survey conducted in May, and identified instrument replacement options.

Reference Morris V.R. 06/11/2019. "Fractional Sky Cover Measurement for the ARM User Facility." Presented by Victor Morris at ARM/ASR PI Meeting, Rockville, MD. (https://asr.science.energy.gov/meetings/stm/posters/abstract/2197)

To purchase 2 additional DL units to replace older units at SGP EFs and deploy new systems to the supplementary sites at Southeast US AMF3. ARM has procured 2 additional DLs in FY21 for the same purpose. Instruments will be initially shipped to SGP.

Procurement of TSI SMPS 3938 requested for deployment in ENA for the ambient particle size measurement from 10 to 500 nm. Categorized as a baseline measurement for size distribution, the instrument will also serve as a reference measurement for ACSM volume closure comparison and a potential calibration unit for size and count dependent measurements in ENA.

In 2018, we undertook the upgrade of the NSA High-spectral Resolution Lidar (HSRL; ENG0003895). That upgrade was completed in 2020 at which time the lidar was deployed first to SGP and then to SAIL. The current plan is to deploy this lidar with the Raman at the SGP when not committed elsewhere to enable a three-wavelenth (355 nm, 532 nm, 1064 nm) capability at SGP. A second three-wavelength HSRL/Raman pair will be set up at the southeast US site. However, this leaves the NSA without an extinction-capable lidar. The intent of this ENG is to nail down the requirements for a multi-channel, extinction-capable lidar to be deployed at Barrow and then to proceed with the procurement of that instrument.

The goal of this ENG is to revive the CCN Profile VAP that has been developed, update it, and get it running in production. A technical report on this VAP is already available and much of the development is already done. The technical report is available at : https://www.arm.gov/capabilities/vaps/ccnprof At the time the VAP was developed, the CCN and RL data needed for the VAP were not sufficiently high quality to generate a reliable and high-quality VAP. The data quality has dramatically improved to the point where this VAP can be re-visited.

Cloud thermodynamic phase identification is important to understand many cloud processes such as ice particle production, precipitation formation, and cloud lifecycle evolution. The CAPI working group has proposed development of an ARM cloud phase VAP to make this important value available to the ARM and broader scientific community.

Cloud thermodynamic phase classification can be determined from remote sensing measurements. Specifically, active remote sensing instruments such as lidar and radar provide vertically resolved cloud thermodynamic phase identification. Due to different wavelengths, lidar backscattered signal is more sensitive to small particles with high number concentrations such as liquid droplets. Lidar depolarization measurements can be used to identify the nonspherical particles such as ice crystals. However, lidar signal is quickly attenuated by cloud droplets. Radar signal is dominated by large particles such as ice crystals and is susceptible to attenuation only in case of strong precipitation. In addition, microwave radiometer measurements can provide constraints on the presence of liquid layers. Since no one instrument can provide sufficient level of information to cover from small droplets to large ice particles and snow, we use the multisensor method developed by Shupe (2007) to classify cloud phase. Similar to the ARM Ka band Zenith Radar (KAZR) Active Remote Sensing of Clouds (KAZR-ARSCL) VAP, the multisensor approach uses coincident ARM Micropulse lidar (MPL) with depolarization, Ka-band ARM Zenith Radar (KAZR), and microwave radiometer measurements, together with atmospheric thermodynamic profiles from radiosonde measurements to determine cloud thermodynamic phases.

For the initial implementation, the VAP will be implemented at the ARM North Slope of Alaska (NSA) atmospheric observatory central facility at Utqiaġvik. An HSRL system was deployed and has been acquiring measurements at the NSA Utqiaġvik site since April 2011. The Ka-band ARM zenith radar (KAZR) replaced the millimeter-wavelength cloud radar system starting in May 2011, providing measurements with high temporal and vertical resolution and reduced spectral artifacts. These measurements provide high quality inputs for the VAP. Over the NSA Utqiaġvik site, persistent stratiform mixed-phase clouds are frequently observed during the transition seasons such as in spring and fall. Liquid clouds are often observed during summer while ice clouds in winter. These seasonal variations of major cloud thermodynamic phase provide a good opportunity to apply the cloud phase identification methods.

A draft cloud phase identification implementation plan has been developed and uploaded.

Background: Accurate simulation of aerosols, clouds, radiation, and precipitation is an ongoing challenge for current state-of-art climate models. Developing and implementing metrics for aerosol-cloud interactions (ACIs) in the current ARM-DIAGS is particularly important for boundary layer clouds and mixed-phase clouds. The University of Arizona (UA) group led by Prof. Xiquan Dong has been working on classifying and investigating aerosol properties and interactions with clouds for decades, using remote sensing and in-situ measurements, as well as model simulations. With their expertise in the field of ACI and experience in retrieving and processing ARM data, in this collaboration work, we propose to implement standardized analysis tools and associated ARM-site data sets for ACI metrics into ARM-DIAGS, as a new set of process-oriented diagnostics suite.

Tasks: The development and implementation of the ACI metrics into current infrastructure of ARM-DIAGS involves following tasks: • Write Python scripts to pre-process variables from raw ARM instrument data required to generate the necessary statistics. • Create an observationally based database for ARM-DIAGS by following the data format defined by CMIP conventions • Write Python modules to compute performance metrics for CMIP models

The UA group will work closely with the LLNL ARM team to interface the data and Python modules with existing ARM-DIAGS framework and document the data products and the diagnostics package at the end of the funding period.

Timelines: October 1, 2021: Project starts. March 31, 2022: Complete Phase 1: Implement basic metrics on ACI, which includes the climatology of cloud and aerosol properties, and the magnitude of an index describing the ACI process. Preprocess variables such as the aerosol optical depth (AOD), surface cloud condensation nuclei (CCN), cloud-top height, cloud-base height, cloud fraction etc., which are required for the metrics from various data sources collected at multiple ARM sites. September 30, 2022: Complete Phase 2: Implement process-oriented diagnostics on ACI at selected ARM sites, such as SGP, ENA and NSA.

ARM Ground-based cloud radars have difficulty detecting the small cloud particles present at the top of many high-altitude cirrus clouds. To address this issue, we propose to implement the ground-based lidar simulator in COSP for ARM lidars to simulate the ARM lidar measurements. · Address issues associated with lidar attenuation by low clouds and aerosols · Produce ARM lidar data for evaluating lidar simulator output · Collaborate with ARM/ASR radar/lidar experts and climate modelers to address any issues associated with the development and implementation of the lidar simulator to GCMs.

Take files, both individual PPI scans and RHI scans, value add, combine and add metadata to create radar volumes suitable for users. Ensure metadata meets ARM standards. In addition monitor output from the radar and provide weekly reports using the ARM Campaign Dashboard for SAIL. Report any issues seen in a timely fashion to both CSU and ARM staff managing the contract. If needed report out to the SAIL science team.

This ENG encompasses the LASSO work related to CACTI during fiscal year 2022. Each milestone will be included as sub-component of this ENG.

This ENG encompasses tasks related to LASSO-ENA work in fiscal year 2022. Each milestone will be included as sub-component of this ENG.

(note, text taken from SOW which will be attached) As part of the SAIL field campaign ARM deployed a X-Band radar system owned and operated by Colorado State University. This system produces radar moments and dual polarimetric observations from hydrometeors and other scatterers. The system is located approximately 8km to the southeast of the main site. Due to the complexity of microphysical habits of snowfall (dendrites, rods, plates and vast varieties of hybrids and aggregates of the aforementioned) relating radar measurements to snowfall rates (liquid equivalent rates, the rates in mm/hr if the snowflakes were all melted) is challenging. Matrosov et al 2009 nicely shows the range of uncertainties in reflectivity-based snowfall retrieval of the form Z=ASB. There are many other papers on snowfall retrieval (rarer at X-Band than, say, S-Band) our aim here is to start simple first and then add more sophisticated retrievals, including using other measurements, as we establish verification procedures.

The ARSCL family of VAPs is one of the ARM Program's most downloaded products. This VAP chain and family of VAPs includes KAZRCOR, ARSCL, KAZRARSCL, and WACRARSCL VAPs. It is important that this product chain continue to improve in order to provide the best cloud location and characterization possible. Over the last few years, some new and useful sources of cloud data have been added to ARM's Archive. Also in recent years, ARM infrastructure has been increasingly adopting Python as the programming language of choice, as well as the open source paradigm, where appropriate. As a result of the new data sources and updated software engineering practices, the time has come for KAZRCOR, KAZR-ARSCL and WACR-ARSCL upgrades. Accordingly, the following upgrades are suggested to take full advantage of new ARM Program capabilities and advances in software engineering practices:

KAZRCOR: -- Convert to Python -- Accept either KAZR, (M)WACR, possibly SACR VTP scans as input -- Improve computational efficiency -- Open source appropriate portions

ARSCL: -- Convert to Python -- Accept corrected and/or calibrated KAZR, (M)WACR, possibly SACR VTP scans as input -- Add additional data sources: MPLCMASKML, RWP observations, COGS where available -- Explore use of spectra-based decluttering techniques -- Eliminate manual clutter height review -- Open source appropriate portions

The above is only a rough outline of the proposed project. Please see attached overview document for additional details.

This workscope will use CCNAVG, CCNSPECTRA, and UHSAS datastreams to generate a CCNKappa VAP product for ENA site. This VAP will closely follow the methods used in developing the CCN/SMPS kappa VAP with modification to account for the different scan range of the UHSAS. Kappa is a parameterized representation of an aerosol particle's hygroscopicity and is widely used in models to calculate CCN concentrations and cloud properties. The critical dry diameter at various supersaturation (SS) setpoints in the CCN instrument will be calculated by assuming particles greater than critical diameter are CCN-active and reconciling CCN number and particle size distributions integrated to that diameter. Kappa-Köhler theory (Petters and Kreidenweis, 2007) will be used to generate the time series of kappa values as a function of SS.