ARM Priorities

The Air-ARM project is procuring a replacement aircraft for the AAF G-1. Air-ARM does not include onboard scientific instrumentation. This ENG is designed to capture the replacement and development activities for the said onboard scientific instrumentation.
Over 50 AAF owned and operated instruments have served the community during the G-1 deployments. AAF aims to maintain/replicate the current AAF instrumentation capability while (where possible) also achieving improvements that serve the broader science community.

As a part of a recent EMSL-ARM pilot study, the ARM TBS and AAF teams had deployed three TBS impactor (TBI) packages during 2019. The TBS team completed 17 aerosol-related flights at the Southern Great Plains (SGP) atmospheric observatory in July and September, and nine flights at the Oliktok Point site, Alaska in August 2019. Three impactors collected hundreds of substrates for micro-spectroscopy analysis (such as scanning TEM and XRDSEM). The above deployments enabled the ARM TBS and AAF team to understand the operation envelope for meaningful micro-spectroscopy analysis at the SGP site. However, further study is still required to improve the sample collection in the Arctic area. Additionally, we also propose to keep exploring the TBS deployment and TBI operation requirements for the other potential EMSL analysis, such as Inverted Confocal Raman Spectrometer, nanoDESI, and Time-of-Flight SIMS for both the SGP site and the OLI site. The inverted Confocal Raman Spectrometer can provide valuable information for a more comprehensive understanding of the investigated sample, such as ice nuclei particles. The nanoDESI can further analyze the chemical compounds, especially organic compounds in the atmospheric aerosols. The TOF SIMS can probe the surface properties of the atmospheric aerosols. To understand how to collect meaningful samples for the above analysis will pave the road for the future ARM-EMSL joint proposal call.

SDS upgrades have been approved for FY20 for SGP, ENA, NSA, and VSN. This ENG will be used to track the procurement, configuration, and installation of the hardware.

This is a continuation of data discovery upgrade ENG…., the scope of this ENG is to capture Data Discovery phase II development activities recommended by stakeholders. Some high-level changes include:

• Enable "Epoch" type search functionality within guided search and highlight epochs on the data details page
• Enable "Location" type functionality within guided search and Spatial search (ability to draw a polygon/rectangle and do data subsetting on spatial domain)
• Integrate Data Stream search
• Enable the functionalities in "Your Orders" section of the ''My Account Dashboard" (track order, reorder, new data availability)
• Include an option for users to filter data based on data quality flags
• Improve the display of field campaign results
• Add the ability to have a shareable link to details pages
• Add the ability to show timelines in search results
• Add a preference on "My Account Dashboard" for choosing regular or pretty email formats
• Update plot viewer to be a much more integrated tool within DD
• Improve citations to include more citation formats

Part a (by January 30): SN upgrades to address delayed tickets: the objective of this task is to capture the reason for any ticket that cannot be automatically acted on and closed. We expect this to be straightforward, adding a few key features and automatic reporting within SN.

Part b (by April 30): routine instrument metadata workflow automation (a follow-on/phase 2 to the INST process) will include integration of existing capabilities (e.g., PCM processes list), and will automate a check of database (DB) tables, and automated/programmed cloning where appropriate. Metadata Service (MDS)->Service Now(SN) connections will allow MC coordinators tp review the automated metadata. This saves manual clones within MDS, and minimizes potential human error (e.g., cloning wrong data level). Other automation will include integrated checking of Data Discovery (DD) (link generation) within SN.

Part c (by September 30): guest instrument metadata and data archival workflow modernization will include development of an Online Metadata Editor (OME)<->ServiceNow(SN)<->Metadata Service (MDS). In addition to providing helpful automation for data submitters and metadata team, this capability will also allow us to provide key features (e.g., a "coming soon" button on field campaign pages) and allow for better tracking of PI-provided dataset submissions (e.g., a SN dashboard) for transparency to management.

Benefit to sponsor/program: decreased overhead with a modern metadata and data flow (increased efficiency) and better tracking (e.g., dashboard) of datasets expected and received; increased transparency in what datasets will be included in a campaign for the user community can push up user metrics (e.g., data use, publications citing data).

Motivation The vast ARM data archive contains an extensive resource of atmospheric measurements spanning over 25 years. User feedback suggests that it can be difficult to find high-quality data of scientific interest. ARM has begun a number of activities to help improve accessibility of ARM data, such as upgrades to the Data Discovery Tool and updating the list of recommended datastreams. To complement these activities, ARM has developed the concept of data epochs (defined as time periods with high quality data of scientific interest) and tagged descriptors (e.g., dimension of quality, other atmospheric variables in use by other teams, hierarchical terms) to enhance existing metadata. For a given datastream, ARM metadata currently provides many descriptors such as location, time, data source (i.e. instrument, VAP, external source, campaign, PI product), type of instrument, types of measurements, and amount of processing (data level). The introduction of the new tagging to ARM data holdings adds another layer of metadata that will enrich the user experience.

This engineering change is to refine the scope and guidelines for defining data epochs, identify appropriate metadata tags in accordance with Dublin Core key principles, develop necessary database tables to hold this information, and metadata tools to assign and edit these tags. Finally, search capabilities on the web site and Data Discovery tool will be enabled for users to filter and order data of interest. These activities will be in coordination with activities under ENG 4341 (2021), which focuses on an initial set of tagged descriptors aimed at recommended data and collected during the rationale statement process.

During the 2021 Developers Meeting, participants of the Data Tagging breakout session provided ample feedback and suggestions for tagged descriptors and data epochs. We will use this information to develop the design document and guidelines.

ARM currently owns several Version 2 AERI instruments that are reaching their end of life and are in need of replacement. In addition, ARM needs to replace the ASSIST instrument that was recently retired from one of the SGP Extended Facilities that are part of the 4 boundary layer profiling sites. This first procurement will be to replace one instrument at SGP and potentially a second instrument to replace the ASSIST in FY19. Additional procurements are planned for FY21.

The DOE ASR science community has relied on accurate and routine measurements of atmospheric emitted thermal radiation at moderate spectral resolution (520-3300 cm-1) from the AERI for over 15 years. Data obtained from these systems are used in combination with other observations to derive vertical profiles of temperature and water vapor and to develop and test retrievals of cloud microphysical properties. More recently, the long-term AERI datasets at two ARM observatories were used to understand the surface radiative forcing of carbon dioxide and methane, two important greenhouse gases. Maintaining a consistent long-term dataset is critical to these types of high-impact research projects.

The Eppley radiometers (NIP, 8-48, PSP) are in need of replacement. This ENG is to track the requirements, testing, procurements and deployment of the new radiometers.

Introduction To obtain retrievals of fractional sky cover over its atmospheric observatories, the ARM user facility uses Total Sky Imagers (TSI), which provide real-time processing of hemispheric visible images of daytime sky conditions. For a continuous representation of cloud life-cycles, an Infrared Sky Imager is also operating at the Southern Great Plains facility that captures infrared images of the sky, during both the day and night.

Background TSIs were initially installed at the ARM sites from 2000 to 2003. The instruments were re-engineered to increase their performance in the harsh environments of the remote sites (ENG0000378) and later modified to increase imager resolution (ENG0000674), provide site-specific imager calibration (ENG0000644), and improve autonomous control (ENG0003147). However, the TSIs are aging and many of the parts have become unserviceable and/or are no-longer in production.

Objective The purpose of this request is to consider options for the replacement of the ARM TSIs with newer, commercially-available, technology. I will present the current status of the TSIs, results of the ARM Sky Imager Measurement Needs survey conducted in May, and identified instrument replacement options.

Reference Morris V.R. 06/11/2019. "Fractional Sky Cover Measurement for the ARM User Facility." Presented by Victor Morris at ARM/ASR PI Meeting, Rockville, MD. (https://asr.science.energy.gov/meetings/stm/posters/abstract/2197)

The high temporal resolution convective cell tracking measurements effectuated with the second generation C-SAPR2 during the CACTI field campaign in Argentina, revealed the necessity for the development of an automated adaptive scanning system for the radar. For the upcoming TRACER campaign in Houston, Texas, we propose to develop and implement on the C-SAPR2 a convective cell tracking algorithm for the high resolution polarimetric scans. We would prefer to implement two scenarios: a) The C-SAPR2 will perform low elevation surveillance scans for cell identification, based on the reflectivity threshold limits. Once the cell is identified, the radar will follow the selected cell until it satisfies the tracking parameters. b) To increase the temporal resolution, when it is possible, the radar will use information from an external radar or network, and will do only the cell tracking. The C-SAPR2 will accept the tracking parameters (azimuth and range of the center of the cell to be followed), and an identifier for the scanning task.

AMF2 AOS02 is currently the only AOS without an ACSM assigned to it. An ACSM (preferably ToF) was requested by the SAIL science team, but the availability of the AMF3 ToF-ACSM is limited due to conflict with the SEUS deployment. We propose procurement of a new ToF-ACSM for AMF2 AOS02. We note that the 10x better sensitivity for non-refractory aerosol species and higher time resolution of the ToF system compared to the quadruple ACSM offers benefits to the SAIL campaign, but also to future deployments AMF2 AOS02. Our immediate plans for the new ToF-ACSM system also include participation in European ACTRIS ACSM inter-comparison.

VAPs are currently under development for the existing AMF3 ToF-ACSM, and the data structure and acquisition for the new system will be the same. Based on what we now know after the MOSAiC deployment that was both long-term and remote, the ToF-ACSM is somewhat easier to use compared to the quadrupole, but is still a sophisticated instrument. Additionally, deployments at both Oliktok and MOSAiC have demonstrated the increased measurement sensitivity.

Planetary boundary layer (PBL) depth is important to a wide range of atmospheric processes including cloud formation, aerosol mixing and transport, and chemical mixing and transport. The CAPI working group has proposed development of an ARM PBL height VAP to make this important value available to the ARM and broader scientific community.

Numerous instruments and algorithms may be used for PBL height detection, each with their own strengths and weaknesses. We plan to implement several different algorithms for PBL height detection using different instruments as the comparisons between the different methods will provide useful information about the reliability of the PBL height retrievals. We will develop this VAP in a phased approach, in which with we start with relatively simple algorithms and successively add new instruments/algorithms. Additionally, we will initially focus on the convective daytime boundary layer, as it is more easily detected and algorithms are more reliable than for the stable nighttime boundary layer.

In the first phase of this VAP (covered by this ECR), we will focus on implementing methods for radiosondes, ceilometer, and MPL as these instruments exist at all ARM sites. Two radiosonde methods (Heffter and bulk Richardson) and three ceilometer/MPL algorithms (1D gradient, wavelet, combined 2D gradient and wavelet) have been identified. More details on these methods are given in the implementation plan. Code for these algorithms will be provided by ARM scientists, and will be implemented in ISDE. In the next stage of development, which will be defined in more detail at a later date, we will explore implementing retrievals for more advanced instrumentation (AERI, Raman Lidar, Doppler Lidar, Radar wind profilers) that do not exist at all sites.

Initial evaluation of the VAP will focus on several IOP periods: MC3E, Azores, CARES, and AMIE. As part of the VAP development, we will also develop a database of PI PBL height retrievals for these IOP periods. We do not plan to run a formal inter-comparison, but will use this database to 1) evaluate the implementation of the VAP methods, 2) identify promising methods to implement in the next state of development, 3) identify areas of uncertainty and methods of flagging PBL height retrievals, and 4) encourage science PIs to submit their retrievals as formal ARM PI products.

A draft PBL height implementation plan has been developed and is available from Sally upon request.

Summary: Develop New Three-Channel Cloud Optical Depth VAP for CSPHOT

Proposed VAP name: csphotcod2chiu* Science Sponsor: Christine Chiu Translator: Laura Riihimaki Developers: Wei Wu and Laurie Gregory

This ECR is to propose a new ARM operation VAP to implement Christine Chiu’s three-channel cloud optical depth retrieval algorithm.

Motivation

A new three-channel (440, 870, and 1640 nm) retrieval algorithm for cloud optical depth has recently been developed by Christine Chiu and her co-workers [Chiu et al., 2012]. The algorithm is developed for simultaneously retrieving key cloud properties (optical depth and effective radius), by using ground-based “cloud mode” zenith radiance measurements from the Cimel Sunphotometer. These radiance measurements are available routinely from the Aerosol Robotic Network (AERONET) worldwide (including all recent ARM CSPHOT deployments). Based on the results from extensive tests, this new algorithm is capable of enhancing the current two-channel cloud mode product currently in production at AERONET. The new algorithm will provide higher retrieval quality, while retaining the unique characteristics of inexpensiveness and worldwide coverage.

The plan proposes to implement Christine Chiu’s three-channel cloud optical depth retrieval algorithm as an ARM operational VAP. First, simultaneously retrieved cloud optical depth and effective radius datasets will be generated by using the zenith radiance measurements taken from the ARM program and the NASA AERONET. Satellite-based (e.g., MODIS) surface albedo data are needed as input for the proposed retrieval. Finally, the algorithm produces uncertainties along with the retrieval products.

Implementing the three-channel retrieval algorithm as an operational VAP for generating key cloud products (e.g., optical depth and effective radius) has been identified as a high priority for cloud modeling activities (e.g., scheme development). The retrieved cloud properties are critical to studying cloud microphysical and radiative properties, key to advancing our understanding on precipitation initiation, parameterizations of cloud sub-grid variability in climate models, and evaluation of satellite-based observations.

Data Product Development Tasks

Christine’s algorithm is already well developed and has been run for the MAGIC field campaign. In order to develop the code further for operations, the following is needed:

1) MODIS satellite products will need to be collected to retrieve albedo. The XDC will need to set up collections to routinely fetch and process MODIS data. The data product currently proposed to be used is the MODIS “MCD43A1” combined Terra and Aqua data product, which is at 500m data measured over 16-day period. Proposed ARM name is “modisalb” (MODIS Albedo). Files will be subsets for the area needed at each site.

2) The XDC will need to collect and ingest zenith radiances from AERONET which will contain calibrated radiances for all channels collected by ARM Cimels. Proposed ARM name: csphotzenrad (csphot zenith radiance).

3) The code will need to be updated to run daily with options to run for all ARM sites and time periods for which cloud mode has been implemented.

4) The COD output need to be converted to netCDF and updated to follow ARM data standards.

*”csphotcod2chiu” is the second version of COD algorithm as indicated by “2chiu”. The 2-channel algorithm currently available at AERONET is also planned to be integrated as csphotcod1chiu. Plans for this will be available in a separate ECR.

References

Chiu, J. C., Marshak, A., Huang, C.-H., V√°rnai, T., Hogan, R. J., Giles, D. M., Holben, B. N.,ence Sponsor: Christine Chiu Translator: Laura Riihimaki COD VAP data product development & evalua

The ARMBE products (armbecldrad and armbeatm) has been generated for the SGP, NSA and TWP sites. The purpose of this effort is to create ARMBE for the ENA and AMF sites.

This effort will include identifying appropriate ARM datastreams at ENA and other AMF deployments and modifying the code accordingly. The modified code will be run with ADI to streamline the process and meet ARM standards. The ARMBE products for ENA and AMF deployments will be generated and released to the ARM Archive.

In phase 1 of this work (ENG1204), we built the framework to compare climate model output to Climatology data collected at the ARM SGP site and CMIP model point output. Phase 2 of this work will be a continuation/expansion of the development of phase 1.

At the moment users can only subset ordering using data stream name, start date and end date. Given the huge volume of radar data users need to be able to subset the data based on radar volume contents. This will create a summary data point for each radar data volume CMAC2.0 touches and save it to a Cassandra database within the ADC for future use by the ADC in data discovery and ordering.

This is an update to ENG0001219, which has become out of date as plans have evolved. This revised processing plan was presented at the November 2019 AMSG meeting by John Shilling. Quadrupole ACSM processing tasks are:

1) Finish the production of an autonomous b1-level datastream that includes content from the raw instrument files. The b1 data will be processed assuming a collection efficiency of unity (CE=1). a) Mass concentrations of organics, sulfate, nitrate, ammonium, and chloride. b) House-keeping values reported in the DAQ. c) Other relevant system information including calibration details. d) New QC tests based on the raw mass spectra and housekeeping values. e) Calculation of the volume from the ACMS mass to facilitate comparison to integrated size distribution measurements.

2) Comparison of ACSM volume generated from b1 files with volume measurements from co-located instruments. SMPS is most appropriate for this comparison, but UHSAS will also be used.

3) Merging of the b1 datastream with the mentor-processed data to generate a b2 datastream. Mentor-processed b2 datastream also assumes CE=1, but includes an additional correction and has been examined by the mentor.

4) Generation of an autonomous c1 datastream/VAP. The c1 VAP will calculate the composition dependent collection efficiency (CDCE) described by Middlebrook et al. 2012 and apply this to the data. See: (https://doi.org/10.1080/02786826.2011.620041)

5) Evaluation of the CDCE correction. Comparison of ACSM volume c1 files with volume measurements from co-located instruments. SMPS is most appropriate for this comparison, but UHSAS will also be used.

6) Merging of the c1 data with mentor-processed data to generate a c2 datastream. The mentor-processed c2 datastream will include a mentor-generated CE and corrections to the data based on this mentor-generated CE.

7) Comparison of the c1 and c2 datastreams.

A schematic of the data processing and datastream names is attached. Because of the differences in the datastructure between the ToF and quadrupole ACSM, we will generate an analogous but separate set of tasks for the TOF ACSM.

The goal of this ENG is to revive the CCN Profile VAP that has been developed, update it, and get it running in production. A technical report on this VAP is already available and much of the development is already done. The technical report is available at : https://www.arm.gov/capabilities/vaps/ccnprof At the time the VAP was developed, the CCN and RL data needed for the VAP were not sufficiently high quality to generate a reliable and high-quality VAP. The data quality has dramatically improved to the point where this VAP can be re-visited.

This is an update to ENG0003103, which was requested by Connor Flynn. Work is currently underway under ENG000313. Updating this ENG will allow the new aerosol translator (John Shilling) to better track the work. The tasks in this ENG are similar to those in ENG0003103, with some updates and minor changes.

The goal of this ENG is to finish the harmonization of the AOS size distribution information. Initial ingest of the instrument data to the A level is finished, though we may continue to find minor errors that need to be updated. The overall goal is to generate b-level datastreams for the SMPS, nano-SMPS, UHSAS, and APS. In some cases (APS, UHSAS), this requires the a-level data to be converted to engineering units from raw counts. HTDMA data will be handled separately.

This workscope will use CCNAVG, CCNSPECTRA, and SMPS datastreams to generate a CCNKappa VAP product. Kappa is a parameterized representation of an aerosol particle's hygroscopicity and is widely used in models to calculate CCN concentrations and cloud properties. The critical dry diameter at various supersaturation (SS) setpoints in the CCN instrument will be calculated by assuming particles greater than critical diameter are CCN-active and reconciling CCN number and particle size distributions integrated to that diameter. Kappa-Köhler theory (Petters and Kreidenweis, 2007) will be used to generate the time-series of kappa values as a function of SS.

This ENG is an overarching container for tasks related to the development of the data bundle contents for the deep-convection LASSO scenario that will focus on the CACTI field campaign. The scope includes case selection, development and implementation of the modeling strategy, selection and processing of appropriate observations, identification of skill scores and diagnostics to use with this scenario, and development of the data bundle format and contents.

While related, the scope of this ENG does not include development of the user interfaces and methods for distribution of the resulting deep-convection data bundles. For example, modifications to the Bundle Browser to accommodate the new scenario will be handled under a different ENG.

Development of this new scenario is guided by the following FY2020 milestones: 1. Determine preliminary selection of cases pending evaluation of observations, computational cost, and simulation results. 2. Propose initial LES configuration, skill scores, and diagnostics at PI Meeting. 3. Perform set of forcing tests for different reanalysis/forecast products using mesoscale domains. 4. Determine LES configuration and output strategy. 5. Provide an example data bundle for a selected case by year-end.

The specifics of the modeling approach will be guided by the foundation established at the LASSO Expansion Workshop held in May 2019. The science drivers used to prioritize development decisions focus primarily around convective initiation and the upscale growth during the early portions of the convective lifecycle. Resolving convection dynamics has been deemed important with the desire to resolve thermals within the convective cores. Frequent sampling of the thermals is important for identifying the evolution of their statistics and motion.

Observations used for the deep-convection scenario will differ substantially from those used for the shallow-convection scenario. Regional measurements will take a leading role over point-based measurements, and many of the model-observation comparisons will be performed statistically rather than via a one-to-one comparison. This is due to the large size of the CACTI convection as well as the difficulty of modeling specific convective clouds. Examples of the regional measurements are reflectivity from the scanning radars and brightness temperature from GOES-16.

Initial mesoscale modeling at kilometer-scale grid spacings will be done to select cases and forcing(s) to use for each case. Cases will be deemed as potentially viable based on the ability of these mesoscale simulations to capture the salient convection features, such as the general timing, location, and size of the storm(s) on the given day. The best-performing mesoscale simulations will be used to drive the large-eddy simulations with 100-m grid spacing for the final production runs.

The aim is to have an initial list of observations, a basic model configuration, and concept for the data bundles ready to present at the 2020 Joint ARM/ASR PI Meeting. This will serve as a check on the development process to inform the path forward from that point. We want to make sure we are incorporating all the relevant community needs to gain the most value from the expensive LES runs that will be performed. The initial concepts will be refined and tested during the remainder of the year. The end date for this ENG is the end of FY2020. At that point, progress will be reviewed and the ENG extended and/or replaced by a new ENG for FY2021 with a potentially modified scope based on the state of the development.

The ARM TBS and AAF teams have collected almost 350 hours of flight data at SGP and AMF3 from 2018-2020. Data collected during these TBS deployments are available on Data Discovery for the following datastreams: tbsground (surface temperature, pressure, wind speed, gust wind speed, wind direction, and relative humidity), tbspops (aerosol number concentration and size distribution from 140 nm – 3 µm), tbscpc (aerosol number concentration from 10 nm – 1 µm), tbsimet and tbsimetxq (airborne temperature, pressure, relative humidity, and altitude), tbswind (airborne wind speed and gust wind speed), tbsslwc (airborne supercooled liquid water content), and tbsdts (airborne high-resolution temperature). The data currently exist on Data Discovery as unique datasets, and a user would require familiarity with TBS operations to request and interpret data from multiple instruments flown simultaneously. The ARM user community would benefit from the creation of integrated datasets and visualizations of TBS flight data, to aid in the identification of flight data collected during atmospheric conditions of research interest, determining when multiple instruments of interest were operating, promoting a broader community understanding of data available from TBS flights, and improving the ease of use of TBS datasets. To achieve these goals we will merge individual TBS datastreams into an integrated dataset and create visualizations of the integrated data. Data from the surface-based ceilometer will also be added to allow the user to determine cloud base height and if the flight occurred under clear or cloudy conditions. A process will be developed to allow this integrated dataset and visualization to be generated during future ARM TBS data ingests.

The goal of this ENG is to produce a merged size distribution VAP that will combine data from the SMPS and APS into a single, continuous size distribution. We have nearly finished the harmonization of the size distribution instruments and data are now available in a consistent format and with consistent naming conventions. There has been a longstanding request from users of the aerosol data to produce a single size distribution from multiple instruments. The challenge is that some instruments use different measurement principles and therefore don't measure on an identical diameter basis. As a result, the process is not a simple file merge. Specifically, the APS measures an aerodynamic diameter, the SMPS measures a mobility diameter, and the UHSAS measure an optical equivalent diameter based on calibration with PSLs.

An initial review of the data was conducted in collaboration with the instrument mentors at BNL. We agreed that a logical start would be to merge the SMPS and APS data at the SGP site. Therefore, the initial goal is to convert the APS aerodynamic diameter to a mobility diameter and re-bin the APS data to the SMPS bin widths. We will largely follow the Beddows et al method described in the attached manuscript.