Work with ARM Data

Resources and information are provided to help ARM users work with data.
 

ARM Data Formats

Most ARM data are made available as time-series data in netCDF (network Common Data Form), which is a set of libraries and self-describing, platform-independent data formats that support creating, accessing, and sharing array-oriented scientific data. It is relatively compact, appendable, and capable of storing descriptive “metadata” along with measurement data.

NetCDF includes data access libraries for a wide variety of programming languages. Example software programs for reading and writing netCDF files using C, C++, Fortran, IDL®, JavaTM, MATLAB®, PythonTM, and Perl, as well as more information about netCDF, including downloads, documentation, and frequently asked questions, can be found on the Unidata website.

Preview Plots of ARM Data

Quick-look plots provide users the ability to assess the content of ARM data files without specialized software or downloading. The plots are available from the ARM Data Center through Data Discovery after selecting a set of data to order.

After data have been ordered, ARM’s DQ-Zoom provides the means to perform a variety of data extraction, data conversion, and data visualization tasks.

Open-Source Programming

Scott Collis, Argonne National Lab, led the first-ever PyART short-course during the 2014 Joint User Principal Investigator Meeting. This course covers an intro to radar data and what it looks like in PyART, plotting RHIs, and basic analysis. It is an at-home exercise.

Scientists working with ARM data are encouraged to share experiences and codes and are invited to engage in community software development and make use of the ARM GitHUB account—especially for routine processing activities—to accelerate the collective application of these data.

Coding guidelines have been developed to help streamline the process of integrating PI-developed algorithms with ARM processes and libraries. The ARM development team has also created an open-source tool called the ARM Data Integrator (ADI) to integrate and transform multiple diverse datastreams and facilitate the creation of datastreams that meets ARM standards. ADI is available to use on OS-X and can be downloaded from the ADI folder in the ARM GitHUB account.

Other available open-source toolkits include the Python ARM Radar Toolkit (Py-ART) and the Atmospheric data Community Toolkit (ACT). Py-ART, which combines a variety of utilities for processing ARM radar data, is the first ARM open-source project. ACT, a toolkit for working with atmospheric time-series data sets of varying dimensions, has functions for every part of the scientific process: discovery, input/output, quality control, corrections, retrievals, visualization, and analysis. ACT is a community platform for sharing code with the goal of reducing duplication of effort and better connecting the science community and ARM infrastructure.

Computing Resources

Justin Monroe, from the ARM Data Quality Office at the University of Oklahoma, led a tutorial on ARM NetCDF files and how to navigate the ARM Python notebook during the 2014 Joint User Principal Investigator Meeting.

The ARM Data Center offers a public software development space that provides a mechanism to work with large volumes of data without having to download them. ARM’s Cumulus high-performance computing cluster provides programming ability in ADI, Python, MATLAB, and IDL, as well as specialized software such as Py-ART.

Some data files contain measurements distributed over a region from a single time (e.g., satellite images), which are stored in HDF (Hierarchical Data Format) files. Raw model data files may also be stored in GRIB (GRIdded Binary) and GRIB2 file formats. For more information, see Reading netCDF, HDF, and GRIB Files.

Data Questions

For questions regarding ARM data, please use the Data Questions form, or call 1-888-ARM-DATA. If your question is instrument-specific, please use the ARM contact form and select Instruments from the topics. If your question is related to Value Added Products (VAP), please contact the translator associated with that VAP.