Work with ARM Data

Most ARM data are made available as time-series data in netCDF (network Common Data Form), which is a set of libraries and self-describing, platform-independent data formats that support creating, accessing, and sharing array-oriented scientific data. It is relatively compact, appendable, and capable of storing descriptive “metadata” along with measurement data.

NetCDF includes data access libraries for a wide variety of programming languages. Example software programs for reading and writing netCDF files using C, C++, Fortran, IDL^®, Java^TM, MATLAB^®, Python^TM, and Perl, as well as more information about netCDF, including downloads, documentation, and frequently asked questions, can be found on the Unidata website.

Quick-look plots provide users the ability to assess the content of ARM data files without specialized software or downloading. The plots are available from the ARM Data Center through Data Discovery after selecting a set of data to order.

After data have been ordered, ARM’s DQ-Zoom provides the means to perform a variety of data extraction, data conversion, and data visualization tasks.

Scientists working with ARM data are encouraged to share experiences and codes and are invited to engage in community software development and make use of the ARM GitHUB account—especially for routine processing activities—to accelerate the collective application of these data.

Coding guidelines have been developed to help streamline the process of integrating PI-developed algorithms with ARM processes and libraries. The ARM development team has also created an open-source tool called the ARM Data Integrator (ADI) to integrate and transform multiple diverse datastreams and facilitate the creation of datastreams that meets ARM standards. ADI is available to use on OS-X and can be downloaded from the ADI folder in the ARM GitHUB account.

Other available open-source toolkits include the Python ARM Radar Toolkit (Py-ART) and the Atmospheric data Community Toolkit (ACT). Py-ART, which combines a variety of utilities for processing ARM radar data, is the first ARM open-source project. ACT, a toolkit for working with atmospheric time-series data sets of varying dimensions, has functions for every part of the scientific process: discovery, input/output, quality control, corrections, retrievals, visualization, and analysis. ACT is a community platform for sharing code with the goal of reducing duplication of effort and better connecting the science community and ARM infrastructure.

The ARM Data Center offers a public software development space that provides a mechanism to work with large volumes of data without having to download them. ARM’s Cumulus high-performance computing cluster provides programming ability in ADI, Python, MATLAB, and IDL, as well as specialized software such as Py-ART.

Some data files contain measurements distributed over a region from a single time (e.g., satellite images), which are stored in HDF (Hierarchical Data Format) files. Raw model data files may also be stored in GRIB (GRIdded Binary) and GRIB2 file formats. For more information, see Reading netCDF, HDF, and GRIB Files.

For questions regarding ARM data, please use the Ask Us, and select Report Data Quality Issues or Finding ARM Data, whichever option is more appropriate. If your question is instrument-specific, please select Instruments and VAPs from the topic list.