Reading netCDF, HDF, and GRIB Files

Most ARM data are stored in netCDF format. This format allows for the definition of data fields and storage of operational information in the header of the file. All ARM netCDF files are in UTC time and represent time as “seconds since January 1, 1970,” which is called the “epoch time.” For example, an epoch time of 1 means “Thu Jan 1 00:00:01 1970”; an epoch time of 992794875 is “Sun Jun 17 16:21:15 2001.” To learn more about how to convert epoch time, see Time in ARM netCDF Data Files.

More information on the netCDF format and tools is available from UCAR at their netCDF site.

Some data files also contain one or more measurements distributed over a region from a single time (e.g., satellite images). These data are stored in the HDF format developed by the National Center for Supercomputing Applications. See The HDF Group for more information about HDF and to access necessary tools.

Technical Notes

The metadata can be listed using the ncdump -h command of the HDF v3.3r3 release. Readers for HDF files are available in several commercial packages, including IDL (see also Liam Gumley‘s HDF IDL routines) and NOeSYS (which includes a free downloadable HDF browser). The free NCSA software is linked to above.

Reading GOES data in ERDAS

Ideally, one would simply want to fill in the length of the header in bytes in the ERDAS import window. Then ERDAS would skip those bytes and start plotting at the beginning of the image. Unfortunately, the length isn’t mentioned anywhere in the header. So, the other option is to extract each channel from the HDF file and feed them to ERDAS.

Once these utilities are downloaded and installed, you can use the following command to pull out each channel from the HDF file. This example is for channel 1: hdp dumpsds -n gvar_ch1 -d -o ch1.bin -b goes.hdf Execute this command for each channel and then import the 5 channels (ch1.bin .. ch5.bin) into ERDAS. When filling in the ERDAS import window select BSQ (band sequential), unsigned-32 bit, and fill in the correct lines (rows) and samples (columns) for your image.

Also remember to change the default 1 channel to 5 channels (unless you want to import only one) and click on the multiple channels on the right side of the screen. Another window will come up expecting the pathname for each channel.

Some raw model data files are stored in GRIB (GRIdded Binary) file format. For example, ECMWF and RUC raw files are stored in this format. Beginning 2/1/2008, RUC raw files are stored using GRIB2.
Available utilities for reading and converting grib files include:

• GRADS—The Center for Ocean-Land-Atmosphere Studies (COLA) developed a Grid Analysis Display System (GrADS) that reads and displays ECMWF GRIB data. The XDC is using GrADS to convert ECMWF GRIB to netCDF.
• NCL (NCAR Command Language)—software including NCAR Graphics for scientific data processing and visualization. NCL supports NetCDF, GRIB1, GRIB2, and HDF formats. Included are the ncl_filedump command to view contents of files and the ncl_convert2nc command which converts grib files to netCDF.
• wgrib2—For extracting data from grib2 files and for simple conversions to netCDF.
• wgrib—Extracts data from grib1 files.
• cnvgrib—Code for converting between grib1 and grib2.