My, how time flies… This June, the ARM Data Quality (DQ) Office celebrates its 10-year anniversary as the official gatekeeper of ARM data. Staffed by personnel from the University of Oklahoma, the DQ Office ensures quality data—the hallmark of the user facility—are collected by field instrumentation located at ARM’s sites around the world.
Created in 2000, as a result of a recommendation from the infrastructure review of 1999 to consolidate data quality efforts, the DQ office reviews more than 5000 datastreams—all the current ARM datastreams—each week. As part of this review, the DQ office identifies any data anomalies they see based on quality control procedures and tools established with the help of instrument mentors. Once identified, problems are recorded into the ARM problem reporting systems and tracked by the DQ office and ARM’s site scientists to resolution. These data quality assessment reports are shared with site operations and instrument mentors to assist in the corrective maintenance of the instruments.
“We couldn’t do this if we didn’t have an automated system. It’s the heart of our operations,” said Randy Peppler, ARM Data Quality Manager, and former member of the Southern Great Plains (SGP) site scientist team.
Randy was hired as ARM Data Quality Manager in January 2000 because the SGP site scientist team was working with other ARM site scientists to develop automated data quality software tools. Chad Bahrmann, SGP site scientist team at the time, had developed a program to review Soil Water and Temperature System (SWATS) data. This program became the prototype for Data Quality HandS Explorer, which is used today by DQ office analysts and instrument mentors to monitor ARM data.
DQ HandS provides the DQ office and instrument mentors the ability to inspect and assess data online in a near real-time basis using automated checks and diagnostic tools. In addition, Sean Moore from Mission Research developed a data plot browser and interactive data plotting tool to assist the DQ office in quickly visualizing datastreams and works on a half-time basis. A fully developed wiki web area, by Sean and Assistant Manager Ken Kehoe, is used to effortlessly communicate with among staff and the instrument mentors and houses data quality inspection guidance. Additional tools are under development for monitoring long term data sets to visualize long term trends, which will show calibration drift and degradation of the instrument. By using these tools, the DQ office has developed an automated quality control process highly regarded in the scientific community and recognized in The Open Atmospheric Science Journal, “An Overview of ARM Program Climate Research Facility Data Quality Assurance” in 2008.
Over the past 10 years, the DQ office has grown from a staff of two to an office with four full-time staff members and five part-time OU students. Original members of the DQ team include Randy, Karen Sonntag (2000-2008), and Andy Dean (2001-2004). Ken joined the DQ Office in 2004, followed by Justin Monroe in 2008, when ARM began increasing its real-time datastreams by adding mobile facilities to the DQ Office responsibilities. With funding from the American Recovery and Reinvestment Act of 2009 further increasing ARM capabilities by adding 143 new instruments, a fourth full-time research associate, Adam Theisen, was hired in May. In addition to new instruments, the DQ office is set to begin providing data quality assessments to the ARM value-added products, which Justin will be leading.
“Everyone who works here at the DQ Office is top notch!” said Randy. “And the reason ARM data are valued so much by the scientific community.”