Cumulus High Performance Cluster

The Cumulus cluster is the Atmospheric Radiation Measurement (ARM) User Facility’s colocated high performance computing (HPC) environment, pairing petascale storage with a flexible, heterogeneous architecture. Cumulus is a midrange Dell system that consists of 16,384 processing cores with a 4-petabyte General Parallel File System (GPFS).

Cumulus enables data-proximity computing so researchers can run next-generation atmospheric simulations, large-scale data processing, and machine learning workflows directly on the cluster without large data downloads. It is also used by ARM infrastructure staff for Large-Eddy Simulation (LES) ARM Symbiotic Simulation and Observation (LASSO) development and operation, radar data processing, large-scale reprocessing, value-added product generation, and data quality analysis.

Cumulus supports a wide array of programming environments—Python, MATLAB, IDL, and OpenMPI—along with domain-specific toolkits such as the Python ARM Radar Toolkit (Py-ART), Atmospheric data Community Toolkit (ACT), and ARM Data Integrator (ADI). A full list of supported environments is available in the Cumulus quickstart guide.

The Cumulus cluster is available to users with an active ARM account to conduct ARM-approved computationally intensive science projects that involve working with large volumes of ARM data, applying computationally intensive codes to ARM data sets, or analyzing LASSO output data sets. Use of the Cumulus cluster for ARM user science projects requires submission of a short proposal to ARM.

ARM users may request use of the Cumulus cluster by submitting a high performance computing request (HPCR), a short proposal describing the activities that will make use of the Cumulus cluster. All investigators who will need access to the computing resources for a given project must be included as a principal investigator (PI) or co-PI on the request. All investigators must have active ARM user accounts before submitting their HPCR proposal. Proposals to use the cluster cannot be submitted until accounts have been created for all PIs and co-PIs.

Proposals to use the ARM HPC resources should focus on science activities that meet one or more of the following criteria:

• involve large volumes of ARM data, which would be prohibitive to download to other computer systems
• require parallel processing using computationally intensive code applied to ARM data sets
• analyze voluminous LASSO outputs.

Unless explicitly requested through a special call for proposals, requests will not be considered that primarily involve running dynamical model simulations, without meeting one or more of the above criteria.

Proposals for the use of Cumulus will be reviewed monthly by the ARM Infrastructure Management Board (IMB). Larger efforts may require additional information and undergo a longer scientific peer review. Computational requests must:

• clearly indicate the relevance of the proposed computational activities to the ARM mission
• describe the ARM data sets to be analyzed
• explain how the proposal meets the criteria above
• explain why ARM computational cluster resources are the most appropriate computational resources for achieving the science goals.

To submit an HPC proposal, PIs must submit the HPCR form. The request will be included in the next monthly IMB review that occurs after responses from the PI to all requests for clarifying information have been received. Results of the review of the proposal and responses to PIs will be communicated after the monthly review.

Once approved, the ARM Data Center HPC team will communicate with PIs to enable access to Cumulus and provide any technical support needed.

Projects focusing primarily on model simulations or data analytic applications that do not require the use of large volumes of ARM data or do not apply computationally intensive code to analysis of ARM data may be more suitable for the U.S. Department of Energy’s (DOE) National Energy Research Scientific Computing Center (NERSC) or other DOE leadership computing user facilities, such as the Oak Ridge National Laboratory Leadership Computing Facility (OLCF) and Argonne National Laboratory Leadership Computing Facility (ALCF). Allocations for these computational facilities should be requested directly from the individual facilities.

1. Code of Conduct. PIs must review and agree to ARM’s Code of Conduct when requesting access to ARM’s HPC resources.
2. Project Description. PIs must provide a description of their proposed computational project and its science goals. The project description must clearly show use of a large volume of ARM observational data or LASSO output, or the application of computationally intensive codes to ARM data sets. The project description should also include the computational approach, the planned ARM and non-ARM input data volumes, and all output data volumes required.
3. Status Reports. If the project extends beyond one year, PIs must provide annual status reports to ARM.
4. Data Submission. HPC projects that create new data products should plan to submit their data to the ARM Data Center as a PI data product. PIs should provide a detailed description of the product and data volume storage needs.
5. Final Report. A final report for the project is required to be submitted to ARM within six months of project completion.
6. Acknowledgment of ARM Support. Investigators who receive ARM support for their work should use the following acknowledgment in associated publications:

“This research was supported by the Atmospheric Radiation Measurement (ARM) User Facility, a U.S. Department of Energy (DOE) Office of Science user facility managed by the Biological and Environmental Research Program.”

When closing out a project, the PI is required to submit a brief, final report of the outcome to the ARM Field Campaign Administrator to complete the ARM documentation.