New Micropulse Lidar Cloud Mask Machine Learning VAP Released

Published: 21 October 2021

This graphic shows four panels, labeled log10 NRB, LDR, MPLCMASK Cloud Mask and MPLCMASKML Cloud Mask. The y-axis on both plots is labeled Height (km).
This sample MPLCMASKML plot from the Southern Great Plains atmospheric observatory shows the log of the normalized relative backscatter (log10 NRB), linear depolarization ratio (LDR), and cloud masks from the MPLCMASKML and MPLCMASK value-added products for December 24, 2017. The strength of the machine learning approach is apparent on this day when moderate aerosol loading is likely interfering with accurate cloud boundary identification for the MPLCMASK algorithm in the boundary layer between 10-14 Coordinated Universal Time (UTC). In addition, MPLCMASKML can identify the attenuating cloud below 500 meters between 10-11 UTC.

Diverse cloud detection algorithms have been developed and applied to atmospheric lidar data to identify cloud boundaries and produce cloud masks. The general approach that most algorithms use reflects the complex physics of interactions between laser light and atmospheric particles to detect and distinguish clouds from other features (atmospheric and non-atmospheric) in the data.

This type of cloud detection algorithm was implemented to provide cloud boundaries from the Atmospheric Radiation Measurement (ARM) user facility’s micropulse lidars (MPLs) in the original Micropulse Lidar Cloud Mask value-added product (MPLCMASK VAP).

While these algorithms can identify a wide variety of cloud types and conditions, it is often observed that the algorithms can still fail to accurately detect clouds that are readily discernable when visually inspecting lidar images. Based on this observation, an alternative approach for cloud detection is to take advantage of machine learning capabilities and the trained eye as an interpreter of lidar images, and in turn, train a neural network to recognize the desired features in the lidar data.

This new approach has been applied to data from ARM MPLs to develop a machine learning model that can produce pixel-to-pixel predictions of clouds in lidar images (Cromwell and Flynn 2019). These predictions are available in ARM’s new Micropulse Lidar Cloud Mask Machine Learning (MPLCMASKML) VAP. This VAP also gives users the cloud mask generated from the prediction output, the number of cloud layers, and the cloud layer boundaries.

The new VAP uses the model to produce a cloud mask from the MPL attenuated backscatter and linear depolarization ratio. MPLCMASKML reduces cloud misclassification by almost half compared with the MPLCMASK cloud mask when evaluated against hand-labeled cloud masks created by a lidar expert.

MPLCMASKML can capture clouds below 500 meters and prevent cloud layer merging. In addition, the new VAP captures finer details of the cloud layers.

Evaluation data from the new MPLCMASKML VAP are now available for the following ARM sites and date ranges:

Future data will come from the 2019–2020 Multidisciplinary Drifting Observatory for the Study of Arctic Climate (MOSAiC) expedition.

Eventually, all available ARM fast-switching polarized MPL data will be processed with MPLCMASKML.

More information about MPLCMASKML is available on the VAP web page.

To share your experience using the evaluation data or to ask questions, contact ARM translator Damao Zhang, assistant translator Donna Flynn, or VAP developer Erol Cromwell.

Access the data in the ARM Data Center. (Go here to create an account to download the data.)

To cite the MPLCMASKML data, please use doi:10.5439/1637940.

Reference: Cromwell E and D Flynn. 2019. “Lidar Cloud Detection With Fully Convolutional Networks.” In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), 619-627, doi:10.1109/WACV.2019.00071.

# # #

ARM is a DOE Office of Science user facility operated by nine DOE national laboratories.