POISSON TOTAL VARIATION DENOISING FOR MICROPULSE WATER VAPOR DIAL

We have adapted the Poisson Total Variation lidar signal processing technique for Micro-Pulse DIAL water vapor estimates. This processing technique ingests data at 10 second-37.5 meter resolution where it adaptively adjusts the retrieval resolution based on signal-to-noise and performs range deconvolution. The result is high resolution data in regions with ample signal, preservation of sharp discontinuities where they exist, and a data product that better represents of the acquired photon counting data than the standard processing technique. technique


INTRODUCTION
Water vapor DIAL uses two closely spaced wavelength, one tuned to a water absorption feature (online) and a second tuned off the feature (offline) to estimate range resolved water vapor concentration in the atmosphere [1]. Through knowledge of the water vapor absorption cross section, the two observations can be directly related to the water vapor number concentration. The power of this approach is that it requires no ancillary observations of atmospheric state for processing or calibration, making it a truly independent of other water vapor observations. In a collaboration between the National Center for Atmospheric Research (NCAR) and Montana State University, a Micro-Pulse DIAL (MPD) has been developed based on diode laser technology for remote unattended operations over long periods of time [2,3,4]. The instrument lasers are invisible and eye-safe and the receiver employs narrowband filtering and photon counting to observe weak signals day and night in clear and cloudy conditions. In order to obtain useful water vapor estimates, the standard MPD signal processing employs fixed temporal smoothing between 1 and 5 minutes and range smoothing between 150 and 750 m (depending on signal-to-noise).
Poisson Total Variation is a processing technique recently developed at University of Wisconsin for high spectral resolution lidar (HSRL) retrievals and has been shown to significantly improve extinction and lidar ratio estimates from the UW HSRL instrument [5]. The technique leverages numerical methods developed for Poisson image denoising and deblurring [6,7]. PTV assumes that the estimated lidar data products can be approximated by a linear piecewise function and effectively adapts the region of estimation based on signal-tonoise. It leverages correlations in the observed image (in the case of lidar a time/range curtain) to infer the data product from the random noise without forcing the underlying signal to be band limited.
In a collaboration between NCAR and University of Wisconsin, we have demonstrated the PTV technique can be adapted for MPD water vapor estimates. While the approach is still under development, it shows promise for improving the instrument's capabilities by providing flexibility to capture variations at higher resolution and increasing retrieval accuracy.

METHODOLOGY
The PTV method for HSRL is detailed in [5], and the technique was adapted for processing DIAL signals without sacrificing the advantages of the conventional DIAL technique. Specifically, it allows estimation of water vapor without direct knowledge of the atmospheric aerosol loading, instrument effects common to both wavelengths (e.g. geometric overlap) or scalar differences between the two wavelength channels.
PTV requires a forward model to evaluate retrieved parameters against raw photon counts.
To implement this we add a second parameter retrieval. The forward model consists of a profile that is dependent on the wavelength ( ), the water vapor density ( ) and a term that represents all terms that are common multipliers between the two wavelength channels ( ). The observed photon counts at bin i and operational wavelength is given by Where ( ) is a constant multiplier specific to the laser wavelength of operation, ( ) is the water vapor absorption cross section at that wavelength, Δ is the range between data points and is the background.
In principle, the common term can be estimated using a conventional DIAL processing approach, but it is not necessary and provides limited scientific value. For this analysis, we require estimating to forward model the retrieval and evaluate its accuracy.
In our retrieval we allow the estimator to adjust the scalar gain of each channel ( ). Like , this falls out in standard DIAL processing but is needed for forward modeling. It has no impact on the actual water vapor estimate.
The forward modeled profiles are evaluated using the inverse log-likelihood for a Poission distribution.
Lower values of this function indicate a better fit. We also include a regularizer term to penalize variation in the retrieved profile. The overall minimization is performed on where the forward model ( ) is a function of the water vapor density ( ) and multiplier term ( ). The term ‖ ‖ evaluates the sum of the absolute value of all changes in in both time and range and and are regularization coefficients that determine the amount of penalty imposed for variation in the estimated water vapor and common terms. Figure 1 shows an example of MPD retrievals applied to data from August 16, 2018. The top panel shows our standard water vapor processing routine. The middle panel shows the water vapor estimate from the PTV processing method described here and the bottom panel shows the common terms obtained through the PTV method. We note that there are some key differences between the standard and PTV processing techniques. First the PTV method captures sharper discontinuities in the water vapor profile because PTV approximates the water vapor field as a linear piecewise function. This is in contrast to the our standard Savitzky-Golay filtering and Gaussian smoothing approach used in the top panel, which effectively limits the sharpness of the profile.

RESULTS
In the region above 3 km, the PTV better captures the dryness of the air by using larger pieces, while the standard processing approach has substantial noise that deceptively looks like enhanced water (a product of the color scale having a minimum value of zero).
The PTV estimate shows an artificially high water vapor content at the top of the profile above 5.5 km. We suspect this artificially high region is the result of low signal, where the optimizer did not adjust the water vapor estimate because it had little impact on the forward model error.
The PTV method also appears to be having problems estimating water vapor in the lowest range bins below 500 m where there is significant striping throughout the time series. This may be the result of the outgoing laser shot contaminating the return signal. We note that the standard method also appears to have problems in this region, though they are less severe. To analytically assess the two retrieval methods we use the inverse log-likelihood ( , ),without the regularizer term where we compare the forward modeled retrievals against a verification observation that is statistically independent from the original retrieval data. These separate retrieval and verification datasets are obtained from a single photon counting MPD profile using Poisson thinning, where the underlying signals are common to both datasets, but their Poisson noise is statistically independent. We forward model the two water vapor estimation approaches using the common terms from the PTV method in both cases. We found using the standard approach to estimate made its forward model evaluate worse and we do not want to penalize the approach for a secondary, typically unused, parameter. The difference in the inverse log-likelihoods are shown in Figure 2, where positive values indicate better agreement from PTV water vapor estimates and negative indicates better agreement from the standard approach.
From Figure 2, it is clear that the PTV method provides a better overall reflection of the lidar profiles. The preservation of higher frequency structure seems to improve the forward signal reconstruction below 2 km and above 4 km. Notably the artificially high water vapor does not seem to have a significant impact on the verification process, again suggesting that there is not enough information content in this region (i.e. there is high signal uncertainty) to estimate water vapor concentration.

CONCLUSIONS
The PTV signal processing method appears to be promising, for developing high accuracy water vapor estimates from photon counting MPD. However there are practical challenges to the technique. Signal processing is computationally expensive. The data set shown here was processed on a desktop computer and took approximately one day to complete. At present, there is no fast way to estimate uncertainty in the retrievals. Current techniques employed for HSRL retrievals employ ensemble approaches with various initial conditions making the computationally expensive though parallelizable.
Never-the-less these issues are not necessarily major limitations. Advances in and availability of high speed computing resources continue to improve. There is currently work to implement PTV on GPUs and we continue to pursue opportunities to develop faster uncertainty estimates.