Digital Holographic Microscopy applied to 3D Computer Microvision by using Deep Neural Networks

The application of advanced microscopy imaging techniques to 3D specimens or motions faces the problem of the limited depth of field of optical lenses [1]. Thanks to numerical focus computations, digital holography (DH) releases these limitations and extends significantly the allowed axial range of imaging. However, the digital cost of focus distance determination and of object reconstruction makes real-time 3D imaging hardly possible, especially when both in-plane and out-of-plane metrics must be extracted simultaneously.


Introduction
In the field of computer vision and robotics, accurate determination of position and trajectories in 3D environments is of high importance for a wide range of applications [1].Neural networks, such as convolutional neural networks (CNNs) and Vision Transformers (ViT) neural networks, have become powerful tools for visual data processing [2].In addition, the development of digital holography (DH) in microscopy has enabled the analysis of amplitude and phase of an object in a single image, which can improve the ability of a computer vision system to figure out an in-focus position with high accuracy without any mechanical displacements.The combination of deep neural networks (DNN) and DH offer a promising solution for advanced optical control of complex trajectory determination of micro-objects in automated microscopy, enabling real-time constrains [3].

Theoretical Background and context 1.1 Deep Neural Networks
DNN are a machine learning technique that is inspired from the structure and functioning of biological neural networks, to process, classify and predict complex data.These networks are organized in multi-layers, that process and analyse data by harnessing non-linear transformations from an input layer to the output one in order to realize, for example, linearization in a higher dimension space.The delivered DNN results are optimized with a learning step, which consists in training the network with knowing input and output pairs of data.The training step needs to have access to a huge amount of data to guarantee a prominent level of performances.Efficiency of deep neural networks has been proved in image classification, computer vision or predicting metrics values for solving complex problem such as autofocusing in DH [2,3].Last generation of DNN, such as convolutional CNNs and ViT models, have been shown to be highly effective in processing visual data and extracting complex features from 2D and 3D images.

Digital Holographic Microscopy, computer micro-vision and micro-robotics
DH is an advanced coherent imaging technique that can register from a real object the entire wavefield in amplitude and in phase by using a solid-state-sensor.The object information can be reconstructed numerically by directly calculating Angular Spectrum Method (ASM).Digital holographic microscopy (DHM) consists in applying digital holography principles to micro-size objects [2].Digital autofocusing can be implemented without any mechanical displacements, by choosing focusing metrics to develop automated microscopy apparatus and optical 3D pose control and measurement of micro-objects.
, 13011 (2023) This study aims to explore the capabilities provided by DH coupled with computer micro-vision approach based on phase correlation techniques to perform sub-voxel sample pose measurements in micro-robotics [4,5].
Crucially, the quality of an image and the amount of information it has depends not only on the employed instruments.Computational imaging approaches can further be enhanced or optimized with DNN algorithms.
Recently, many works have shown that processing times can be reduced for auto-autofocusing in DH, using approaches based on statistical image reconstruction by deep learning methods.Autofocusing is then approached as a classification or regression tasks where the physical parameters necessary to calculate the reconstruction of the clear image are no longer needed.
In this context, the challenges are to greatly improve multiscale sensitivity in automated microscopy in the pose estimation of samples versus the six degrees of freedom (DoF) by DHM, crucially while maintaining extended capabilities in terms of field of view and depth of field by using a 2D pseudo-periodic pattern as referencing sample (Fig. 1(c) and (d)).Indeed, the instrumentation at the level of high-tech microassembly platforms in robotics requires translation and/or rotation stages according to several directions of space (Fig. 1(a)).The tasks associated with these devices are increasingly complex, and nanoscale positioning resolution is needed while simultaneously keeping a large-scale movements range beyond the centimetre range.The determination of the various typically inaccessible metrics in the imaging devices used in microrobotics represents a large challenge such as instrumented microstructures, automated microscopy, micro-force sensors etc. Applications addressed in the study include digital autofocusing and micro-scale 6DoF positioning measurements of micro-objects augmented by very recent and efficient DNN.The common aim here is to target real time applications not feasibly without such advanced method.

Discussion and Results
We address this issue by applying deep neural networks to video-rate microvision measurement of 3D trajectories with DH.On the one hand, the ability of new generation of deep neural networks such as ViT to predict the focus distance with a high accuracy was demonstrated [2].On the other hand, the micro-structured pattern used as inplane position encoder has allowed a 10 8 range-toresolution ratio through robust phase-based decoding applied to conventional imaging [5].Here we present deep neural networks dedicated to hybrid approach combining computer microvision and DHM, able to perform simultaneous in-plane and out-plane measurements, at video-rate and without in focus object reconstruction.3D trajectories were reconstructed using the experimental setup presented in Fig. 1.It consists in a DHM, a hexapod capable of precise motions along the 6DoF and a micro-encoded pattern.We also show a typical hologram obtained and its reconstruction (Fig. 1).
The interferometric character of DH converts out-ofplane position of the sample in phase data that, combined with in-plane information retrieved from the microstructured pattern, allows accurate measurement of 3D trajectories.DNN speed up data processing and infer video-rate position detection.DNN require to be trained to realize expected tasks and to reach the best performances.In our work, the training step is conducted from dataset constituted by simulated or experimental holograms.Spherical aberrations, noise, have been implemented in simulated hologram datasets, with the aim of being able to mimic real experimental conditions.Such a prospect would significantly improve the current capabilities of combining computer microvision and DH for pose measurement and sensing applied to automated 3D microscopy.

Fig. 1 .
Fig. 1.(a) DHM observing a micro-structured pattern moved by the hexapod.(b) Experimental hologram of the pattern.Image reconstruction (c) in amplitude and (d) in phase.In focus distance at 130µm.