Unravelling an optical extreme learning machine

. Extreme learning machines (ELMs) are a versatile machine learning technique that can be seamlessly implemented with optical systems. In short, they can be described as a network of hidden neurons with random fixed weights and biases, that generate a complex behaviour in response to an input. Yet, despite the success of the physical implementations of ELMs, there is still a lack of fundamental understanding about their optical implementations. This work makes use of an optical complex media to implement an ELM and introduce an ab-initio theoretical framework to support the experimental implementation. We validate the proposed framework, in particular, by exploring the correlation between the rank of the outputs, (cid:2) , and its generalization capability, thus shedding new light into the inner workings of optical ELMs and opening paths towards future technological implementations of similar principles.


Introduction
Deep neural networks are becoming a ubiquitous tool across many scientific domains and areas of knowledge.Yet, the performance of neural networks is intrinsically tied to its scalability, and with the impending plateau of Moore's law, emerges a need to continue performance scaling in the absence of electronics miniaturization.A promising route lies within hardware specialization, especially for machine learning applications.In particular, significant performance gains can be achieved by leveraging optics as hardware accelerators due to its intrinsic parallelism, high bandwidth, the easy implementations of linear operations through propagation and low energy consumption.
A common approach to implement an optical version of this framework is to employ either non-linear light propagation, or ensure a non-linear detection.While effective, such works have not explored the effect of the random projection of their systems.In this work, we aim to shed some light on the mathematical intricacies of a common implementation of an optical ELM based on complex optical media.

Results and discussion
In Fig. 1 we present an illustration of the concept and experimental setup.In short, we use a digital micromirror device (DMD) as the optical encoder capable of both amplitude and phase modulation enabled by Lee holography.The light is then coupled to a multi-mode fiber where the information is mixed.The output field is a speckle pattern which is known to possess gaussian circular statistics that guarantees the randomness required by an ELM.Then, this pattern is measured on a high-speed CCD camera both in the linear and non-linear regime.We have developed a theoretical framework which demonstrates https://doi.org/10.1051/epjconf/202226613034EPJ Web of Conferences 266, 13034 (2022) EOSAM 2022 that simple linear dynamics followed by intensity measurements yield an effective activation function of polynomial and/or sinusoidal nature.Furthermore, this analysis allows us to infer on the amount of information that is carried out to (i.e. its rank) based on the optical encoding, as well as its generalization capability, which has been experimentally confirmed, as seen in tables 1 and 2. The unmatched values of rank(H) predictions' are due to experimental noise, which is out of the scope of this manuscript.Fig. 2. Classification performance under amplitude modulation of the wavefront.a1) and b1) rely on a simple direct linear encoding mechanism according to table 1, and linear detection on the camera.c1) and d1) employ the same encodings, but with non-linear detection (i.e.camera saturation).
Summary of results from figure 3. The encoding variable ܾ ଵ is a one-to-one mapping of the ‫ݔ‬ coordinate of each point to a linear phase modulation of the wavefront from 0 to 2ߨ in distinct regions of the DMD screen.
In addition, we also studied the effect of an intense physical non-linearity through saturation in the camera and its effect on the rank of the output.At the same time, the machine is benchmarked on standard regression and classification tasks, and we have found that it can yield remarkable performance under certain conditions, as seen in figures 2 and 3.In conclusion, we have found a connection between the rank of the outputs, , and its generalization capability which aligns with the theoretical framework developed.However, the relationship is not unique, thus establishing the rank of the outputs only as a good rule-of-thumb when predicting the performance of an ELM, yet not the ideal metric.As for the effect of a strong non-linearity, our results are consistent with the ones present in the literature which makes a stronger case for the need of physical nonlinearities for good performance.Optical ELMs are a promising platform for hardware specialization, and our work paves the way for a better understanding of such devices, allowing for a better design and task selection.

Fig. 1 .
Fig. 1.Illustration of the optical set-up and methods for an optical extreme learning machine.

Fig. 3 .
Fig. 3. Regression task performance under phase modulation of the wavefront.The target function is represented by the dashed line.a1) and b1) rely on a simple linear encoding mechanism, while c1) bears a nonlinear encoding (see table1).All with linear detection on the camera.d1) employs the same encoding as b1), but with non-linear detection (i.e.camera saturation).Semitransparent circles represent training data and opaque triangles test data.

Table 1 .
The two classes are represented as red triangles and blue circles.Summary of the results from figure2.The encoding variables (ܽ ଵ , ܽ ଶ ) is a one-to-one mapping of the ‫,ݔ(‬ ‫)ݕ‬ coordinates of each point to a linear amplitude modulation of the wavefront in distinct regions of the DMD screen.