KM3NeT Readout and Triggering

The KM3NeT collaboration is constructing new-generation neutrino detectors in the Mediterranean Sea. The main goals are the study of the high-energy neutrino flux (KM3NeT/ARCA, off-shore Capo-Passero, Italy) and the determination of the neutrino mass ordering (KM3NeT/ORCA, off-shore Toulon, France). This contribution describes the system to collect, transfer and process the data, which consists of time and intensity measurements of photons recorded by photo-multiplier tubes resulting from charged products from neutrino interactions, together with data from the acoustic position system. The trigger and data-acquisition system faces the challenge of transporting up to 30 Gbps of data to shore and filtering the neutrino signal from an overwhelming (O(107)) background due to photons from ambient 40K and bioluminescence.


Introduction
The KM3NeT Collaboration is currently constructing the first phase of a neutrino telescope infrastructure which consists of two detectors. With the KM3NeT/ARCA detector the collaboration aims at identifying the sources of high-energy neutrinos with a cubic-kilometer scale detector in the Mediterranean Sea at 3.5 km depth, 100 km off-shore of Capo-Passero, Sicily. With the KM3NeT/ORCA detector, which instruments 4 megaton of sea-water at 2.5 km depth, 40 km off-shore of Toulon, France, the collaboration aims at determining the ordering of the neutrino mass-eigenstates. By recording the arrival time and amount of Cherenkov photons produced by charged products of neutrino interactions, the incoming direction, energy and type of the incoming neutrinos can be determined. The KM3NeT/ARCA detector is optimized for high-energy neutrinos, TeV to PeV in energy, with an astrophysical origin. The KM3NeT/ORCA detector is denser and aims to measure neutrinos from cosmic-ray interactions in the atmosphere, after propagation through the Earth. The oscillation patterns in energy and zenith angle of atmospheric neutrinos in the 1-10 GeV range, strongly depend on the neutrino mass-ordering [1]. The photons are detected using 3-inch photo-multiplier tubes of which 31 are housed in a 17-inch pressure resistant glass sphere, together with control and read-out electronics. This assembly is called a Digital Optical Module (DOM). The DOMs also contain a piezo-electric hydrophone used for acoustic positioning and other instruments such as a compass-and tilt-meter. Communication to and from the DOMs is done over optical Ethernet, and therefore each DOM contains a laser-transceiver with a pre-defined frequency on the ITU grid with a 50 GHz spacing. While the scale of both detectors is different, the technology is the same. Both detectors consist of vertical lines (Detection Units, DUs), each with 18 DOMs. In the case of ARCA, the total height of a DU is about 700 meters, and the horizontal spacing between DUs is about 100 meters. In the case of ORCA, the total height is almost 200 meters, and the horizontal spacing is about 20 meters. The final configuration of ORCA will consist of one block of 115 DUs, while ARCA will consist of 2 blocks of 115 DUs. See the contribution [2] for more details on the DOMs and DUs. Each site has a seafloor network of electro-optical cables to supply the components with power and transport the optical signals. The communication infrastructure is asymmetrical. Communication to all elements is done via a physical broadcast on a single wavelength, while to shore each element has a selected wavelength. This optical network also provides nanosecond accurate timing between all elements via the White-Rabbit protocol. For more details on the networking aspect see the contribution [3]. For details of the control of the detectors, on-and off-shore, see the contribution [4]. This contribution will focus on the off-and on-shore elements for the acquisition and filtering of the data recorded by the PMTs and hydrophones.

Signal and Background: Data-acquisition philosophy
The data-acquisition system is designed around the all-data-to-shore principle. That means that all photons recorded by the PMTs are digitized and sent to the processing facility onshore without any selection. The counting rate of each PMT is not only high, about 7KHz, but is utterly dominated by background photons from 40 K decays and bioluminescence. The background to signal -defined as photons from neutrino interactions-is of order O(10 7 ). To deal with this challenge, the optical properties of the Mediterreanean Sea are exploited, together with the innovative multi-PMT design of the DOM. Due to the long scattering lengths, the information contained in the spatial and temporal distribution of photons is retained over long distances. A nano-second accurate timing precision on the photon arrival times is required to leverage this. The arrival time and coarse pulse-width information is sufficient and it is not essential to record the wave-form of the PMT signals. This allows for the photon hit information to be encoded in 6 bytes, reducing the optical data-stream up to a maximum of 20 Megabits per second per DOM (and thus about 40 Gigabit per second per block). Data are collected in timeslices of 100 ms. Each DOM sends these timeslices via Ethernet to shore, resulting in a time-multiplexing system. The on-shore infrastructure collects all timeslices belonging to the same period and sends them to a node in a computer farm, in a round-robin way. This can be done as the DOMs are synchronized and include the timeslice time in a header. Different detector-wide timeslices are thus processed in parallel on different CPUs. This system also allows to run multiple algorithms in parallel and when computing demands increase the processing power can be scaled up by adding CPUs. Nanosecond scale timing accuracy is essential and achieved via the White-Rabbit protocol [5] which can synchronise nodes in an optical Ethernet system to a master switch to subnanosecond accuracy. This does not require a dedicated clock line, but piggy-backs on the Ethernet signal. The digital acoustic piezo hydrophone samples at 193.5 kHz at 12 bit. This information is used to extract the acoustic pulses from fixed beacons. Together with information from the compass and tilt-meter this allows for a positioning accuracy of about 10 cm.

Off-shore systems in the Digital Optical Module
The signal from the PMTs is encoded at their bases in a low-voltage-differential-signal which is logic high as long as the PMT pulse amplitude exceeds a settable threshold. The arrival time (t 0 ) is thus encoded in the rising edge of the pulse, and the amount of photons is coarsely encoded in the width of the pulse, referred to as time-over-threshold (ToT). This signal is routed via signal collection boards to the main electronics board, the Central Logic Board (CLB) where a Xilinx Kintex Field-programmable-gate-array (FPGA) is the endpoint of the signals. The FPGA has 32 time-to-digital converters implemented via provided deserialisers to timestamp the 31 PMT signals, their ToTs and the acoustic data to nanosecond accuracy. First-in-first-out buffers contain the streams of t 0 s and ToTs. The buffers are monitored and flags are raised to stop filling and avoid overflow in case of excessive photon rates. For standard data-taking conditions a high-rate veto of 20 kHz was chosen. The timestamps result from a White-Rabbit PTP core, which implements the White-Rabbit capable Ethernet in the FPGA. The photon and acoustic data in a timeslice are split into several UDP packets and subsequently sent to the taget machine on-shore via separate streams. A dedicated LM32 processor implemented in the FPGA allows for two way communication via a simple retransmission protocol over UDP and controls the settings and state of the data-acquisition system and instruments.

On-shore trigger systems
All DOMs send their timeslices to the DataQueue processes which assemble the timeslices from multiple UDP jumbo frames of maximum 9 kB and send the timeslices for a specific time-period to a single Optical DataFilter over TCP. In case of the acoustic data, the DataQueue assembles the timeslices, but sends all data from a specific DOM to the same Acoustic DataFilter process. This is required as the acoustic processing needs to integrate over periods longer than a timeslice and at this stage the processing needs only to be done per DOM. If the timeslice causes a trigger, potentially containing a neutrino event (but more often atmospheric muons) the photon hits that activated the trigger are collected in a DAQ Event, together with all photon hits in a period of about 10 microseconds before and after these hits. These data are sent to a process called the DataWriter which writes all triggered events to file. Control of the on-and off-shore software processes is done by a unified detector acquisition control system (see reference above). Configuration parameters, such as PMT High-Voltages and settings of the trigger processes are communicated via this system. The calibration and detector configuration are retrieved from a central database. A state machine is implemented in all on-and off-shore processes. The (optical) network infrastructure and the computing cluster are located in shore stations close to where the subsea main electro-optical cables arrive. For ARCA this is in the harbour of Portopalo di Capo Passero, Italy and for the ORCA site this is in La-Seyne-sur-Mer, France. Detector control (shifts) is possible from these locations or from remote locations such as the home institutes of the collaboration members. Data are sent to the central storage at the computing centre in Lyon and are accesible for off-line analyses.

Optical Trigger Algorithms
The trigger algorithms are designed to filter the signals from possible neutrino interactions from the background dominated stream and store them for further analysis. The underlining principle, shared by the trigger algorithms is that photon arrival times (hits) from a neutrino interaction are causally related in a known way. The aim is then to find a set of pair-wise causally related hits with a minimum size M, a so-called clique problem. The most general causal relation for hits i and j, measured at location x i and x j at times t i and t j is given by Where c is the speed of light and n the refractive index. This relation can be made more restrictive by assuming a direction of travel of e.g. a muon, then the relation becomes 2 tan θ c where θ c is the Cherenkov angle. Using the latter condition requires a scan over N directions, where an event is selected when in at least one direction there is a cluster of size M or greater. An additional feature is that the distance between two hits in the plane perpendicular to the assumed direction can be used to reduce the number of possible hits due to the geometrical dilution and absorption of light in water. It is obvious that given the large data-rate of single hits, called L0 hits, poses a computational problem. The problem is particularly hard because due to the nature of the causality criterion, there are no transitive properties that can be exploited; if hits a and b are correlated, as well as hits b and c, it is not necessarily the case that a and c are related. Besides using restrictive causality criteria as those above a way to reduce the required CPU power is to pre-select hits that are coincident with another hit on a DOM within a timewindow of about 10 ns. These hits are called L1 hits. By additionally exploiting the directionality of the PMTs on a DOM, this can be reduced even further. The amount of hits to be considered is typically reduced with a factor of 400. This, however, increases the trigger theshold as for example on average 10 L1 hits requires a higher energy neutrino than 10 L0 hits. By combining L1 hits with L0 hits which are close in distance, the threshold can be lowered. This latter optimization is in particular efficient for electron-neutrino events in ORCA. An important ingredient to the algorithms above is sorting the hits in time, either locally (DOM) to find L1 hits, or globally to seed the clustering algorithm. Sorting takes currently 50 to 70% of the CPU power, depending on the trigger algorithm and parameters.

Performance
Given the considerations above, the required CPU power for one block of 115 DUs with a PMT countrate of 7KHz is around O(50) cores for the optical triggering. In ARCA this leads to a trigger rate due to atmospheric muons of about 100 Hz, with a negligible rate due to 40 K and bioluminescence. The efficiency depends on the channel, but for all channels it is above 80% for neutrino energies above 10 TeV. In ORCA the trigger rate on atmospheric muons is about 40 Hz, and for the random noise 20 Hz, while the efficiency for atmospheric neutrinos with an energy above 4 GeV exceeds 80%.

Status
In 2015 the first DU was deployed at the ARCA site. Since then, 2 more lines have been operated of which one in ORCA configuration at the French site. The whole data-acquisition system has been validated with those DUs. Currently the KM3NeT Collaboration is constructing Phase-1 (6 ORCA, 24 ARCA DUs) and preparing Phase-2 (1 ORCA block, 2 ARCA blocks).