KM 3 NeT tower data acquisition and data transport electronics

In the framework of the KM3Net European project, the production stage of a large volume underwater neutrino telescope has started. The forthcoming installation includes 8 towers and 24 strings, that will be installed 100 km off-shore Capo Passero (Italy) at 3500 m depth. The KM3NeT tower, whose layout is strongly based on the NEMO Phase-2 prototype tower deployed in March 2013, has been re-engineered and partially re-designed in order to optimize production costs, power consumption, and usability. This contribution gives a description of the main electronics, including front-end, data transport and clock distribution system, of the KM3NeT tower detection unit.


Introduction
The KM3NeT tower is a semi-rigid three-dimensional structure composed of 14 floors and a basemodule (Fig. 1a).The floors are 8 m long structures that, after unfolding, are vertically spaced by 20 m, with the lowermost one located 100 m above the sea bottom, and are mechanically interlinked by a set of tensioning ropes kept taut by a system of buoys at the top of the tower.The base-module anchors the whole structure to the seabed.In the next sections, an overview of the layout of the KM3NeT tower and a brief description of the electronics devoted to data acquisition and data transport will be given.

Floor electronics
Each floor supports 6 Optical Modules (OMs), two digital hydrophones, and an optional oceanographic instrument (Fig. 1b).A Center Floor Vessel (CFV) is located at the center of each floor, and is connected to the aforementioned peripherals by dedicated electrical cables, one per each peripheral.Each CFV is in turn connected to the tower-base module by a dedicated electro-optical cable containing two optical fibers and two conductors.a e-mail: carlo.nicolau@roma1.infn.itThis is an Open Access article distributed under the terms of the Creative Commons Attribution License 4.0, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.From the standpoint of the data communication and connections topology, the tower is thus organized hierarchically, starting from the tower-base module, passing through the floors CFVs, up to the floor peripherals.Each floor is assigned a pair of wavelenghts in the Dense Wavelength Division Multiplexing (DWDM) spectrum to allow bidirectional communication; the wavelenghts are optically multiplexed/demultiplexed at the tower-base, where a pair of optical fibers provide the link to the shore station.A system of junction boxes provides the connectivity for the various detection units [1].

Optical module electronics
Each OM consists in a 13 "glass sphere containing a 10" photomultiplier (PMT) with its high voltage power supply and the Front-End Module (FEM) board.Two OMs per floor are equipped with a LED beacon, which can be used to calibrate the detection unit [2].The FEM is a 9 cm by 8 cm board mainly dedicated to acquire the pulses generated by the PMT and transmit them to the CFV, and to perform slow-control tasks at OM level, including high voltage control, environmental monitoring, and LED beacon control.
The FEM board is built around a Xilinx Spartan 6 FPGA and an ADS4149 (a 14 bit differential FastADC by Texas Instruments) and is connected to the CFV by means of three differential twisted pairs using Low-Voltage Differential Signaling (LVDS) standard.One of the pairs is used to receive the clock signal generated by the CFV, one pair is used to receive slow-control commands and timing synchronization information, one pair is used to send the acquired data and the slow-control data back to the CFV.The common modes of two of the twisted pairs are used to feed the 5 VDC power.
The FEM board was designed with an "all-data-to-shore" philosophy: no pre-elaboration is performed and the most of the acquired information (in this case, the waveforms generated by the PMT) is transmitted to the shore-station as far as possible.In particular, the current pulses generated by the PMT are low-pass filtered and digitized at 200 Msps; in order to retain only the meaningful part of the acquired waveform the FPGA logic implements a zero-skipping mechanism with programmable threshold.The digitized pulses are locally time-stamped and put in a 2 ksample FIFO.Although timestamping is performed at the ADC sampling rate (5 ns), hit times can be calculated with a precision of the order of 200 ps by implementing interpolation algorithms at the shore station.The acquired pulses are then 8b/10b encoded and sent to the CFV at a maximum rate of 20 Mbps.To further reduce the produced datarate (and therefore sustain higher hit-rates) a non-lossy compression is implemented on this data channel; in particular, low level signals such as those of single photo electrons (s.p.e.) are Very Large Volume Neutrino Telescope (VLVnT-2015) encoded using an 8 bit sample word size.Given the 50 kHz average s.p.e.rate at the selected submarine site, the average data-rate produced by each FEM is of the order of 6 Mbps and the maximum sustainable s.p.e.rate if of the order of 150 kHz.
The FPGA logic also embeds a microblaze-based system on chip that performs the slow-control duties, including control and monitoring of the analog front-end and the high voltage power supply, temperature and humidity sensor monitoring, and full remote re-programmability.

Center floor electronics
The CFV hosts two electronic boards: the Floor Control Module (FCM) and the Power Supply System (PSS).
The main tasks of the FCM consist in providing a communication channel for each of the floor peripherals (the 6 OMs, the two digital stereo hydrophones, and the optional oceanographic instrument), distributing clock and timing information, distributing, controlling and monitoring their power and performing monitoring at CFV level.The link toward the shore station consists in a DWDM standard electro-optical Small Form-factor Pluggable (SFP) transceiver that is attached to a couple of single-mode optical fibers contained in the back-bone cable connecting the CFV to the tower-base module.A custom self-synchronizing 800 Mbps communication protocol embedding clock, data and timing information is implemented in a constant-length frame-oriented fashion.Each peripheral in the floor (including the FCM itself) is assigned a sufficient and constant amount of space in each frame payload, following a time-division multiplexing scheme.
The heart of the FCM board is a Xilinx Spartan 6 FPGA which implements the main functionalities, including the fixed-latency transceivers, the PLLs used for generating local and distributed clocks, the communication interfaces to the 6 OMs, a couple of stereo AES3 standard receivers for the hydrophones, two RS232 ports (externally opto-isolated), and a microblaze-based system on chip.The latter is in charge of performing all the slow-control duties consisting in monitoring the power consumpion of all the floor peripherals (by means of dedicated onboard ADCs), controlling the power switches, the optical transceiver (including wavelength shift monitoring and tuning), and the on-board temperature and humidity sensor.Also, it implements full remote re-programmability.
The PSS board provides the various DC voltages needed by the floor peripherals, including sequenced 1.2 V and 3.3 V for the FCM itself, 5 V for the OMs, and 12 V for the digital hydrophones and the optional external oceanographic instrument.The PSS is built around a Vicor DC/DC converter which receives the 375 V DC voltage from the tower-base module through a couple of conductors in the back-bone and produces a 5 V voltage that is in turn converted to the other needed voltages by separate low power DC/DCs.

Tower-base electronics
The tower-base is primarily devoted to provide connectivity and power to all the tower devices; it is connected to the shore station, through the junction boxes, by a pair of optical fibers (one per data direction) each carrying 15 DWDM wavelengths (14 for the floors, one for the tower-base itself).It includes three pods -the photonics pod, the electronics pod, and the power pod -and supports a digital hydrophone and an optional laser beacon.
The photonics pod contains the optical components to multiplex/demultiplex the DWDM wavelengths from/to the CFVs and the electronics pod to/from the two optical fibers coming from the junction box.
The power pod is oil filled and contains the Power Control System (PCS) board that implements the set of switches working at 375 V DC voltage that independently control the power lines of the floors.Both voltage and current of each floor are monitored.A Microchip PIC microcontroller communicates 05011-p.3 EPJ Web of Conferences with the electronics pod in order to actuate switch commands and provide monitoring information through a RS232 interface.In case of failure of the electronics pod, the microcontroller starts an autonomous procedure to sequentially switch the floors on at known times, while monitoring current absorption to prevent short-circuits.
The electronics pod contains the same electronics hosted by the CFV (a FCM and a PSS) dedicated to control and monitor the electronics pod and to provide connectivity to the photonics pod, the power pod, the digital hydrophone, and the laser beacon.

Data and timing link model
As already mentioned, the communication channel between the shore station and each of the FCM boards consists in a couple of wavelengths in the DWDM spectrum.In the shore station, an equivalent number of transceivers are hosted by a set of PCIe boards (the NaNet 3 boards [3]) that are devoted to communicate with the floors.Data collected from the floors are then sent to a CPU through the PCIe bus to be further elaborated.Each NaNet 3 board can host up to four SFP transceivers, allowing to communicate with up to four floors.
All the boards receive, from a single GPS receiver, a common clock together with an IRIG-B encoded timing reference signal, thus sharing the same time-base.The time information is then sent to the FCMs by means of a self-synchronous protocol embedding both data and timing in the same stream.The FCMs, in turn, propagate this timing information to all the floor peripherals, including the FEMs, which use it to time-stamp the PMT hits.Thanks to the fixed-latency implementation along all the relevant paths (including transceivers in the NaNet 3 boards, the FCM, and the FEM), the whole apparatus (i.e. the 8 towers and the relevant elements in the shore station) is fully synchronized and all devices are time-stamp aware with a sub-nanosecond accuracy.

Conclusions
In this paper the main electronics developed for the KM3NeT tower was outlined.The whole architecture was fully tested and characterized.Due to the necessity of extremely tight timings and the adoption of a custom self-synchronous transmission protocol, extensive tests where performed on the propagation of the timing information from the shore-station, passing through the whole transmission chain, up to the OMs; a jitter of the order of 50 ps was measured on the 200 MHz clock driving the FastADC.The presented electronics was mass-produced and is currently being integrated into the 8 towers that will be deployed in the next future.

Figure 1 .
Figure 1.Layout of a KM3NeT tower (a) and of a floor (b).