The DAQ systems of the DUNE Prototypes at CERN/

DUNE is a long baseline neutrino experiment due to take data in 2025. Two prototypes of the DUNE far detector were built to assess candidate technologies and methods in advance of the DUNE detector build. Here are described the data acquisition (DAQ) systems for both of its prototypes, ProtoDUNE single-phase (SP) and ProtoDUNE dual-phase (DP). The ProtoDUNEs also break records as the largest beam test experiments yet constructed, and are the fundamental elements of CERN’s Neutrino Platform. This renders each ProtoDUNE an experiment in its own right and the design and construction have been chosen to meet this scale. Due to the aggressive timescale, off-the-shelf electronics have been chosen to meet the demands of the experiments where possible. The ProtoDUNE-SP cryostat comprises two primary sub-detectors a single phase liquid Argon TPC and a companion Photon Detector. The TPC has two candidate readout solutions under test in ProtoDUNE-SP RCE (ATCAbased) and FELIX (PCIe-based). Fermilab’s artDAQ is used as the dataflow software for the single phase experiment. ProtoDUNE-DP will read out the dual-phase liquid argon detector using a microTCA solution. The timing, triggering, and compression schemes are described for both experiments, along with mechanisms for sending data offline to permanent data storage in CERN’s EOS infrastructure. This paper describes the design and implementation of the TDAQ systems as well as first measurements of their performance.


Introduction
The Deep Underground Neutrino Experiment (DUNE) [1] is a leading-edge, international experiment for neutrino science and proton decay studies. An intense beam (1.2-2.3 MW) of neutrinos will be fired from Fermilab at a large detector 1300 km away. The physics goals include the study of CP violation in the lepton sector, proton decay, and the study of supernovae. The far detector will comprise four 10+ kt liquid argon (LAr) supermodules 1 . The supermodules host liquid argon time projection chambers (LAr-TPC), with excellent 3D imaging and energy measurement capability, and include integrated photon readout.
Two prototypes have been built at CERN, representative of alternative designs. The first of which is a single phase liquid argon cyrostat, using proven technology used in other liquid argon neutrino experiments such as ICARUS [2] and MicroBooNE [3]. The DUNE singlephase supermodule will consist of 150 such Anode Plane Assemblies (APAs) measuring 2.3× 6 m.
The second prototype, a dual-phase argon cryostat, offers several potential advantages over the single-phase approach, with an gas amplification layer above the liquid surface. The corresponding DUNE dual-phase supermodule will consist of 80 Charge Readout Planes (CRPs). [Right] ProtoDUNE-DP cryostat with charge readout planes and vertical E-field Figure 1 shows cutaway views of both the single and dual phase ProtoDUNE cryostats. Both the single and dual-phase detectors are designed to demonstrate the viability of detector modules, associated electronics and readout solutions. In some cases, more than one candidate technology has been installed, such that their performances can be compared. The detectors use full-size prototype detector planes to emulate a slice of the DUNE Far Detector supermodules.
The prototype detectors have been installed as an extension to CERN's North Area beam test facility. Unlike DUNE, they are situated at the surface and as such, their dominant signal source is from cosmic rays rather than the beam delivered by the CERN Super Proton Synchrotron (SPS).

ProtoDUNE-SP
The ProtoDUNE-SP TPC consists of six Anode Plane Assemblies (APAs) each with 2560 wires, see figure 1 left. Light bars are inserted along the length of each APA for photon collection. The cryostat is flanked front and back (with respect to the beam direction) by a set of Cosmic Ray Taggers used primarily to select cosmic muons traversing the cryostat. The data acquisition system (DAQ) is electrically isolated from the cryostat through optical fibre. This has been done to minimise electrical noise at the front-end electronics. Data are triggered and compressed online in hardware and software prior to event building. Events sent offline at up to 20 Gb/s. Figure 2 shows the dataflow through the ProtoDUNE-SP DAQ system. The electronics are driven by a 50 MHz clock via the timing system and events are triggered and selected based on input from beam instrumentation, a cosmic ray tagger and the photon system.  Figure 2. Diagram overviewing the logical data flow and connections in the ProtoDUNE-SP DAQ system. Data flows from the TPC and photon detectors (bottom left), through their readout boards to the event-building farm, where the events are stored in a temporary data store prior to being transferred offline to CERN's EOS storage system and subsequently replicated to Fermilab.

Front-end Electronics
The TPC wire planes and the photo-detectors inside the liquid argon are read out via digitisation electronics on the top of the cryostat. The TPC is read out via Warm Interface Boards 2 (WIBs) and the photo-detectors are read out via Silicon Photo-multiplier Signal Processor modules (SSPs). These two board types act as the interface from the cold electronics to DAQ and the primary data producers for ProtoDUNE-SP. As previously mentioned, both systems have optical connections to and from the DAQ to minimise noise to the front-ends. The WIB multiplexes data from four front-end boards, and five WIBs serve an APA, totaling 30 WIBs for ProtoDUNE-SP. The WIBs continuously send data to the DAQ and triggering is performed further downstream. Four SSPs read light signals from 10 light bars installed along the length of the APA, totaling 24 SSPs for the detector. They contain a high-speed waveform digitiser and a constant fraction discriminator that can be used to trigger on light signals detected. This is used primarily for identifying cosmic rays in ProtoDUNE-SP.

Timing system
The ProtoDUNE-SP timing system provides a 50 MHz clock to all endpoints multiplexed on an 8b10b encoded data stream. Each timing endpoint (such as WIB, or SSP) receives the data stream over a single mode optic-fibre and a clock-data recovery circuit is used to demultiplex the clock and data. The timing system interfaces to the CERN SPS beam presence signals and can be used to switch modes for data taking with and without beam. A dedicated interface is used to the central trigger board; the timing system data stream provides the trigger distribution. The timing system is partitionable allowing parts of the experiment to run independently. There is also an inhibit interface to allow throttling of triggers if some part of the system exhibits back-pressure due to too high data rate. The source clock is driven by a GPS clock provided by CERN and 64 bit timestamps are employed to provide unambiguous event time in data irrespective of file name, or event/run numbers.

Central Trigger Board
The trigger system takes inputs from beam instrumentation (scintillators and cherenkov counters), the Photon Detector system, and the Cosmic Ray Tagger. The trigger system is based on Xilinx Zynq 7020 -MicroZed System-on-Chip. It has approximately 100 inputs including optical, NIM, TTL, LVDS. Two trigger levels (low and high) have been developed in firmware and can be reconfigured at run time. Multiple combinations of triggers can be employed, including the ability to prescale certain triggers to limit their rate or to veto on certain combinations. This versatility allows ProtoDUNE-SP to select particles based on their particle ID -a requirement for the physics program of the experiment.

TPC Readout
The TPC has two candidate readout solutions -RCE [4] and FELIX [5]. The both read continuous data streams from the WIBs -the data payload being identical in both cases albeit with different transport protocols.

RCE
The Reconfigurable Cluster Element (RCE) is an ATCA-based readout solution used in several experiments -LSST, Heavy Photon Search, ATLAS Muon. The RCE is designed to put a lot of processing power into a small form-factor (the entire ATCA crate occupying 12 U rack space). Nine clustered processing elements are mounted on a COB (cluster-on-board). Each of the RCEs is based on a Xilinx Zynq 7045 SoC, Dual core ARM A9 with 1 GB DDR3 RAM. Interconnecting the RCEs and the periphery is a 10 Gb/s onboard switch. The experiment interface is customised with a Rear Transition Module (RTM), which for ProtoDUNE-SP has 4× QSFP+ input transceivers which receive data from the WIBs. The ProtoDUNE-SP RCE firmware uses "High Level Synthesis" for C++ like algorithmic development for deployment on FPGA fabric. This allows for more rapid development of sophisticated algorithms on the FPGA. The RCE selects the triggered data coming from the WIBs based on the trigger timestamp. The selected data are compressed on the FPGA prior to being sent for event building.

FELIX
The Front-End LInk eXchange (FELIX) is a PCIe-based solution designed for the ATLAS LAr Calorimeter for the Phase-I detector upgrade. The FELIX card is based on the Xilinx Kintex Ultrascale FPGA. It has 48 duplex optical links capable of 14 Gb/s each. The PCIe interface is Gen3 x16 capable of approximately 100 Gb/s effective data rate. It has onboard DDR4 memory up to 16 GB. FELIX is used to read out 1 APA (one sixth of the TPC) for the experiment. Triggering and compression are performed in software. Compression is accelerated using Intel QuickAssist TM Technology.

Dataflow software -artDAQ
artDAQ [8] is a data acquisition software toolkit developed at Fermilab as a generic solution that can be adopted and customised for high-energy physics experiments. It is predominantly used in neutrino experiments. artDAQ features include: hardware readout/configuration; event building; data logging; infrastructure for filtering, compression, online analysis; loadbalancing; data quality monitoring; a configuration database for storing/retrieving hardware and software configurations. Figure 3 shows the dataflow among artDAQ processes.  Figure 3. Data flow from the hardware (RCE, FELIX, SSP) to a set of customised BoardReader process which are responsible for assembling and tagging triggered data into artDAQ fragments. These fragments are requested by the event builders and combined to make complete events. The events are logged to storage by DataLoggers and, in parallel, a fraction of events are sent to the online monitoring system.

Run Control
Run control (see Figure 4) for ProtoDUNE-SP is based on CERN's JCOP (Joint COntrols Project) [6] extension to Siemens WinCC-OA framework. This framework was developed to serve the needs of the LHC experiments. For ProtoDUNE-SP, it has been interfaced to artDAQ. A finite state machine is used to represent the states of the artDAQ processes, and initiate their actions (configure, start run etc.). The run control exploits the partitioning enabled by the timing system to allow for parallel operation of separate parts of the system. It is the primary front-end to data taking; interacts with a Run database backend; catalogs run information, and submits to a logbook. It is the essential tool for error information and problem diagnosis. Run control also acts as the front-end to operational monitoring, such as data sizes and rates. The artdaq processes report their metrics for these and other quantities to the run control, and are archived to CERN's central Oracle databases.

Online Monitoring
The online monitoring is used to assess the detector performance during data taking and provide feedback for calibration. It is based on artDAQ and LarSoft [7] physics software for LAr-TPC experiments. Raw decoding software are used to unpack the artDAQ events and perform low-level analysis. High-level analyses are also performed, but at a lower rate (due to processing time). The resultant histograms are saved and propagated to an online web display called Monet [9] developed for the LHCb experiment and adapted for ProtoDUNE-SP.

ProtoDUNE Dual-Phase
The dual-phase detector readout has similar constraints to the single-phase detector. The DAQ and electronics on the cryostat are electrically isolated via optic-fibre. The bandwidth to offline storage is 20 Gb/s. The differences in DAQ strategy between the two detectors starts close to the front-end electronics. Compression begins in the readout cards installed on top of the cryostat. A large data volume is continually written to disk and a large processing farm is used for online reconstruction and data quality.

Front-end electronics
The TPC signals are read out via the top of the cryostat and digitised externally. The readout cards conform to the microTCA standard. There are 12 microTCA crates to read out the entire TPC and one crate for light readout. The signals from charge readout plane (CRP) of the TPC are read out via an AMC card with an Intel Cyclone GX FPGA with NIOS processor with a 12 bit ADC and 10 Gb/s connectivity. Lossless Huffman-like compression (factor 10) is performed on the FPGA.
Light readout (LRO) is based on the same architecture as the charge readout AMC card. The light readout card can trigger its readout based on internal light activity from the photomultiplier tubes in the cryostat or external beam triggers.

Timing and Trigger
Timing is based on the White Rabbit (WR) project [10] which offers sub-1 ns accuracy. Figure 5 left shows a White Rabbit Grand Master switch is used to interconnect all endpoints. A trigger PC with a WR slave is used to timestamp external triggers from beam instrumentation, cosmic ray counters and the light readout system. The microTCA DAQ architecture is integrated with the WR network via WR slave in the microTCA crates. A high-quality GPS-based clock provided by the CERN Control Centre is used as the primary clock source.

Event Building
ProtoDUNE-DP has a two stage event building architecture, see Figure 5 right. There are two Level-1 event building PCs (DELL R730, 256 GB RAM) each with 8 10 Gb/s links for the incoming data stream and one 40 Gb/s link for output to Level-2. Each of the L1 event building PCs collates data for a drift (each PC does half-detector). Four high-end servers (DELL R630, 128 GB RAM with one dual-port 40 Gb/s network link) form the second level of event building hardware. Level-2 assembles whole event from half-event fragments and prepares multi-event files for writing offline. Interconnectivity is provided via a Brocade ITX7750 26 port switch at 40 Gb/s.

Storage/Processing
The online storage and processing facility has 1 PB of storage with 384 core for processing, with an internal bandwidth of 20 GB/s. It is a key element of the online analysis, which includes identification of cosmics, purity measurement, gain and event filtering. During operation, a dedicated operator is responsible for looking at the results of the online processing to ensure data quality.

Status and Preliminary Results
Here are described the current status of the single and dual phase prototypes. The latest data challenge is outlined showing performance of the DAQ and offline systems in terms of bandwidth, throughput and storage. ProtoDUNE-SP managed to take beam data in September and October of 2018 and preliminary results are given. ProtoDUNE-DP is currently being commissioned and is due to take cosmic data in 2019 -results of a smaller dual-phase prototype are given in [11].

Data Challenge
"Data challenge 2" involved the data acquisitions systems for the single and dual-phase detectors and offline computing from CERN and Fermilab. Monte-Carlo data was generated to simulate the data writing of the DAQ systems at their design data rates (25 Hz). In both SP and DP cases, continuous data read from the online data storage systems was possible without impeding the simultaneous writing of data to the same storage systems. As such, no unwanted backpressure would be introduced into the DAQ from the offline data transfer. Full 20 Gb/s bandwidth was sustained from the experiments to EOS in CERN's IT department for 24 hours.

ProtoDUNE-SP Beam Data
The DAQ, trigger and electronics performed to their expectation (and are still taking data at time of writing), and managed to exceed the baseline data rates described in the technical design report (25 Hz). Rates of 60 Hz were achieved with a 36 MB event size (≈ 17 Gb/s) The results mark the performance and success of the experiment in all aspects of construction, installation and commissioning. Figure 6 shows a candidate beam event among cosmic rays as seen in the ProtoDUNE-SP online monitoring. Data seen in both the collection and induction planes signifies the excellent HV, noise performance and argon purity in the experiment. Figure 6. Event view from the online monitoring display of the collection plane (left) and one of the TPC induction planes (right). Focus is given to the region around the beam window. A shower event can clearly be seen in both images. The data have a per-wire fixed pedestal subtraction but are otherwise unprocessed raw data.