The Data Acquisition System for the ATLAS Phase-II Tile Calorimeter Demonstrator

The Tile Calorimeter (TileCal) is the central hadronic calorimeter of the ATLAS experiment at Large Hadron Collider (LHC). The TileCal readout system consists of about 10,000 channels. In 2025, the LHC will be upgraded leading into the High Luminosity LHC (HL-LHC). The HL-LHC will be capable to deliver an instantaneous luminosity up to seven times compared to the LHC nominal luminosity. The TileCal Phase-II upgrade will replace the majority of the on-detector and offdetector electronics using a new readout schema for the HL-LHC era. The on-detector electronics will digitize and transmit calorimeter signals to the off-detector electronics at the bunch crossing frequency. In the counting rooms, the off-detector electronics will store the digitized samples in pipelined buffers and compute reconstructed trigger objects for the first level of trigger. The TileCal Phase-II upgrade project has undertaken an extensive R&D program which includes the development of a Demonstrator module to evaluate the performance of the new clock and readout architecture for the HL-LHC. A Demonstrator module containing the upgrade on-detector readout electronics was built, tested during several test beam campaigns, and inserted into the ATLAS experiment. The Demonstrator module is operated and read out using a Tile PreProcessor (TilePPr) Demonstrator which enables backward compatibility with the present ATLAS Trigger and Data AcQuisition and the Timing, Trigger and Command systems. This contribution describes the components of the clock distribution and data acquisition system for the Demonstrator module, and its implementation in the ATLAS experiment.


I. INTRODUCTION
HE Tile Calorimeter (TileCal) [1] is the main hadronic calorimeter of the ATLAS experiment [2] at the Large Hadron Collider (LHC). It covers the central part of the experiment up to a pseudorapidity of |η|<1.7. This subdetector contributes to the measurement of hadrons, jets, taus, and missing transverse energy.
TileCal is divided into four cylindrical readout sections along the beam axis: one central long barrel (LBA, LBC) and two extended barrels (EBA, EBC). Each barrel is segmented azimuthally into 64 wedge-shaped TileCal modules.
As illustrated in Fig. 1, TileCal modules are composed of alternating layers of steel plates and plastic scintillator tiles, which are placed perpendicularly to the beam direction and radially divided into eleven sections. The light produced by a charged particle passing through a plastic scintillating tile is collected by Wave-Length Shifting (WLS) fibers and transported to photomultiplier tubes (PMTs) in the outermost part of the modules.
The WLS fibers from individual tiles are grouped in cells, with a dimension of ∆η × ∆Φ = 0.1 × 0.1 in the first two radial layers (A and BC) and ∆η × ∆Φ = 0.2 × 0.1 in the last layer (D). Each cell is read out by two PMTs, where Long Barrel modules are equipped with 45 PMTs and Extended Barrel modules with 32 PMTs. The complete readout of the detector is done using approximately 10,000 PMTs.
In the current data acquisition system, the analog PMT signals are amplified with two gains (1:64 ratio), digitized, and stored in pipeline memories in the front-end electronics at 40 MHz. Upon the reception of an accept trigger signal from the Level-1 trigger system, the selected events are transmitted via optical fibers to the Read-Out Drivers (RODs) [3] in the counting rooms at a maximum trigger rate of 100 kHz. Fig. 2 presents a block diagram of the present readout architecture in TileCal. Fig. 2. Sketch of the current readout architecture in TileCal, representing in detail all the on-detector components [4].
The Data Acquisition System for the ATLAS Phase-II Tile Calorimeter Demonstrator II. ATLAS PHASE-II UPGRADE The LHC will undergo a series of upgrades during the Long Shutdown 3 (2025-2027) leading to the High-Luminosity LHC (HL-LHC). The HL-LHC is aimed to deliver an instantaneous peak luminosity up to 7.5·10 34 cm -2 s -1 . It is expected that HL-LHC provides a total integrated luminosity of 4000 fb -1 by the end of the data-taking. The ATLAS Phase-II Upgrade [4] will accommodate all the subdetectors and its electronics to operate in the stringent radiation environment of the HL-LHC and with the elevated pile-up levels, which can reach up to 200 simultaneous collisions per bunch crossing.
As part of the ATLAS Phase-II Upgrade, TileCal will completely replace the on-detector and off-detector electronics implementing a new readout strategy with radiation tolerant electronics. The new readout architecture will be capable of providing full-digital information per bunch crossing to the ATLAS Trigger and Data Acquisition (TDAQ) system for the computation of complex trigger algorithms. Fig. 3 presents a block diagram of the TileCal upgraded readout architecture. The upgraded on-detector readout electronics will transmit digitized signals from the PMTs for every bunch crossing (~25 ns) to the off-detector electronics located in the counting rooms, about 100 meters away from the detector.
In the off-detector electronics, 32 Tile PreProcessor (TilePPr) modules will handle and store the detector data in pipeline memories until the reception of a trigger acceptance signal, activating data transmission to the ATLAS Front End LInk eXchange (FELIX) [5] system. The upgraded readout system will comprise 6,000 optical fibers and a total bandwidth of 40 Tbps.
In parallel to the detector data acquisition, the TilePPr modules will compute the cell energies per bunch crossing and will transmit the results to the Trigger and Data Acquisition Interface (TDAQi) boards, which will calculate and transfer pre-processed trigger objects to the first level trigger.

III. ATLAS TILE DEMONSTRATOR MODULE
As part of the Phase-II Upgrade Demonstrator program, a fullsize Demonstrator module containing all the upgraded readout electronics was constructed in 2014 as a Long Barrel module. The Demonstrator module was tested during several test beam campaigns between 2015 and 2018 in the H8 beam line of the SPS accelerator at CERN. The test beam setup included other TileCal modules instrumented with legacy electronics in order to study the performance of the new electronics.
In July 2019, the Demonstrator module was installed into the ATLAS experiment to complete the validation of the new readout electronics for the HL-LHC. The performance of the upgrade electronics has been studied with Charge Injection, Laser, and Cosmic runs, showing excellent performance in terms of low noise, signal quality, and timing.

A. Demonstrator on-detector electronics
The Demonstrator module consists of four independent minidrawers hosting 12 PMT blocks, all the services, and electronics required for its operation (Fig. 4). Each mini-drawer is composed of a mechanical aluminum substructure which supports one Mainboard [6], one Daughterboard v4 [7], one high voltage regulation board, and up to 12 PMT blocks equipped with upgraded 3-in-1 front-end boards (FEBs). For improved reliability and robustness, all readout electronics and power distribution are separated into independent halves each of them hosting 6 FEBs.
The upgraded 3-in-1 FEBs provide shaping and amplification with two gains (1:32 ratio) to the PMT analog signals, which are transmitted to the MainBoard for digitization. These FEBs also provide calibration capabilities and include an integrator readout chain for Cesium calibration and luminosity measurements. The upgraded 3-in-1 FEB provides better linearity and lower noise levels than the previous version.
One MainBoard digitizes the signals from 12 PMT blocks using dual 12-bits ADCs at 40 MSps for the two gain signals and 16-bit ADCs for the slow integrator signal. This board hosts 4 Altera Cyclone FPGAs to control and configure the FEBs.
The digitized samples are transferred for every bunch crossing to the DaughterBoard via LVDS lines. The DaughterBoard packs and transfers the digitized data to the offdetector electronics using four redundant GigaBit Transceiver (GBT) [8] links operating at 9.6 Gbps. The timing, control and configuration commands for the operation of the front-end electronics are received via a 4.8 Gbps GBT link from the offdetector electronics. Each DaughterBoard v4 is equipped with two Xilinx Kintex 7 FPGAs, two GBTx [9] ASICs, and two QSFP modules. The GBTx ASIC recovers the LHC clock with fixed and deterministic latency. The recovered clock is distributed for the PMT sampling and to drive the FPGA highspeed transceivers.
Finally, in order to keep backward compatibility with the current ATLAS trigger system, the adder cards sum the low gain analog signals of the FEBs to produce trigger towers, which are transmitted to the Level-1 Calorimeter trigger system.

B. Tile PreProcessor Demonstrator
The Tile PreProcessor Demonstrator (TilePPr) [10] is responsible for operating and handling the data coming from the Demonstrator module, acting as a bridge between the upgraded on-detector electronics and the current ATLAS TDAQ and Trigger, Timing, and Control (TTC) systems.
The TilePPr Demo (Fig. 5) is a double mid-size Advanced Mezzanine Card (AMC) form factor, which includes one Xilinx Virtex 7 (Main FPGA) and one Kintex 7 FPGAs (Trigger FPGA). The Main FPGA interfaces with four QSFP modules providing high-speed communication with the on-detector electronics through 16 GBT links operating at 4.8/9.6 Gbps. It also communicates with the ROD and FELIX systems for event data transmission. On the other hand, the Trigger FPGA is intended for the development and testing of complex preprocessing trigger algorithms for the HL-LHC. IV. DATA ACQUISITION SYSTEM The data acquisition system for the Demonstrator module is implemented according to the readout strategy envisaged for the HL-LHC and presented in Section II. In addition, the Demonstrator module provides analog trigger tower signals to the current Level-1 calorimeter trigger system as the rest of the legacy modules.
As explained in the following sections the Demonstrator module keeps backward compatibility with the ATLAS TDAQ system via the TilePPr Demo, which enables the translation of commands from the TTC system and transmits triggered data to the ATLAS TDAQ system.

A. Configuration and data readout path
The configuration for the front-end electronics is received in the TilePPr Demo from the TTC system via an optical fiber at 80 Mbps. The TilePPr Demo extracts the timing and configuration commands and translates them into upgraded commands before transmitting them via 3 downlinks to each DaughterBoard. Commands are decoded in the DaughterBoard FPGAs and then executed or forwarded to the MainBoard FPGAs for the configuration of the FEBs. Fig. 6 presents a complete block diagram with all the firmware blocks involved configuration and data readout of the Demonstrator module. The downlink communication uses the GBT protocol with Forward Error Correction (FEC), where the FPGA transceivers are configured to operate with fixed and deterministic latency at a data rate of 4.8 Gbps.
Regarding the data readout path, the TilePPr module receives continuously event data for every bunch crossing via 4 uplinks per DaughterBoard (16 uplinks in total), each pair sending redundant data from one mini-drawer side. Uplinks are implemented using a modified version of the GBT protocol with fixed and deterministic latency. The modified version doubles the original data rate to 9.6 Gbps and solves the Clock Domain Crossing between the on-detector and off-detector clock domains using Blind Oversampling Clock and Data Recovery units [11].
The Main FPGA extracts the integrator, monitoring and data samples routing them to the proper interfaces after tagging the data with the corresponding Bunch Crossing Identifier (BCID).
In the case of the data samples, they are stored in 96 pipeline memories (one per gain and channel) with a depth of 12.8 μs. Once a Level-1 trigger acceptance signal is received via the TTC interface, the selected events are extracted from the pipeline memories, packed, and transmitted to the legacy RODs (7 samples, 10-bit precision, 1 gain) and to the FELIX system (16 samples, 12-bit precision, 2 gains) using the proper data formats. The integrator and monitoring data are read out via IPbus [12] protocol upon request.
The Demonstrator module is fully integrated with the ATLAS TDAQ software, enabling the acquisition of calibration and physics runs together with the rest of the TileCal modules. From the point of view of the ATLAS TDAQ system, the Demonstrator module is operated and read out as any other legacy TileCal module. Fig 7. compares the average noise levels between the Demonstrator module and a legacy module obtained with dedicated runs through the ATLAS TDAQ software and the RODs. As observed, the Demonstrator module (LBA14) shows lower noise levels compared with the legacy module (LBA15).

B. LHC clock distribution
An adequate distribution of the LHC clock is required for the synchronization of the on-detector electronics with the accelerator. This is achieved through a combination of highspeed links with fixed and deterministic latency and GBTx ASICs.
First, the TilePPr Demo recovers the LHC clock from the legacy TTC system using an Analog Devices ADN2814 chip. The recovered clock is externally jitter-cleaned using a Texas Instruments CDCE62005 chip to obtain a high-quality clock to drive the FPGA transceivers. Then, the TilePPr Demo distributes the LHC clock to the on-detector electronics embedded with configuration commands via the downlinks.
In the on-detector electronics, one GBTx ASIC per DaughterBoard recovers the LHC clock from the incoming GBT link at 4.8 Gbps. The FPGAs of DaughterBoard receives the LHC clock and buffers it to the MainBoard for the digitization of the PMT signals. The LHC clock is also driven to the FPGA transceivers in the DaughterBoard for proper communication with the off-detector electronics.

V. CONCLUSIONS
The new HL-LHC conditions imply the complete redesign of the readout electronics of the Tile Calorimeter. The new readout electronics will operate in a high radiation environment using a new readout strategy able to provide full-digital input for the ATLAS trigger system with improved granularity.
A Demonstrator module was constructed in 2014 with the upgraded on-detector readout electronics. The Demonstrator module was tested in several test campaigns with particle beams between 2015 and 2018 to study the performance of the new readout electronics. The upgraded clock distribution and readout strategy for the HL-LHC was implemented for the operation of the Demonstrator module. The on-detector electronics transmit digitized data to the TilePPr Demo at the LHC frequency, which also distributes the LHC clock for the sampling of the PMT signals embedded in the downlinks.
The TilePPr Demo interfaces the Demonstrator module with the current ATLAS TDAQ and TTC systems, enabling backward compatibility. One of the goals of the Demonstrator program is to keep the Demonstrator module in ATLAS during Run-3 (2022-2025) to continue studying its performance in real operation conditions.