A novel centralized slow control and board management solution for ATCA blades based on the Zynq Ultrascale+ System-on-Chip

. Data acquisition systems (DAQ) for high energy physics experiments utilize complex FPGAs to handle unprecedented high data rates, especially in the ﬁrst stages of the data processing chain. The complexity of developing and commissioning these systems increases as additional local processing intelligence is placed closer to the detector directly on the ATCA blades. On the other hand, sophisticated slow control is as well desired. In this contribution, we introduce a novel solution for ATCA based systems, which combines the IPMI, a Linux based slow-control software, and an FPGA for custom slow-control tasks in one single Zynq Ultrascale + (ZUS + ) System-on-Chip (SoC) module.


Introduction -The Phase-2 tracker electronics system
In the context of the high-luminosity upgrade of the LHC (HL-LHC), the level-1 CMS trigger system will be upgraded by adding tracking and high-granularity calorimeter information [1]. The CMS silicon tracker will be replaced with a novel design, based on double-sided detector modules arranged in a tilted geometry [2]. The new modules can select pairs of hits compatible with high transverse momentum tracks (p T > 2-3 GeV), distinguishing them among a large number of background hits of little interest for use in the level-1 trigger. The selected pairs of hits, known as "stubs," are then sent to the back-end system at the full collision rate, where they are reconstructed into candidate tracks that will contribute to the global level-1 trigger decision. Furthermore, the modules are also capable of streaming the full set of hits of those triggered events to the data-acquisition system.
In the back-end electronics system, track reconstruction is done based on a timemultiplexed architecture (shown in Figure 1) requiring the use of two layers of data processing [3]. The first layer is represented by the Data, Trigger, and Control (DTC) boards, which extract and pre-process the stub data -for example, by converting the position from module-local coordinates to experiment-wide 3-dimensional coordinates -and route it to the correct card in the Track Finding layer. Routing of stubs follows a set of rules aimed at splitting the data as a function of time and geometrical origin in the detector (time multiplexing is based on event number and azimuthal sector position). The DTC cards are also responsible for interpreting and repackaging the raw hit data from triggered events, sending the latter to the DAQ & TTC Hub (DTH) cards. They also provide the control and timing signals to the tracker front-end modules. The second layer, the Track Finding Processor (TFP), receives all data for a given event and reconstructs the trajectories of charged particles in the tracker.

Hardware Developing Platforms
CMS has adopted the Advanced Telecom Communications Architecture (ATCA) as a platform for the Phase-2 back-end electronics. Three types of ATCA node blades (the OT-DTC, IT-DTC, and the OT-TFP) will be involved in back-end processing for the Phase-2 Tracker. Most of the design and operating requirements for these blades are common. Currently, two hardware approaches are being developed as part of the R&D program within the level-1 trigger community [5]. The Serenity [6] board uses a commercial COM-Express Type-10 module with an Intel Atom CPU running CentOS Linux, assisted by a small Artix7 FPGA for low-level control (e.g., I 2 C, JTAG). The COM-Express communicates to the main FPGAs via PCIe Gen2 protocol and uses the CERN-IPMC [7] for shelf management tasks running the Pigeon Point commercial software [8]. It has two processing sites where different FPGAs can be mounted using a Samtec Z-Ray interposer interconnect. Both processing sites support up to 96 optical links at 28 Gb/s, using Samtec Firefly optical modules.
The Apollo [9] board separates the blade physically into generic infrastructure and application-specific parts. The Apollo "Service Module" (SM) carries a seven series Xilinx Zynq System-on-Module (SoM) [10] providing a dual ARM-A9 processor and FPGA logic in the same package. It communicates with the main FPGAs via two lanes of AXI Chip to Chip and can either mount the CERN-IPMC or the UW-IPMC [11] by changing resistor configurations in the board. The Apollo "Command Module" (CM) could be tailored to each application. The current configuration contains two FPGAs with several hundred optical links with speeds up to 16 or 25 Gb/s, where Samtec Firefly modules are used for the optical connections as well. The CM contains a micro-controller performing temperature monitoring and low-level configuration for different devices.

Unified slow control architecture
Several efforts are being made to develop common hardware infrastructure based on the ATCA standard, which could support the technical requirements of the CMS level-1 track trigger. Some of the challenges include the appropriate selection of slow-control and shelf manager devices to enable full control of the blade and seamless integration with other subcomponents of the CMS experiment.
Current prototypes contain small FPGA parts, micro-controllers, and SoCs to perform specific tasks that could be all integrated into a single heterogeneous device like the Zynq Ultrascale+ (ZUS+) from Xilinx as shown in Figure 2.
The ZUS+ SoC provides FPGA logic, high-performance ARM-A53 multi-core processors, and two ARM-R5 real-time capable processors. The ZUS+ is divided into four power domains, which can be enabled or disabled independently.
The ARM-R5 cores are used to implement time-critical and deterministic tasks using freeRTOS or bare-metal applications. They provide the Intelligent Platform Management Controller (IPMC) functionality and communicate via the backplane of the ATCA crate with the shelf manager to negotiate the card power-up and subsequent stable operation. The ARM-R5 are also connected to the power supplies (via PMBus), to voltage and current monitors, to clock generators and jitter cleaners (via I 2 C, SPI) providing the required configuration at start-up.
Once full power is enabled from the crate, a CentOS Linux based operating system starts on the ARM-A53 cores. The FPGA is used to implement some of the low-level interfaces, including IPBus or glue-logic. The SoC is the central entry point to the main FPGAs on the motherboard via IPMB and TCP/IP based network interfaces. The communication between the ZUS+ SoC and the main FPGAs uses the AXI chip-to-chip protocol via MGT pairs keeping infrastructure requirements in the main FPGAs to a minimum.

Setup Description -Trenz to Serenity Adapter
An adapter board (in Figure 3) was designed to map the COM-Express and CERN-IPMC footprint found on the Serenity Board to a commercially available ZUS+ module manu-factured by Trenz Electronic [12] featuring the XCZU4EG device. The adapter uses two voltages, 3.3 V, and 1.8 V, which are used to power all components. The main 3.3 V is either generated by regulating down the "Payload" 12 V voltage, or by connecting the "3.3 Standby" power present in the CERN-IPMC SODIMM. The TPS2121 priority power multiplexer allows the adapter to receive power from either one source or switch between them seamlessly. Several peripherals were integrated inside the adapter providing different functionalities: • USB: The USB3340 physical layer (PHY) is used to provide USB 2.0 host capabilities, it uses the low pin interface (ULPI) to connect to the MIO Pins at the ZUS+ device and a USB type A female connector present in the motherboard.
• Ethernet: The 88E1512 Ethernet PHY serves ethernet connectivity to the module, the RGMII interface connects to the MIO pins while the Media Dependent Interface (MDI) is connected to the COM-Express pinout which in turn has wired the Ethernet switch present in Serenity. This switch grants connectivity to the two back-plane base interfaces and ETH links to other devices on the board.
• microSD: A self-designed 7x7 mm 24 pin QFN adapter was produced containing the NVT4857UK chip. It allows the SD 3.0-SDR104 protocol with up to 104 MB/s data rate. It significantly increases the speed of reads and writes to the memory. The microSD adapter in Figure 4 was done so that the small 0.4 pitch WLCSP package could be contained in its own project where appropriate production quality measures were implemented (e.i x-ray imaging for a whole batch production).
• Sata: One processor-high-speed-transceiver is connected to an M.2 SATA SSD present on the motherboard. The Linux operating system can recognize the drive, format it, and mount it as a persistent storage unit.
• On-board Programmer: A micro USB port provides a JTAG and two UART interfaces powered by the USB connector and using the FTDI FT4232 chip. 5 V are regulated down to 3.3 V reducing the load from the 3.3 V standby power when the circuit is not needed.
• AXI-C2C: The programmable logic MGTs of the ZUS+ are connected to each of the main processing FPGAs and the auxiliary FPGA present on the motherboard, it is possible to communicate via AXI-C2C or PCIe.
• IPMC: Two independent I 2 C buses are available at the low power domain peripherals for communication with the shelf manager using the hot-swappable controller LTC4300-1. The IPMC starts by communicating the transition from the virtual state M0 to state M1, where the FRU is inactive. The ShM requests the Device ID and information about the installed FRU like the name, firmware version, and supported features. When the front panel handle is in the closed position, the IPMC notifies the transition M1-M2 state, and the ShM responds, requesting the SDRs of the blade. The ShM instructs the transition M2-M3. In M3, the power negotiation takes place to allocate available power in the crate according to the board requirements. The FRU information is also requested by the ShM to know which E-Key interfaces to enable. At this point, the board enables the payload power, configures the programmable logic, and launches the uboot loader followed by CentOS7 on the A53 cores. Then the IPMC notifies the M4 FRU active state to the ShM. The deactivation process starts by opening the handle or by request of the ShM bringing the FRU state back to M1.

Pigeon Point IPMC Software
The Pigeon Point IPMC software is a commercially available software stack that has been tailored and customized for utilization on the ZUS+ architecture. It is based on an implementation targeting other processor architectures. Therefore the implementation on ZUS+ is not yet an off-the-shelf choice but rather an on-going development here presented. Inside the software, there is the functionality to select where peripherals are connected utilizing a hardware abstraction layer, minimizing the number of files that need user modification. However, this does not apply to the more underlying features requiring specific architecture functions to instruct the PMU of the ZUS+ device, for example.
The Pigeon Point implementation is extensive and complete. However, not all the features in the code apply to the setup and have not been validated using the ZUS+ architecture. Some require specific hardware components that are not present in the current setup. Further enabling and validation of features will take place in the future.

OpenIPMC Software
OpenIPMC [13] is a free and open-source firmware designed to operate as an Intelligent Platform Management Controller (IPMC). It is based on the FreeRTOS real-time operating system, and it is designed to be architecture-independent, allowing it to be run on a variety of different processor architectures. Having open-source code is novel for this kind of firmware, allowing full customization by the user.
The OpenIPMC makes use of signal call-backs and a hardware abstraction layer to hide from the main functionality the specifics of the hardware platform it is running on. The code has been tested on two different hardware platforms with identical results on the implemented functions. The code has been written to prioritize those core functionalities in the ATCA standard, which are essential and continued towards those required but not critical. So far, the dual multi-master IPMB_A/B interface has been implemented, bringing the board to the state where the payload is active. Power-up and -down sequences were verified, signaling the transition states using the blue ATCA LED in the front panel. FRU information is stored and read from an EEPROM memory.

Results
The IPMC functionality was verified via its redundant dual connection to the backplane with two independent software implementations achieving the mandatory power-up sequence, negotiating the different ATCA states with the shelf manager. The software "ATCA Tester" from Polaris Networks [14] was used to test the level of conformity against the standard with the Pigeon Point implementation. Of the 106 tests performed, 72 were successful, and 34 failed. The faulty features were not enabled in the software at the time of testing and therefore expected to fail. The OpenIPMC implementation was tested in an ATCA crate but not formally using the conformance test yet.

Conclusion
Future DAQ systems will require local processing intelligence for slow-control, monitoring, and calibration. Heterogeneous SoC architectures allow integration of these tasks into a single device. It was demonstrated that a unified slow control architecture based on Zynq Ul-trascale+ SoC is possible. Implementing time-critical applications like the IPMC on the R5 cores and a CentOS Linux on the application A53 processors was achieved utilizing isolation of memory and unique assignment of peripherals to the two software stacks. Two IPMC implementations were explored; a full-featured commercial solution and a basic-feature opensource version. Though this is still work in progress, the results are very promising. Our future work will focus on implementing the missing features and testing in a realistic environment.