Upgrade of the KEDR detector DAQ system

The KEDR experiment is ongoing at the VEPP-4M e+e− collider at Budker INP in Novosibirsk. The collider center of mass energy range covers a wide spectrum from 2 to 11 GeV. Most of the up-to-date statistics were taken at the lower end of the energy range around the charmonia region. Planned activities at greater energies up to the bottomonia would lead to a significant increase of event recording rates and accelerator backgrounds, thus stressing the existing DAQ and trigger systems beyond their limits. The described DAQ upgrade plan includes: the redesign of the trigger electronics using modern components to improve the trigger decision time; the development of new readout processors using ethernet connections; new software for collecting events and electronics management; high level of parallelization of data transfers and events processing; improved reliability based on readout computing cluster with redundancy. The upgraded DAQ system is going to be very flexible and could be considered as a concept prototype of the projected BINP Super Charm-Tau Factory.


Introduction
The KEDR detector [1] (Figure 1) is a universal magnetic detector for experiments at the VEPP-4M collider and consists of the following systems and parts: a vacuum chamber, a vertex detector, a drift chamber, aerogel Cherenkov counters, scintillation counters, a cylindrical calorimeter based on liquid krypton, a superconducting magnet, a muon system, a magnet yoke and an endcap CsI calorimeter. The physics program of the KEDR detector is very wide and includes the following basic measurements: • the precise mass measurements of -the J/ψ, ψ and ψ(3770) mesons; D 0 and D ± mesons; and τ lepton in the region of low energies (E cm =3-6 GeV); -the Υ(1s), Υ(2s), Υ(3s), and Υ(4s) mesons in the high energy region (E cm =9-10 GeV); • the measurement of the R ratio (ratio of the cross sections for the reactions e + e − → hadrons and e + e − → µ + µ − ) in the energy range E cm =2-11 GeV; • the measurement of the Υ → hadrons cross section and investigation of two-photon processes; • the measurement of the probabilities of radiative transitions in cc and bb quark systems. The data acquisition system [2] (DAQ) of the KEDR detector is based on the KLUKVA 1 standard developed by the BINP in late 1980s. It is composed of special crates with a fast data bus and a set of electronic units. There are several types of KLUKVA DAQ units. Information boards (IBs) receive the signals from the detector elements, measure the amplitude, arrival time or shape of the input pulses, depending on the subsystem, and digitize them.
The readout processor (RP) reads data out of the IBs according to the program preloaded from a computer and saves them in one of its two memory blocks. Then data are transfered to the DAQ computers via special exchange boards.
A simplified DAQ scheme is presented in Figure 2. It consists of several large units: • Information boards; • Two-level hardware trigger; • DAQ synchronization subsytem; • Event building system; • Interface to the event storage system.
The planned physics activities at greater energies including bottomonia would lead to a significant increase of the event recording rate and accelerator backgrounds, thus stressing the existing DAQ and trigger systems beyond their limits. So an upgrade of the DAQ system is required to effectively collect data under the new conditions. The DAQ upgrade plan described below includes: • redesigning the trigger electronics using modern components to improve the trigger decision time; • developing new readout processors using ethernet connections for parallel data transfer and new synchronization system;  To maintain compatibility with the existing information boards and to reduce overall costs this upgrade touches only the readout processor on the KLUKVA side.
2 Upgraded DAQ hardware 2.1 Trigger system The trigger system consists of special electronic modules combining signals from several subsystems. The hardware trigger of the KEDR detector includes the primary and secondary triggers. The third-level trigger is provided by the software.
The primary trigger (PT) of the KEDR detector is located near the detector, side by side with the electronics forming the signals for it. The primary trigger takes a decision within 450 ns. Since this time is shorter than the interval between beam collisions in the VEPP-4M accelerator (620 ns), the PT has zero dead time and is capable of processing every beam event.
A short operating time imposes rigid requirements on detectors, whose signals are used in the PT. Only the scintillation counters, as well as the endcap and cylindrical calorimeters, are the sources of signals for the PT.
New input shapers for scintillation counters and calorimeters were developed for this upgrade. New shaper units have frequency meters on each input for hardware monitoring and have the ability to catch the signals presence on each input and pass this information to the DAQ system.
The secondary trigger (ST) is located in the main equipment room of the KEDR detector. The ST receives the signals from all the KEDR subsystems, except for the aerogel Cherenkov counters. A block diagram of the secondary trigger is shown in Figure 3. The input trigger signals are called "arguments" and their combinations are called "masks".
During this upgrade, "Mask units" and a "Decision unit" were reimplemented as a single unit using modern hardware. The new version checks signals coincidence in parallel for all configured combinations, thus significantly reducing the decision time.

Synchronization system
The DAQ hardware synchronization is provided by the Synchronization and Commands Module (SCM), which has the following functionality: 1. Receives signals from hardware trigger; 2. Manages the events queue and does events enumeration; 3. Generates reference synchrosignals and distributes them to all devices of the DAQ system; 4. Generates commands for readout processors to read data from information boards.
SCM units can also be connected in chain to distribute signals from upper level SCMs. This synchronization module was developed from scratch based on ideas and technologies tested and approved earlier for the CMD-3 detector electronics [3,4].

Readout system
The readout system consists of readout processors (RP), DAQ network switches and a DAQ computing cluster. The readout processor is a completely new device developed based on the CMD-3 collaboration experience. It has KLUKVA form-factor for compatibility with existing IBs. It receives commands from the SCM, reads data from the information boards and sends them to the DAQ network ( Figure 4).
The readout processors and SCMs compose a synchronization and commands distribution tree ( Figure 5): one main SCM at the tree root, then one SCM for each subsystem and several readout processors attached to the subsystem's SCM. A tree topology here is important to have a single point of control during the work. Part of DAQ hardware forming one synchronization tree is called a partition. The synchronization and the commands distribution is provided by the C-Link interconnect developed at BINP.
On the other hand, the DAQ network is based on the Ethernet technology and has a mesh-like topology, which gives us the ability to flexibly balance and distribute event streams between DAQ computers, and also provides some redundancy for the data transfer.
The DAQ computing cluster is composed of several similar computers, running the same specially developed software, interacting with each other. It receives data from the DAQ network, process them according to hardware configuration and builds the events from the fragments transmitted by each RP.

Software
The data acquisition system software covers a full set of management and control tasks, such as hardware management and diagnostics, network diagnostics, management of the software services and the data acquisition itself. It logically binds all hardware components together.
The DAQ system has a central role for experiment to collect data, so it is designed to be as much available as possible including hardware redundancy. The software takes care of the hardware configuration, monitoring and failover.
The DAQ software is implemented as a set of independent services exchanging messages. The software has the following components ( Figure 6): • Distributed online database of the states and configuration; • Cluster coordination service; • Data processing subsystem; • Monitoring and data quality subsystems; • Command-line and application interfaces. The distributed online database is a key component for the whole system coordination. It provides consistent, distributed key-value store for the state and configuration of all software services and hardware devices. Our solution is based on etcd key-value store [5]. All kinds of configuration information are stored in this database -from relatively permanent service and hardware configuration to current running services states, exchange points, devices associations and other information that should be consistently exchanged between services. Also the database is able to notify services upon key modification.
The coordination service monitors the availability of the required services, starts and stops them on request. The data processing system does all the work related to data acquisition. This subsystem is composed of four services. The first one receives data packets from the readout processors and synchronization modules (data sources), and routes the data to the appropriate fragment collector service. The fragments collector does the first data filtration, the same as the current readout processor does in hardware, and builds an event fragment from it. The data is then sent to the event builder service, which builds the full detector event from the fragments and sends it to data storage. The fourth part of this subsystem is the partition manager service. The partition is a part of the DAQ hardware collecting data together (represent single synchronization tree). The partition manager sets the configuration of the hardware and software components, and starts the required processes via the coordination service.
The command-line and application interfaces are the entrance points for the DAQ system users and the user applications respectively. They provide full access to all the features of the system.

Conclusion
The data acquisition system upgrade planned for the KEDR detector is in progress. The system is scalable horizontally by design and include new features in hardware and software flexibility. The hardware part is almost complete. We are looking forward to have it in production in a year may give ideas to data acquisition systems for other experiments.