Full Offline Reconstruction in Real Time with the LHCb Detector

This document describes the novel, unique in High Energy Physics, real-time alignment and calibration of the full LHCb detector. The LHCb experiment has been designed as a dedicated heavy flavour physics experiment focused on the reconstruction of c and b hadrons. The LHCb detector is a single-arm forward spectrometer, which measures proton-proton interactions at the LHC. The operational bunch crossing rate is several orders of magnitude above the current abilities of data recording and storage. Therefore, a trigger system has been implemented to reduce this rate to an acceptable value. The LHCb trigger system has been redesigned during the 2013–2015 long shutdown, achieving offline-quality alignment and calibration online. It also allows analyses to be performed entirely at the trigger level. In addition, having the best performing reconstruction in the trigger gives the possibility to fully use the particle identification selection criteria and greatly increases the efficiency, in particular for the selection of charm and strange hadron decays. This results in enhancing the physics programme of the LHCb experiment.


Introduction
The LHCb reconstruction and trigger are sensitive to the LHC running conditions. As the LHC increased its energy from the √ s = 7(8) TeV in 2011 (2012) to √ s = 13 TeV for Run II, (2015-2018), the strategy of data taking had to be redesigned and reoptimized. Additionally, the performance of a large computing cluster used for the software trigger and known as the Event Filter Farm (EFF) roughly doubled between 2012 and 2015, which significantly increased the time budget for the online reconstruction. This allowed an improved reconstruction, and coherent real-time detector alignment and calibration to be performed inside the trigger.
A schematic view of the LHCb detector is shown in Figure 1. The detector includes a highprecision tracking system consisting of a silicon-strip vertex detector (VELO) surrounding the pp interaction region, a large-area silicon-strip detector (TT) located upstream of a dipole magnet with a bending power of about 4 Tm, and three stations (T1-T3) of silicon-strip detectors (IT) and straw drift tubes (OT) placed downstream of the magnet. The tracking system provides a measurement of the momentum p of charged particles with a relative uncertainty that varies from 0.5% at low momentum to 1.0% at 200 GeV/c. The minimum distance of a track to a proton-proton collision, the impact parameter, is measured with a resolution of (15+29/p T ) μm, where p T is the component of the particle momentum transverse to the beam, in GeV/c. Different types of charged hadrons are distinguished using information from two ring-imaging Cherenkov detectors (RICH1, RICH2). Photons, electrons and hadrons are identified by a calorimeter system consisting of scintillating-pad (SPD) and preshower detectors (PS), an electromagnetic calorimeter (ECAL) and a hadronic calorimeter (HCAL). Muons are identified by a system composed of alternating layers of iron and multi-wire proportional chambers (M1-M5). The LHCb coordinate system is a right handed Cartesian system with the origin at the interaction point. The x-axis is oriented horizontally towards the outside of the LHC ring, the y-axis is pointing upwards with respect to the beam line and the z-axis is aligned with the beam direction. Figure 1. A schematic view of the LHCb detector. The Vertex Locator indicates the vertex detector. The labels RICH1 and RICH2 denote the Ring Imaging Cherenkov detectors, TT and T1-T3 are the tracking stations, the latter is composed of IT and OT detectors, M1-M5 indicates the muon chambers, SPD (PS) is the scintillator pad (preshower) detector. Finally ECAL and HCAL mark the electromagnetic and hadronic calorimeters, respectively.
The online event selection is performed by the trigger, which is composed of three stages: one hardware stage, known as Level0 (L0), and two software stages called High-Level-Trigger (HLT1 and HLT2). The HLT1 trigger performs a partial reconstruction of the candidates from many decays. At this stage all requirements are inclusive, which means that the selection is applied only to a subset of the final state particles. The HLT2 trigger contains both inclusive and exclusive algorithms which are more time-consuming but provide a more precise reconstruction. Exclusive algorithms are used to select a specific decay.

Strategy of data taking
The LHC running conditions have significantly changed between Run I and Run II. To meet the challenge due to increasing energy (from 7(8) TeV to 13 TeV) and the operational bunch crossing rate (from 15 to 30 MHz), the strategy of data taking has been redesigned during the 2013-2015 long shutdown. The new strategy results in an increase of the output rate from 5 kHz (Run I) to 12.5 kHz (Run II). Detailed information about the performance of the trigger in Run I can be found in [3].
The schematics of the DAQ strategies in Run I and II are shown in Figure 2. The first stage, knows as the low level trigger (L0), is common for both periods of data taking. The main purpose of the system is to quickly find general signatures of interesting decays and reduce the rate of accepted candidates to about 1 MHz, at which the full detector can be read out. This hardware trigger is based on information from the calorimeter system and muon chambers.
All candidates accepted by the L0 trigger are processed by the software trigger, which runs on the EFF. The farm currently consists of approximately 1700 nodes (800 of them were added for Run II) with 27000 physical cores. During Run I limited resources allowed only a simplified reconstruction to be performed inside the HLT. In particular, faster but less accurate primary vertex and track reconstructions were performed and only partial particle identification information was accessible. Moreover preliminary calibration and alignment were used. After data taking all selected events were processed further through an offline reconstruction together with updated calibration and alignment parameters, resulting in the best performing reconstruction. As a consequence significant differences between online and offline data quality were observed.
Due to operational issues in 2012, the LHC delivered stable beam conditions for approximately 30% of the scheduled time. In the LHCb trigger system about 20% of the L0 accepted events were temporarily buffered on the EFF nodes and processed in the gaps between stable beam periods. This deferred triggering method allowed LHCb to increase the data sample for physics analysis. During Run II the HLT1 and HLT2 stages are split. All events passing HLT1 are buffered on the EFF farm, which doubles the effective processing power. Thanks to this approach further HLT2 event processing can be done during gaps. In addition, this offers the needed time for the run-by-run evaluation of the detector calibration and alignment. Thus the HLT2 provides a full event reconstruction at optimal offline quality.

Real-time alignment and calibration
The detector alignment and calibration vary due to pressure and/or temperature changes, movements caused by operational conditions such as, for instance, switching the magnetic field direction or mechanical interventions. Thus updates of alignment and calibration constants are crucial to achieve optimal physics performance. The correct alignment of the VELO is needed for an accurate identification of primary and secondary vertices, which is visible in impact parameter, primary vertex (PV) and decay-time resolutions. In addition, more precise alignment of the entire tracking system directly implies better mass resolution, as shown in Figure 3. The Υ invariant mass resolution is significantly improved from 86 MeV/c 2 for the very preliminary constants up to 44.3 MeV/c 2 for the last alignment. Finally the precise calibration and alignment of RICH detectors implies better particle identification and allows to increase the purity of the sample. To ensure the best physics performance, the position and orientation of detector elements in the global reference frame must be known with an accuracy significantly better than the single hit resolution. The main alignment and calibration tasks have dedicated HLT1 trigger lines, which collect certain types of events needed by each task, for instance D 0 → Kπ, J/ψ → μμ, or tracks with specific properties. The selected events are saved to the disk, and reconstructed within the EFF. The alignment

LHCb Preliminary
LHCb Preliminary  and calibrations are performed at regular intervals, which can be at the beginning of the run, fill or less frequently. The resulting constants are updated only if they differ significantly from the currently used values.

Detector alignment
The tracking alignment is based on an iterative procedure [4,5], which minimizes the residuals of a Kalman filter and takes into account many effects such as multiple scattering, energy loss in the material, and magnetic field information. In addition, to avoid global distortions, tracks can be combined in vertices, allowing for the use of primary vertex and mass constraints. After the VELO alignment, the algorithm can align all tracking detectors simultaneously.

VELO alignment
The VELO is composed of left and right parts, therefore its two halves can be moved apart during the beam injection period for the safety of the detector and moved back together once the beams are stable. In nominal working conditions the sensitive parts of the VELO sensors are about 8 mm away from the beam. The VELO movement is achieved using stepper motors. The position is read from resolvers mounted on the motor axes with an accuracy better than 10 μm. The automated procedure closes the VELO symmetrically around the beam. In order not to degrade the impact parameter or decay-time resolutions, the VELO sensors need to be aligned with a precision of a few microns in x and y and a few tens of microns in z.
Since the VELO halves are closed for every fill, the precise alignment of this detector for each fill is crucial for the best physics performance. Figure 4 shows the stability of the VELO half alignment for the x and y transitions during 2015 data taking, where the horizontal dashed lines indicate the threshold within the alignment is not updated. Each point denotes the difference between new constants and parameters from the previous alignment. As expected, the constants are updated each few fills.

Tracking alignment
The rest of the LHCb tracking system is aligned relative to the VELO using tracks. This alignment benefits from the possibility to use mass constraints [4] in the evaluation of the alignment constants. This technique increases the precision of the method. Thus a sample of 100,000 D 0 → Kπ decays is used as input of this procedure. The alignment is performed for all tracking stations: IT, TT and OT. Typical alignment degrees of freedom are: displacements in x direction, rotations around the z-axis for the smallest structures (such as modules in the TT and boxes in the IT) as well as displacements in z for the layers. Detectors may be moved due to maintenance during accelerator technical stops, which occurs approximately once per two months. In addition, the magnetic field direction is switched about twice per month. It is observed that this causes a significant movement of the IT boxes and requires a few days for the position of these elements to stabilize. As a consequence, to account for any changes in detector conditions, the alignment is performed for every fill, while it is updated when the variations are significant (i.e. above threshold). Figure 5 shows the stability of IT detector during 2015 data taking using the online per fill alignment approach. As before, the horizontal lines denote the threshold between which the constants are not updated. As can be seen the variations are mostly smaller than the precision of the alignment, showing that infrequent updates are sufficient.

Muon stations alignment
Muon detection is performed in the five stations of muon chambers placed in the outermost layers of the LHCb spectrometer. The first station (M1) is located in front of the calorimeter system, while the remaining four stations (M2-M5) are placed behind the system (see Figure 1). The read-out pads of the muon chambers are less fine-grained than other components of tracking system, and as a consequence the spatial resolution of the muon system is less accurate. Therefore, the precise alignment of muon stations is less critical. Nevertheless, the information from the muon system is used in the L0 trigger, where the momentum of muons passing the detector is estimated from information provided by the first two stations. Simulation studies show that an alignment precision better than 1 mm is required to guarantee high trigger efficiency without any significant charge asymmetry. The alignment of the muon detector is required after each mechanical intervention that moves one or both halves of the muon chamber. If no intervention happens, the position is stable. Nevertheless, due to the importance of this alignment for the hardware trigger performance, the alignment of muon chambers is performed online for monitoring purposes. Figure 6 shows the stability of muon stations during 2015 data taking. The position of the muon chambers are stable, with a maximal variation of O(200 μm), much smaller than needed for the optimal performance. Thus the alignment is updated only after hardware interventions.

RICH alignment
Each of the RICH detectors is composed of spherical and flat mirrors. The light created by particles passing through the detectors is reflected off a primary mirror onto a secondary mirror, and then focused on the photon detection plane. Any misalignment of these mirrors causes the Cherenkov ring on the photodetector plane to be shifted with respect to its expected position. As a consequence, a projected track coordinate is not at the center of the ring. The deviation from the Cherenkov opening angle θ can be parametrized as a function of the azimuthal angle φ in the photodetector plane: where T x and T y are the components of the misalignment of the projected track coordinate with respect to the expected position, and θ is the Cherenkov angle calculated from the momentum of the track and the refractive index of the radiator. The alignment is determined for each mirror by fitting Equation (1) to the histogram of the Δθ(φ), as it is presented in Fig 7. The procedure is iterated until the variations are below a certain threshold. Both RICH detectors contain 110 mirrors to be aligned, which results into 1090 constants. The procedure is used in the monitoring system and runs every few fills, taking about one hour to converge.

Calibration
In addition to alignment, other detector constants have to be calibrated, in particular those of the OT, RICH detectors and calorimeters. A brief description of these calibrations is provided in the following subsections.

Outer Tracker calibration
The Outer Tracker is a gaseous straw-tube detector filled with a gas mixture. The straws have an inner diameter of 5 mm. For optimal spatial resolution, the straw layers are shifted by half a straw diameter inside one module. The time difference between a bunch crossing and the measurement of a signal in one of the straws t meas is composed of several components, including the global time offset t 0 . A global time offset occurs due to the synchronization between the collision time and the LHCb clock and is common for all modules. A shift of 0.5 ns leads to a tracking inefficiency about 0.25%. The parameter in question is evaluated by fitting the residuals of the reconstructed and expected drift time.
As it can be seen in Figure 8 the module time calibration is stable in time.

RICH calibration
The RICH automatic calibration is evaluated and updated in every run. The procedure implements two main calibrations: the RICH refractive index and the Hybrid Photon Detectors (HPDs) images. The refractive index of the gas radiators is sensitive to changes of temperature and pressure, as well as to the exact composition of the gas mixture. All of these components may change in time, thus two corrections parameters, one for each detector, have to be evaluated. They are determined by fitting the difference between the reconstructed and expected Cherenkov angle. In addition, the HPDs may be affected by changes in the magnetic and electrical fields. The calibration consists of 1940 parameters and is obtained by fitting the cleaned anode images using a Sobel filter. Here, the determination of constants is not iterative, and can be determined by fitting the relevant distribution on a single node.

Calorimeters calibration
The LHCb calorimeter system consists of a scintillating-pad detector (SPD), preshower (PS) detectors, and two main calorimeters: an electromagnetic (ECAL) and a hadronic (HCAL) calorimeter (see Figure 1). The system uses two types of the calorimeter gain calibrations: absolute and relative. The absolute calibrations are performed either each month (for ECAL) or in each technical stop (for HCAL), while the relative one runs online every fill. The absolute energy calibration of the ECAL uses the π 0 → γγ decay, which is found by combining a photon hitting the cell to be calibrated with another reconstructed photon. Then, a fit to the invariant mass m(γγ) is performed. The procedure is repeated iteratively. The improvement in the invariant mass resolution between miscalibrated and perfectly calibrated samples is a factor of about 2, from around 18 MeV/c 2 before to about 8 MeV/c 2 after the calibration. In order to apply the procedure to every cell in the ECAL, about 50 million events are required.
The absolute calibration of the HCAL uses two 137 Cs sources, one per detector side. A hydraulic system transports the source to all of the scintillator cells, then the response of the photomultiplier tubes (PMTs) is measured, which allows the HCAL parameters to be set.
Since the rate of the L0 trigger strongly depends on the calorimeter gain, it is crucial to keep it constant in time. The time period between absolute calibrations is too long with respect to the gradual gain reduction, thus relative methods of calibrations are used. The response to LED flashes is found to agree well with the response of particles passing the detector, so for monitoring purposes a dedicated LED system was built. The system detects aging of the PMTs, calculates the required adjustment and automatically updates the high voltage in each fill. In addition, to monitor the system it is possible to use the E/pc distribution of electrons (ECAL) and hadrons (HCAL). The variation of the gain can be determined from the variation of the occupancies ratio, where the occupancy is defined as the fraction of events in which the ADC output is above a threshold. The occupancies ratio is checked with respect to a reference sample and is proportional to the changes in gain. When a significant difference is detected, the high voltage is changed accordingly.

Reconstruction and performance
For 2015, the time budget of the software trigger was set to be 35 ms for HLT1 and 650 ms for HLT2. The optimal reconstruction had to match these time constraints, so many reconstruction algorithms were reoptimized. A big effort was made in terms of code optimization, especially performing faster approximations, applying vectorization or even revisiting entire algorithms. All of this has been done without losing any performance.
It is worth highlighting some examples. The reconstruction of the pp interaction point (primary vertex) has been reoptimized, resulting in an improved reconstruction efficiency and PV resolution as for the offline processing. The global efficiency increased by about 3% with a reduction of fake primary vertices of 70%. In particular, a significant gain is achieved for the PVs with a low number of associated tracks. Figure 9 shows the performance plots of the tracking system. The tracking

Turbo stream
The LHCb trigger for Run II contains a new streaming strategy (Turbo stream [6]) for many decays with a large production cross-section, which suffer due to limited trigger output rate.
In particular, in Run I the much smaller timing budget in the trigger system resulted in a low reconstruction efficiency of low-momentum charged particles, and the lack of full particle identification information in the HLT limited the performance for charm and strange hadron physics. Other issues were resolution differences between the online and the offline reconstruction, leading to difficulties in understanding efficiencies with a high degree of precision.
The new LHCb strategy resolves these problems and allows the selections to be performed with candidates reconstructed directly in the trigger, thus skipping the offline reconstruction. Improved reconstruction algorithms allowed to loosen the p T threshold from 1.2 GeV/c to 500 MeV/c in HLT1, significantly improving the reconstruction efficiency of low-momentum particles. The high-statistic modes can now be selected exclusively using offline-quality selections. Only the candidates selected by the trigger are saved, thus reducing the amount of recorded data by a factor of 10 with no loss in physics performance. The selected candidates are available for data analysis 24 hours after data taking.
This new Turbo stream approach was successfully used in the first LHCb publications on 2015 data [7,8]. In order to maximize the physics output of the experiment, it is planned to use a mixture of both inclusive and exclusive trigger selections in future, with particular use of the new streaming strategy.
are used. This new scheme allows the events to be fully reconstructed with a quality equivalent to the Run I offline performance in the trigger, and enables signals to be selected with purities previously achieved only offline. Using the new Turbo stream and performing the full offline reconstruction in the trigger greatly increases the efficiency with which charm and strange hadron decays can be selected, thus broadening the physics programme of the LHCb experiment.