Trigger Rate Monitoring Tools at CMS

One of the major challenges for the Compact Muon Solenoid (CMS) experiment, is the task of reducing event rate from roughly 40 MHz down to a more manageable 1 kHz while keeping as many interesting physics events as possible. This is accomplished through the use of a Level-1 (L1) hardware based trigger as well as a software based High-Level Trigger (HLT). Monitoring and understanding the output rates of the L1 and HLT triggers is of key importance for determining the overall performance of the trigger system and is intimately tied to what type of data is being recorded for physics analyses. We present here a collection of tools used by CMS to monitor the L1 and HLT trigger rates. One of these tools is a script (run in the CMS control room) that gives valuable real-time feedback of trigger rates to the shift crew. Another useful tool is a plotting library, that is used for observing how trigger rates vary over a range of beam and detector conditions, in particular how the rates of individual triggers scale with event pile-up.


Introduction
The trigger system for the Compact Muon Solenoid (CMS) [1] is extremely important to data acquisition and is responsible for determining what data gets recorded for use in physics analysis. Roughly 900 separate L1+HLT trigger algorithms combine to filter the approximately 40 MHz rate of collisions down to around 100 kHz at L1 and then a further reduction to 1 kHz of data at HLT. Without an effective trigger system physics at CMS would not be possible.
As the LHC pushes to higher beam intensities, CMS has to be ready to respond to emergencies if the trigger rates go out of expected ranges, potentially overwhelming the DAQ or T0. Since the trigger algorithms make use of information from many parts of the various subdetectors, the trigger rate is very sensitive to all aspects of the detector and how they operate. Often it provides the first indication of problems in the detector, as well as provide a useful cross check for issues spotted by dedicated subdetector monitoring.
The Trigger Field Operation Group at CMS has developed a set of software tools to accomplish this task. Fits are made to the trigger rates in previous runs using linear and non-linear regression. These fits are then compared to the instantaneous trigger rate as data is being collected in order to spot (unexpected) deviations in the rates. In addition to this real-time component, the software tools provides a variety of feature that are used in offline analysis. is an example of a single object trigger (muon with p T > 5), while the right plot shows a jet trigger (sum over jet transverse momentum H T > 120). Single and double object triggers primarily have a linear or quadratic dependence on µ , while jet triggers have a much higher sensitivity to µ and are described best by exponential functions.

Fits
In order to determine if trigger rates are behaving as expected, a set of reference fits are first made using a collection of past runs with certified good detector performance. The fits are made as a function of the average number of collisions per LHC bunch crossing, µ (also known as average in-time pile-up). Before fitting, the raw trigger rate is first corrected for deadtime, Level-1 (L1) and High Level Trigger (HLT) prescales, and number of colliding bunches in the LHC. These corrections facilitates comparisons between runs with different conditions and allows a smooth function to be fit across multiple runs. For each trigger, several fit functions are attempted and the final function is selected based on χ 2 minimization, with a small bias given towards simpler (e.g. linear over quadratic) fits. Figure 1 shows comparisons of the attempted fit functions. The fitting and plot making is carried out using the ROOT data analysis software [2].
Ideally, trigger rates should depend linearly on instantaneous luminosity. However, some have nonlinear behavior, mainly due to pile-up effects. Candidates for multiple object triggers may be found in independent collisions, and triggers that rely on measuring calorimeter deposits (such as jets and jet sums) increase in rate as the number of contributions from independent collisions increases. We found that the behavior of most triggers is well-described by either linear, quadratic, or exponential (sinh) functions. Triggers with linear and quadratic behavior are primarily those that fire on single and double objects, respectively. The triggers with the highest sensitivity to pile-up are the jet sum triggers, and are described best by exponential functions. Examples of fits are shown in Figure 2.

Real-time Component
Using fits from previous runs, the trigger rates can be monitored during data-taking by comparing the instantaneous rate to the rate predicted by the fit. A list of approximately 20 L1 and HLT triggers is used for real-time online monitoring in the CMS control room. This list is selected such that all CMS subdetectors are represented. An automated script running continously during datataking checks current trigger rates against the prediction from the fit Examples of adjusted trigger rates versus µ . The points correspond to the per-lumisection rate, color-coded for different runs. The shaded pink band is the 3σ error band for the fit. From the top: examples of triggers whose behavior as a function of µ is best described by linear, quadratic and hyperbolic sine functions, respectively for each trigger in the list. Rate information is displayed in the trigger shifter terminal and is updated every two minutes. When the rate of a given trigger is at least five times higher than the uncertainty on the fit for multiple consecutive updates, the script prints a warning, and an email summarizing the problematic trigger is automatically sent to on-call experts. The script also activates audible alarms in the control room in the event that rates exceed more extreme, fixed thresholds; in this case, warnings are tailored to help diagnose problems with individual subsystems. For example, if the rate of a certain L1 e/gamma trigger is higher than 25 kHz, the warning "critical L1 e/gamma trigger rate: please check that ECAL and calorimetric triggers are behaving correctly" is given.

Summary Plots
In addition to the real-time component, the software produces plots that compare the trigger rate versus µ to the expected rate for each fill. These plots have been integrated into the central CMS Web Based Monitoring service (WBM) [3]; a dedicated page is linked from each Fill Report page on WBM, containing plots with rates for that fill (Figure 3). The page contains rate versus µ plots for all the triggers in the monitored trigger list. Plots for all HLT and L1 triggers, as well as stream and dataset plots, are available via links from this main page. Additional links directly to the plots of the individual triggers are available from the HLT and L1 summary pages. The plots are produced by a cron job run on a dedicated machine that updates the plots on an hourly basis (for the current fill only). Work is ongoing to develop a better implementation which would allow the plots to be updated in real time.

Offline Use
The software is also used for offline data certification. For a given list of runs and list of triggers, rate-versus-lumisection plots can be produced for each run and trigger (one lumisection equals 2 18 orbits of the LHC beam, or about 23 seconds). An overlay of the rate prediction from the reference fit enables easy comparison by offline validators, and a text summary is also produced highlighting the runs and lumisections where triggers deviated significantly from expectation. The software can also be used to track the relative changes in the trigger rates over extended periods of time, by plotting rates from many different LHC fills. This can highlight the impacts of detector calibration adjustments, changes made to particular trigger algorithims, or the changes in the rates after an LHC technical stop or machine development phase. The code has a modular organization, and can be extended to perform additional functions by other collaborators.

Summary
We have described a set of software tools currently in use at CMS to monitor trigger rates. This software is used for real-time monitoring of trigger rates by shifters and on-call experts, as well as offline for data certification and other studies. In particular, the trigger shifter script together with the plots on WBM have allowed fast identification and diagnosis of detector issues. The software has proven to be a critical tool to the successful operation of the trigger and to successful data-taking at CMS.