Data quality monitors of vertex detectors at the start of the Belle II experiment

The Belle II experiment features a substantial upgrade of the Belle detector and will operate at the SuperKEKB energy-asymmetric e+e− collider at KEK in Tsukuba, Japan. The accelerator completed its first phase of commissioning in 2016, and the Belle II detector saw its first electron-positron collisions in April 2018. Belle II features a newly designed silicon vertex detector based on double-sided strip layers and DEPFET pixel layers. A subset of 2 EPJ Web of Conferences 245, 01035 (2020) CHEP 2019 https://doi.org/10.1051/epjconf/202024501035

Abstract. The Belle II experiment features a substantial upgrade of the Belle detector and will operate at the SuperKEKB energy-asymmetric e + e − collider at KEK in Tsukuba, Japan. The accelerator completed its first phase of commissioning in 2016, and the Belle II detector saw its first electron-positron collisions in April 2018. Belle II features a newly designed silicon vertex detector based on double-sided strip layers and DEPFET pixel layers. A subset of the vertex detector was operated in 2018 to determine background conditions (Phase 2 operation). The collaboration completed full detector installation in January 2019, and the experiment started full data taking. This paper will report on the final arrangement of the silicon vertex detector part of Belle II with a focus on online monitoring of detector conditions and data quality, on the design and use of diagnostic and reference plots, and on integration with the software framework of Belle II. Data quality monitoring plots will be discussed with a focus on simulation and acquired cosmic and collision data.

Introduction
The Belle II experiment uses the asymmetric (4 GeV e + , 7 GeV e − ) SuperKEKB collider at the High Energy Accelerator Research Organization (KEK) in Tsukuba, Japan. The design peak luminosity is 8 × 10 35 cm −2 s −1 , about 40 times larger than in the previous Belle experiment, aiming at an integrated luminosity of 50 ab −1 [2].
The requirements on the new Belle II vertex detector were high: it had to achieve excellent vertexing and tracking capabilities at higher event rates, beam backgrounds, considerable noise occupancy, and radiation damage.
The vertex detector, VXD ( Fig. 1), is one of the critical determinants of the Belle II physics performance. The six-layer detector contains four outer layers of 300 -320 µm thick double-sided strip sensors (SVD); the two innermost layers are 75 µm thick depleted p-channel FET (DEPFET) sensors (PXD). The full installation of the vertex detector was completed in January 2019, and the detector took its first physics collision data in March 2019.

The Silicon Vertex Detector of Belle II
The strip layers of the Belle II VXD use double-sided silicon microstrip sensors read out by APV25 front-end chips, initially developed for the readout system of the CMS experiment at CERN [1]. With 50 ns shaping time and six or three samples per hit, SVD allows precise hit timing at the level of nanoseconds, efficient noise reduction, and separation of signal from background. The sensors have 768 strips on the p-side, and 512 (768 in layer 3) strips on the n-side. A charge injection feature provides strip-by-strip charge response calibration. The SVD comprises 172 detectors arranged in 4 layers with radii 39 mm, 80 mm, 104 mm, and 135 mm, with a total length of 600 mm and a total sensor area of ∼1.2 m 2 [2].
The design of Belle II VXD foresaw two DEPFET pixel layers. The DEPFET structure [2] provides in-pixel amplification of detector signals to equivalent current. Layer radii are 14 and 22 mm, and the length is 120 mm. All 14 ladders of Layer 1 were installed, but, due to a series of assembly accidents, only two ladders of Layer 2 were installed to make up for a faulty underlying sensor in Layer 1.

VXD data quality monitoring
The main components and flow of data in the VXD data acquisition system are shown schematically in Fig. 2. Event builder 1 collects detector data from all subdetectors except the PXD and sends them for processing to the High-Level Trigger (HLT) cluster, which does fast reconstruction to identify interesting events. The HLT produces some data quality monitoring  (DQM) data that are then sent to the DQM server. Event Builder 2 adds PXD data to the data of other detectors and sends a subset to the ExpressReco cluster for express reconstruction (up to vertexing). The primary purpose of express reconstruction is online data quality monitoring to check that detectors, slow-and run control work correctly and produce useful data -it produces other DQM data and sends them to the DQM server. The overall bandwidth of the data quality monitoring stream is up to 20-30 MB/s. Pixels (Fig. 2) are read and signals are processed with the Drain Current Digitizer (DCD) and Data Handling Processor (DHP) chips mounted on the sensor ladder. These chips prepare data for the first compression step using pedestal suppression and for transfer to the Data handling Hub (DHH) [3]. An important side-task of the HLTs is fast online track finding and selection of regionsof-interest (ROIs) on PXD sensors based on track projections from the SVD. This mechanism reduces the amount of raw data produced by the PXD by an order of magnitude.
DHH sends data to the Belle II data chain over Online Selection Nodes (ONSEN) [4]. Via an independent network, data are also sent to a separate system for local calibration of PXD sensors and low-level data quality control. ONSEN combines data for export to Event Builder 2. It can buffer up to 2.5 seconds of data flow and reduce the raw data volume by a factor of 30.
Strip detector signals are transferred over Junction Boards to Flash Analog-to-Digital Converter modules (FADC) for zero suppression and data formatting. Local SVD data quality monitoring and calibration generators use data from this phase of data processing. The signal is then processed on the COmmon Pipelined Platform for Electronics Readout (COPPER [5]) boards of the central Belle II DAQ for the HLT and Event Builder 1. Apart from that, data are also sent to the PXD ROI finding and data reduction blocks. PXD ROIs are built using both the HLT and the data concentrator (DATCON) modules that find track candidates and then ONSEN extracts PXD data from the ROIs. [4].

DQM data production
Data monitors provide a condensed overview of the essential aspects of detector performance. They allow fast discovery of detector problems and identification of their sources so that we can prevent significant data losses and achieve uniformly high data quality. More than 7500 histograms are updated every 2 minutes. Separate sets of DQM histograms are produced by HLT and by Express Reco, cf. Fig. 2.
At Belle II, ExpressReco runs full basf2 [6] reconstruction (up to vertexing) on a sampled data flow from all subdetectors, including the PXD. The sampling rate is adjusted based on the immediate trigger rate. Histogram production for data quality monitoring is the primary purpose of ExpressReco. These and HLT DQM histograms are collected by the DQM server and uploaded to the Web for Control Room or remote presentation. Also, at the end of each run, histograms are saved to disk. DQM Control computer manages DQM data processing and communicates with the Run Control system.
ExpressReco monitors confirm data quality and identify runs usable for further processing. They provide a finer footprint of the data allowing to reveal any irregularities in the data acquisition chain and detector settings.
The collection of data for DQM uses the basf2 framework [6] running on a set of isolated computers (HLT and ExpressReco) with conservative update plans to ensure the stability of production-critical processing. Any outside connections to databases or public networks are forbidden; only local services are allowed.

Scope of DQM monitoring
The VXD data quality monitoring system provides several hundred plots of various quantities. Part of them is intended for expert shifters to diagnose sources of DAQ problems. Automatic comparisons with reference data are provided together with color-coded plot canvases indicating serious deviations from reference (green plot frame meaning no problem) so that problems can be spotted quickly among a large number of plots.
A more restricted set of plots is selected for monitoring by Control Room (CR) shifters so that they can spot problems quickly, even without expert knowledge about individual subdetectors. In case of problems or unclear situations, the CR shifter can inform expert detector shifters by a dedicated chat channel, e-mail, or phone. Any problems are logged, and shifters can subsequently label runs as OK or problematic.
Shifters can also consult run-by-run comparisons of selected sets of variables. Fig. 4 shows the PXD efficiency over runs taken between 15 November and 12 December 2019; a hovering window shows details for the run at the mouse cursor (in this case, run 5695). Fig. 5 shows a similar plot for the SVD, efficiency in u and v strips.

Optimizing the DQM: cleaner plots, refresh rates and reference plots
For production monitoring, we aim at plots requiring minimum visual processing by the shifter. Where available, plots also display reference data. The software automatically compares actual data with the reference, and canvas color is changed if a mismatch is found. Special efforts go into the development of DQM modes, such as cosmic acquisition, machine adjustment, detector tuning, or beam collisions with different conditions. For this, the DQM will get a database interface to allow straightforward and automated re-configuration. The PXD and SVD groups invest considerable effort into understanding machine background radiation at beam conditions. These efforts can only partially rely on simulations and require, instead, handy data summaries that are available over long times and different conditions. DQM data seem to provide a useful tool for these observational studies. DQM data serve as data summaries also in the study of detector radiation damage: Slow changes in DQM data can be found in the historical record of DQM data and calibrations; also, the update frequency of reference data are an indication of changes in the detectors. All DQM plots are recorded on a dedicated web site with a refresh rate of ∼2 minutes. The web site provides access to the full history of all DQM plots by run and category, and also allows access to underlying raw datasets.

Conclusions
Belle II has been operating successfully in full detector setup since April 2019. The DQM subsystem has performed to expectation: online data quality monitors work and are being optimized for the best usability. The DQM system follows incoming requests and the development of the DAQ system. The lists of monitored values for pixel and strip detectors evolve according to user requests. Updating of DQM software and reference plots is conservative due to requirements on production system stability but is not frozen. The PXD and SVD detector groups expect that automation of the data acquisition systems and improvements in data quality monitoring will eventually relieve the need for online expert shifts and will allow delegation of the task to central shifters.