An Information Aggregation and Analytics System for ATLAS Frontier

¹ Université Paris-Saclay, CEA/Saclay IRFU, 91191 Gif-sur-Yvette, France
² University of Texas at Arlington, Department of Physics, Arlington Texas 76019, USA
³ Ecole Nationale Supérieure d’Informatique, Alger Oued Smar 16309, Algeria
⁴ University of Valencia, Instituto de Física Corpuscular, Parque Científico, E-46980 Paterna, Spain
⁵ University of Oxford, Denys Wilkinson Bldg, Keble Rd, Oxford OX1 3RH, UK
⁶ University of Chicago, Enrico Fermi Institute, 933 East 56th Street, Chicago IL 60637, USA

^* e-mail: Andrea.Formica@cea.fr
^** e-mail: Nurcan.Ozturk@cern.ch
^*** e-mail: em_si_amer@esi.dz
^**** e-mail: Julio.Lozano.Bahilo@cern.ch
^† e-mail: Elizabeth.Gallas@physics.ox.ac.uk
^‡ e-mail: Ilija.Vukotic@cern.ch
^§ Copyright 2020 CERN for the benefit of the ATLAS Collaboration. CC-BY-4.0 license.

Published online: 16 November 2020

Abstract

ATLAS event processing requires access to centralized database systems where information about calibrations, detector status and data-taking conditions are stored. This processing is done on more than 150 computing sites on a world-wide computing grid which are able to access the database using the Squid-Frontier system. Some processing workflows have been found which overload the Frontier system due to the Conditions data model currently in use, specifically because some of the Conditions data requests have been found to have a low caching efficiency. The underlying cause is that non-identical requests as far as the caching are actually retrieving a much smaller number of unique payloads. While ATLAS is undertaking an adiabatic transition during the LHC Long Shutdown 2 and Run 3 from the current COOL Conditions data model to a new data model called CREST for Run 4, it is important to identify the problematic Conditions queries with low caching efficiency and work with the detector subsystems to improve the storage of such data within the current data model. For this purpose ATLAS put together an information aggregation and analytics system. The system is based on aggregated data from the Squid-Frontier logs using the Elasticsearch technology. This paper§ describes the components of this analytics system from the server based on Flask/Celery application to the user interface and how we use Spark SQL functionalities to filter data for making plots, storing the caching efficiency results into a Elasticsearch database and finally deploying the package via a Docker container.