Understanding the evolution of conditions data access through Frontier for the ATLAS Experiment

¹ Institute of Physics of the CAS, Na Slovance 1999/2, Prague, 18221, Czech Republic
² Sapienza Universita e INFN, Roma I
³ Science and Technology Facilities Council STFC
⁴ Centre National de la Recherche Scientifique
⁵ Instituto de Física Corpuscular (University of Valencia and CSIC)
⁶ University of Texas at Arlington, Department of Physics, 76019 Texas, Arlington, USA
⁷ Scientific Computing Division, Fermilab, Batavia, IL,USA

^* e-mail: Michal.Svatos@cern.ch
^** e-mail: Alessandro.de.Salvo@cern.ch
^*** e-mail: Alastair.Dewhurst@cern.ch
^**** e-mail: emmanouil.vamvakopoulos@cc.in2p3.fr
^† e-mail: julio.lozano.bahilo@cern.ch
^‡ e-mail: Nurcan.Ozturk@cern.ch
^§ e-mail: javier.sanchez@ific.uv.es
^¶ e-mail: dwd@fnal.gov

Published online: 17 September 2019

Abstract

The ATLAS Distributed Computing system uses the Frontier system to access the Conditions, Trigger, and Geometry database data stored in the Oracle Offline Database at CERN by means of the HTTP protocol. All ATLAS computing sites use Squid web proxies to cache the data, greatly reducing the load on the Frontier servers and the databases. One feature of the Frontier client is that in the event of failure, it retries with different services. While this allows transient errors and scheduled maintenance to happen transparently, it does open the system up to cascading failures if the load is high enough. Throughout LHC Run 2 there has been an ever increasing demand on the Frontier service. There have been multiple incidents where parts of the service failed due to high load. A significant improvement in the monitoring of the Frontier service wasrequired. The monitoring was needed to identify both problematic tasks, which could then be killed or throttled, and to identify failing site services as the consequence of a cascading failure is much higher. This presentation describes the implementation and features of the monitoring system.