Experience with Rucio in the wider HEP community

¹ European Organization for Nuclear Research (CERN), Geneva, Switzerland
² Bergische Universität Wuppertal, Wuppertal, Germany
³ University of Oslo, P.b. 1048 Blindern, 0316 Oslo, Norway
⁴ California Institute of Technology, CA, USA
⁵ Georgia Institute of Technology, GE, USA
⁶ Istituto Nazionale di Fisica Nucleare (INFN), Torino, Italy
⁷ STFC Rutherford Appleton Laboratory (RAL), Didcot, England
⁸ Brookhaven National Laboratory, Upton, NY, USA
⁹ Fermi National Accelerator Laboratory, Batavia, IL, USA

Published online: 23 August 2021

Abstract

Managing the data of scientific projects is an increasingly complicated challenge, which was historically met by developing experiment-specific solutions. However, the ever-growing data rates and requirements of even small experiments make this approach very difficult, if not prohibitive. In recent years, the scientific data management system Rucio has evolved into a successful open-source project that is now being used by many scientific communities and organisations. Rucio is incorporating the contributions and expertise of many scientific projects and is offering common features useful to a diverse research community. This article describes the recent experiences in operating Rucio, as well as contributions to the project, by ATLAS, Belle II, CMS, ESCAPE, IGWN, LDMX, Folding@Home, and the UK’s Science and Technology Facilities Council (STFC).