First implementation and results of the Analysis Grand Challenge with a fully Pythonic RDataFrame

¹ CERN, Esplanade des Particules 1, 1211 Geneva 23, Switzerland
² Princeton University, Princeton, NJ 08544, USA
³ Taras Shevchenko National University of Kyiv, Kyiv, Ukraine
⁴ Mannheim University of Applied Sciences, Mannheim, Germany

^* e-mail: vincenzo.eduardo.padulano@cern.ch
^** e-mail: enrico.guiraud@pm.me
^*** e-mail: andrii.falko@cern.ch
^**** e-mail: elena.gazzarrini@cern.ch
^† e-mail: enrique.garcia.garcia@cern.ch
^‡ e-mail: domenic.gosein@cern.ch

Published online: 6 May 2024

Abstract

The growing amount of data generated by the LHC requires a shift in how HEP analysis tasks are approached. Efforts to address this computational challenge have led to the rise of a middle-man software layer, a mixture of simple, effective APIs and fast execution engines underneath. Having common, open and reproducible analysis benchmarks proves beneficial in the development of these modern tools. One such benchmark is provided by the Analysis Grand Challenge (AGC), which represents a specification for realistic analysis pipelines. This contribution presents the first AGC implementation that leverages ROOT RDataFrame, a powerful, modern and scalable execution engine for the HENP use cases. The different steps of the benchmarks are written with a composable, flexible and fully Pythonic API. RDataFrame can then transparently run the computations on all the cores of a machine or on multiple nodes thanks to automatic dataset splitting and transparent workload distribution. The portability of this implementation is shown by running on various resources, from managed facilities to open cloud platforms for research, showing usage of interactive and distributed environments.