| Issue |
EPJ Web Conf.
Volume 337, 2025
27th International Conference on Computing in High Energy and Nuclear Physics (CHEP 2024)
|
|
|---|---|---|
| Article Number | 01218 | |
| Number of page(s) | 7 | |
| DOI | https://doi.org/10.1051/epjconf/202533701218 | |
| Published online | 07 October 2025 | |
https://doi.org/10.1051/epjconf/202533701218
Orchestrated columnar-based analysis with columnflow
Institut für Experimentalphysik, Universität Hamburg
* e-mail: mathis.frahm@uni-hamburg.de
Published online: 7 October 2025
Columnflow is a tool for columnar-based data analysis. It is written in Python, experiment-agnostic in its core, and supports any flat n-tuple format, such as ROOT-based TTrees or Parquet files. Leveraging the vast Python ecosystem, vectorization and convenient representation of event content can be achieved through NumPy, AwkwardArray and other libraries. Based on the Luigi Analysis Workflow (law) package, columnflow provides full analysis automation over arbitrary, distributed computing resources. This approach features persistent, intermediate outputs for purposes of reusing of previously computed results, debugging, and exchange with collaborators. Job submission to various batch systems is natively supported. Remote files can be seamlessly accessed via various protocols using the Grid File Access Library (GFAL2). In addition, a sandboxing mechanism can encapsulate the execution of parts of a workflow in dedicated environments, supporting subshells, Python virtual environments, and containers. This contribution introduces the key components of columnflow and highlights the benefits of a fully automated workflow for complex and large-scale HEP analyses.
© The Authors, published by EDP Sciences, 2025
This is an Open Access article distributed under the terms of the Creative Commons Attribution License 4.0, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Current usage metrics show cumulative count of Article Views (full-text article views including HTML views, PDF and ePub downloads, according to the available data) and Abstracts Views on Vision4Press platform.
Data correspond to usage on the plateform after 2015. The current usage metrics is available 48-96 hours after online publication and is updated daily on week days.
Initial download of the metrics may take a while.