Preparation of the Multi-Site Data Processing at the Vera C. Rubin Observatory

¹ Fermi National Accelerator Laboratory, Kirk Road and Pine Street, Batavia, IL 60510, USA
² SLAC National Accelerator Laboratory, 2575 Sand Hill Road, Menlo Park, CA 94025, USA
³ Brookhaven National Laboratory, Upton, NY 11973, USA
⁴ NCSA, University of Illinois at Urbana-Champaign, 1205 W. Clark St. Urbana, IL 61801, USA
⁵ CNRS, CC-IN2P3, 21 avenue Pierre de Coubertin, CS70202, F-69627, Villeurbanne cedex, France
⁶ Vera C. Rubin Observatory Project Office, 950 N. Cherry Ave., Tucson, AZ 85719, USA
⁷ Lancaster University, Lancaster, UK
⁸ Science and Technology Facilities Council, Rutherford Appleton Laboratory, Harwell, UK

^* Corresponding author e-mail: yangw@slac.stanford.edu

Published online: 7 October 2025

Abstract

The Vera C. Rubin Observatory’s Legacy Survey of Space and Time (LSST) Camera is scheduled to start taking data in the summer of 2025. The Data Release Production will run the LSST Science Pipe software at data facilities in the US, France and the UK. The LSST Science Pipeline consists of complex directed acyclic graphs (DAGs) of tasks. Rubin will use the Production and Distributed Analysis (PanDA) workflow and workload management system to orchestrate this complex workflow and the distribution of workloads to the data facilities. When run end-to-end by a team of data production staff, this processing (the Science Pipelines, distributed by the workflow and workload management system) is referred to as a 'campaign'. This paper describes the central services and data facility specific services that support this multi-site data process model, including the service deployment infrastructure, the workload and workflow system, the Campaign Management tools, and connection to Rubin Data Management. This paper will also mention the experience of processing the Rubin Commissioning Camera data. All these are part of the effort to scale up the processing capabilities for the expected very large data volume from the LSST Camera.