Automated Network Services for Exascale Data Movement

¹ George W. Downs Laboratory of Physics and Charles C. Lauritsen Laboratory of High Energy Physics 1200 E California Blvd Pasadena, California, USA 91125
² Department of Physics, UC San Diego, 9500 Gilman Drive, La Jolla, California, USA 92093
³ SDSC, UC San Diego, 9500 Gilman Drive, La Jolla, California, USA 92093
⁴ Energy Sciences Network, Lawrence Berkeley National Laboratory, 1 Cyclotron Rd, Berkeley, California, USA 94720
⁵ Fermi National Accelerator Laboratory, Kirk and Pine St, Batavia, Illinois, USA 60510
⁶ Department of Physics & Astronomy, Vanderbilt University, Nashville, Tennessee, USA 37235

^* e-mail: jbalcas@caltech.edu

Published online: 6 May 2024

Abstract

The Large Hadron Collider (LHC) experiments distribute data by leveraging a diverse array of National Research and Education Networks (NRENs), where experiment data management systems treat networks as a “blackbox” resource. After the High Luminosity upgrade, the Compact Muon Solenoid (CMS) experiment alone will produce roughly 0.5 exabytes of data per year. NREN Networks are a critical part of the success of CMS and other LHC experiments. However, during data movement, NRENs are unaware of data priorities, importance, or need for quality of service, and this poses a challenge for operators to coordinate the movement of data and have predictable data flows across multi-domain networks. The overarching goal of SENSE (The Software-defined network for End-to-end Networked Science at Exascale) is to enable National Labs and universities to request and provision end-to-end intelligent network services for their application workflows leveraging SDN (Software-Defined Networking) capabilities. This work aims to allow LHC Experiments and Rucio, the data management software used by CMS Experiment, to allocate and prioritize certain data transfers over the wide area network. In this paper, we will present the current progress of the integration of SENSE, Multi-domain end-to-end SDN Orchestration with QoS (Quality of Service) capabilities, with Rucio, the data management software used by CMS Experiment.