Unlocking the compute continuum: Scaling out from cloud to HPC and HTC resources

¹ INFN Sezione di Perugia, Via A. Pascoli 23c, 06123 Perugia, Italy
² EGI Foundation, Amsterdam, Netherlands
³ Jülich Supercomputing Centre, Forschungszentrum Jülich GmbH, Jülich, Germany
⁴ Jozef Stefan Institute (JSI), Jamova cesta 39, 1000 Ljubljana, Slovenija
⁵ Institute of Information Science (IZUM), Prešernova ulica 17, 2000 Maribor, Slovenia
⁶ INFN Sezione di Padova, via Marzolo 8, 35137 Padova, Italy
⁷ INFN Sezione di Pisa, Largo B. Pontecorvo 3, 56127 Pisa, Italy

^* e-mail: ciangottini@infn.it

Published online: 7 October 2025

Abstract

In a geo-distributed computing infrastructure with heterogeneous resources (HPC and HTC and possibly cloud), a key to unlock an efficient and user-friendly access to the resources is being able to offload each specific task to the best suited location. One of the most critical problems involves the logistics of wide-area, multi-stage workflows that move back and forth between multiple resource providers. We envision a model where such a challenge can be addressed enabling a “transparent offloading” of containerized payloads using the Kubernetes API primitives creating a common cloud-native interface to access any number of external hardware machines and type of backends. Thus we created the interLink project, an open source extension to the concept of Virtual-Kubelet with a design that aims for a common abstraction over heterogeneous and distributed backends. interLink is developed by INFN in the context of interTwin, an EU funded project that aims to build a digital-twin platform (Digital Twin Engine) for sciences, and the ICSC National Research Center for High Performance Computing, Big Data and Quantum Computing in Italy. In this talk we first provide a comprehensive overview of the key features and the technical implementation. We showcase our major case studies, such as the scale-out of an analysis facility, and the distribution of ML training processes. We focus on the impacts of being able to seamlessly exploit world-class EuroHPC supercomputers with such a technology.