Abstracting container technologies and transfer mechanisms in the Scalable CyberInfrastructure for Artificial Intelligence and Likelihood Free Inference (SCAILFIN) project

High Performance Computing (HPC) facilities provide vast computational power and storage, but generally work on fixed environments designed to address the most common software needs locally, making it challenging for users to bring their own software. To overcome this issue, most HPC facilities have added support for HPC friendly container technologies such as Shifter, Singularity, or Charliecloud. These different container technologies are all compatible with the more popular Docker containers, however the implementation and use of said containers is different for each HPC friendly container technology. These usage differences can make it difficult for an end user to easily submit and utilize different HPC sites without making adjustments to their workflows and software. This issue is exacerbated when attempting to utilize workflow management software between different sites with differing container technologies.The SCAILFIN project aims to develop and deploy artificial intelligence (AI) and likelihood-free inference (LFI) techniques and software using scalable cyberinfrastructure (CI) that span multiple sites. The project has extended the CERN-based REANA framework, a platform designed to enable analysis reusability, and reproducibility while supporting different workflow engine languages, in order to support submission to different HPC facilities. The work presented here focuses on the development of an abstraction layer that allows the support of different container technologies and different transfer protocols for files and directories between the HPC facility and the REANA cluster edge service from the user’s workflow application.


Introduction
The REANA framework [1], used by the SCAILFIN project [2] to create analysis workflows that are easily reproducible, works on top of Kubernetes in order to orchestrate all service components and the scheduling of workers to run the payloads. This assumes support of docker and a shared file system between services and workers (see Figure 1). While the REANA team is working on supporting different submission backends in the framework (HT-Condor [3] and SLURM [4] at present), these are currently focused on working with CERN resources [5]. The SCAILFIN project on the other hand, focuses on integrating REANA with different HPC environments.
In order to do this in SCAILFIN, we have created a submission backend integrated with VC3 [6], leveraging the authentication to the HPC submit nodes and proper translation of jobs from HTCondor to the different HPC supported batch systems, as seen in Figure 2.
A description of the mechanisms used to support different HPC container technologies and handle the transfer of files between the workers and the REANA cluster edge service by creating a user-level distributed file system will be presented here.

Detecting and supporting different container technologies
Users can interact with the REANA cluster through a python-based client package. This client allows the user to create workflows, upload files to the workflow area and define the docker container images needed to run each step in the workflow (see Figure 3). Docker support is normally not provided by HPC centers, mostly due to the root daemon privilege mode in docker, which imposes a security risk on multi-user facilities such as these. Instead, HPC friendly container technology solutions are frequently provided, such as Singularity [7], Shifter [8] or Charlie Cloud [9], which are compatible with docker images, but run the containers in unprivileged mode.
In order to support different container technologies, the job controller submission backend constructs the job submission object and passes the docker image as a job argument(see Figure 4). Subsequently, a job wrapper running in the worker node detects the container technology available (Singularity and Shifter are supported at present) and launches the container image with the proper parameters, as shown in Figure 5.  This allows workflows to run on multiple HPC facilities supporting container technologies, without involving the user in providing the specific parameters per container technology for launching the container images.

Data transfer management
Input files uploaded by the users through the client are stored in a file system at the edge service by the server. This file system is shared between the REANA service components and the workers when Kubernetes is also used as the scheduler for jobs. With HPC facilities, jobs are submitted using their batch systems instead. Even though the REANA server can upload files to this file system area, HPC workers do not have access to them in this case.  To address this issue, SCAILFIN uses chirp [10], a user-level file system for collaboration across distributed systems such as clusters, clouds and grids without any special privileges. Chirp is integrated with HTCondor; a chirp server is automatically launched when the classad +WantIOProxy=True is added to job submit object. This server can then be used to share files between the VC3-headnode and the workers. This mechanism also takes care of the authentication using cookies created by HTCondor itself when this option is enabled.
In order to access this file system from the worker nodes, parrot [11] is installed using the vc3-builder [12] on the machine. Parrot is used to interact with the chirp server created by HTCondor; while HTCondor has its own client, parrot allows recursive access of directories on the fly, as opposed to the HTCondor chirp client.
As shown in Figure 6, parrot transfers all input files and directories for the job inside the job scratch area. The directory is bound inside the container, using the same absolute path kubernetes workers would normally see these files at in the shared file system, in order to guarantee compatibility between both submission backends.
Finally, Figure 7 summarizes the main differences compared to the original infrastructure in Figure 3, using the chirp server for the workers to connect to the file system the REANA server writes to, as well as detecting and using Singularity or Shifter accordingly, to provide the proper environment for the job.

Conclusions
The SCAILFIN project has implemented mechanisms to automatically detect container technologies supported by HPC centers on the fly and use them in such a way that no modifications in the workflow definitions are necessary. Files are also shared between the HPC facility workers and the REANA cluster edge service in a seamless way by using a user-level file system integrated with HTCondor, which can be used independently of the actual HPC facility supported batch scheduler.