ATLAS Software Installation on Supercomputers

§ Compilation time 9 hours § Installations were validated with unit tests and ATLAS Simulation production jobs § Few tweaks on SummitDev were needed: § Replacement of __linux compiler macro with __linux__ § Replacement of linker option -Wl,-z,max-page-size=0x1000 as virtual addresses and file offsets for the 64-bit PowerPC processor family segments are congruent modulo 64 Kbytes (0x10000) § Removal of external checkpointing packages § Patch to ROOT Cling code generation mechanism (for different functions addressing modes on PowerPC) – reported to ROOT team § Fully automated procedure § Code upload via HTTP § Available for all external packages and Athena (ATLAS software) via git, wget § Distributed files systems such as CVMFS are not available on many supercomputers § Chain compilation ROOT-LCG-EXTERNALS-GAUDI-ATHENA § Athena (ATLAS software) simulation releases § Simulation is one of the most CPU-consuming task in HEP § 1/5 of full ATLAS software (700,000 code lines) Method Results


Introduction
ATLAS (A Toroidal LHC Apparatus) [1] is one of the largest collaborative efforts ever attempted in the physical sciences. Many important findings, such as Higgs discovery, were made and the physics research program is planned for the next 20 years. The new phase of LHC, named as High Luminosity LHC, aims to deliver an integrated luminosity of about 3000 fb -1 over 10 to 12 years of operation [2]. Offline computing will need to use supercomputers to cope with about 10 times larger data volumes. ATLAS software needs adapting for execution on modern PowerPC and HPC architectures which are incompatible with the currently dominant x86-64 architecture.

Usage of Supercomputers in ATLAS Offline Computing
ATLAS already uses resources of HPC centers across the world. HPC delivered 20% of ATLAS wall clock usage in June 2017 -May 2018 [3]. Mira at Argonne Leadership Computing Facility was the first large scale non-x86 HPC where an important workflow of event generation was ported and then its largest jobs routinely run, with millions of parallel threads [4]. Titan at Oak Ridge Leadership Computing Facility (OLCF) was the first large scale HPC integrated with the ATLAS Production System. It allowed to run ATLAS Simulation production jobs on used Titan resources ("backfill") collecting millions of core hours per month [5].

Software Porting at Summit -Next Generation Supercomputer
The new Summit supercomputer at OLCF [6] will deliver more than 5 times the computational performance of Titan. Unlike Titan's x86 CPUs, the Summit's IBM POWER9 CPUs does not allow the direct use of ATLAS binary software distributions which are currently available for x86 platforms only. Installation from sources is a viable alternative to binary distributions as it allows avoiding ineffective x86 chipset emulation and considering hardware information at build time (e.g. by tweaking compiler options). Summit runs Red Hat Enterprise Linux (RHEL) version 7 operating system that is similar to the Linux systems ATLAS uses for software development and distribution. It allows using the same build commands, tools, and shell scripts. However, installing from sources requires taking care of the complex external dependencies of ATLAS software.

Installation from Sources with ATLAS Comprehensive Software Compilation (ACSC) Procedure
The ACSC procedure provides automatic installation of ATLAS software, together with all external packages, from source code on machines running Linux operating system on various platforms, including x86 and PowerPC. The ACSC procedure was successfully applied for the installation from source of the Athena Simulation 21.0.34 software release on Titan HPC (compiler gcc 6.2) and Summitdev PowerPC (gcc 6.3) -a Summit early access system at OLCF, and on a x86 machine at CERN (gcc 6.2) for cross-check. Compilation times are shown in Table 1. Few modifications to software and compiler options were needed: • Replacement of __linux compiler macro with __linux__.
• Replacement of linker option -Wl,-z,max-page-size=0x1000 as virtual addresses and file offsets for the 64-bit PowerPC processor family segments are congruent modulo 64 Kbytes (0x10000). • Removal of checkpointing packages using architecture-specific assembly commands. • Patch to ROOT Cling code generation mechanism for different functions addressing modes on PowerPC (reported to ROOT team). Installations were successfully validated with various tests: ROOT functional tests, GAUDI example algorithms, Athena release unit tests, small sample simulation jobs.

Conclusions
The ACSC procedure for ATLAS software installation from source was developed. With this procedure, an Athena Simulation release was installed and tested on Summitdev PowerPC and Titan HPC without chipset emulation.
The author wishes to thank S. Panitkin and S. Snyder (Brookhaven National Laboratory), C. Leggett and V. Tsulaia (Lawrence Berkeley National Laboratory) for valuable advices. This material is based upon work supported by the U.S. Department of Energy, Office of Science, Office of High Energy Physics.