Adoption of ARC-CE and HTCondor at GridKa Tier 1

Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany

^* e-mail: max.fischer@kit.edu
^** e-mail: eileen.kuehn@kit.edu
^*** e-mail: matthias.schnepf@kit.edu
^**** e-mail: andreas.petzold@kit.edu
^† e-mail: andreas.heiss@kit.edu

Published online: 17 September 2019

Abstract

The GridKa Tier 1 data and computing center hosts a significant share of WLCG processing resources. Providing these resources to all major LHC and other VOs requires efficient, scalable and reliable cluster management. To satisfy this, GridKa has recently migrated its batch resources from CREAM-CE and PBS to ARC-CE and HTCondor. This contribution discusses the key highlights of the adoption of this middleware at the scale of a European Tier 1 center: As the largest WLCG Tier 1 using the ARC-CE plus HTCondor stack, GridKa is exemplary for migrating more than 20 000 cores over the time span of only a few weeks. Supporting multiple VOs, we have extensively studied the constraints and possibilities of scheduling jobs of vastly different requirements. We present a robust and maintainable optimization of resource utilization which still respects constraints desired by VOs. Furthermore, we explore the dynamic extension of our batch system, integrating cloud resources with a lightweight configuration mechanism.