Exploring the virtues of XRootD5: Declarative API

Across the years, being the backbone of numerous data management solutions used within the WLCG collaboration, the XRootD framework and protocol became one of the most important building blocks for storage solutions in the High Energy Physics (HEP) community. The latest big milestone for the project, release 5, introduced multitude of architectural improvements and functional enhancements, including the new client side declarative API, which is the main focus of this study. In this contribution, we give an overview of the new client API and we discuss its motivation and its positive impact on overall software quality (coupling, cohesion), readability and composability.


Introduction
The XRootD [1] project aims at providing low latency and scalable data access for large scientific data sets and is based on a scalable, plug-in centric architecture and a communication protocol. It has been designed with particular emphasis for geographically distributed, file-based repositories. The software suite allows the deployment of federated data clusters and provides important features like access control and WAN data distribution. For almost 10 years now, the XRootD framework has been very successful at facilitating data management of LHC experiments and grew into one of the most important storage technologies in the High Energy Physics (HEP) community. It comes with no surprise that XRootD development is largely driven by the use cases coming from the WLCG project, as it is the backbone of numerous software defined storage solutions (like EOS [2] and DPM [3]) used to accommodate the vast amount of data registered by the LHC experiments at CERN, most notably Atlas [4], CMS [5], LHCb [6] and Alice [7]. One of the key components of the XRootD framework is the C++ client, which is fundamental not only to the command line utilities like xrdcp and xrdfs, but also to XCache (a XRootD file-based caching proxy) [8] [9], XrootdFS (a FUSE based mountable file system) [10] and EOS (the storage service of choice at CERN). In addition, the XRootD client is employed to provide remote data access in many physics analysis frameworks like ROOT [11] and in data movers like FTS [12]. With latest major release (5.0.0) the XRootD framework brought multitude of architectural improvements and functional enhancements, including the new client side declarative API, which is the main focus of this study. This paper first outlines the motivation for introducing the declarative API into the XRootD client library, and discusses the property of composability. As case study we consider a ZIP archive metadata parser. Subsequently, the syntax and fundamental concepts required to understand and use the new API are explained. Finally, we conclude the paper with results of applying some common software metrics (cohesion, coupling, cyclomatic complexity) to the software developed using the new declarative API and we provide a short summary.

The Object Oriented APIs
Before XRootD5 has been released, there were just two types of APIs available in the client, an Object Oriented (OO) synchronous API (see List. 1) and an OO asynchronous API (see List. 2). The OO synchronous API is easy to use, the code readability is very good, but there is virtually no composability [13]. In other words, it is not possible to chain the remote access operations together. Let us consider now an example where the user would like to read in parallel from multiple files. A major limitation of the synchronous API is that concurrent access could be achieved only by adding more threads to take advantage of core parallelization, which will increase context management overhead. The OO asynchronous API allows to chain the remote access operations however code readability is poor (the flow control is not clearly expressed). Moreover, it requires significant amount of boilerplate code, because each of the asynchronous operations requires a special custom callback object.

Use cases
The development of the XRootD client declarative API has been driven by two major use cases: the client erasure coding (EC) [14] plugin for the Alice O2 project [15] and a ZIP archive metadata parser. The former requires extensive parallel remote access to multiple devices for every implemented operation. The later needs to issue consecutive reads in order to parse the ZIP archive metadata. In order to illustrate the difficulties of using the OO asynchronous API, we will consider the logical open operation of an erasure coded file. In our implementation, the first component of a logical EC open are parallel open requests to n data and p parity stripes (in total n+p requests), of which n have to be successful. The second component, is an open, followed by a read, and then followed by a close of a metadata file. Any error happening during metadata retrieval can be recovered at a different location holding redundant copy of the metadata file. To implement this logic using the asynchronous OO API, five custom callbacks have to be provided in order to ensure chaining of the asynchronous operations, which is a significant amount of boilerplate code (see List. 3). Moreover, the program is difficult to understand as significant fraction of the logic is hidden in the callback objects. For instance, other than for the comment, it is not clear from inspecting the EcOpen function that the metadata are being actually read. After analysing the extra code that had to be written in order to chain the OO operations, we have extracted the common patterns (e.g. callback classes), applied significant amount of template meta-programming and flavored it with a pinch of operator overloading. As a result we got a new declarative API that is in line with the modern C++ programming practices (ranges v3 inspired, support for lambdas and std::futures), offers greater code readability and genuine composability.

The declarative API
Let's consider what would be a good model for asynchronous remote I/O programming. As in case of any other software engineering problem, we would like to be able to decompose larger tasks into smaller ones. For instance, we would like to be able to decompose a file update operation into an open, a write and a close. Afterwards, we can finally program each of the primitive I/O operations. However, there is one more critical step, we have to be able to compose those smaller operations back together into the original, bigger problem. The most important thing is that those operations have to be trivially composable, if the programmer has to know the internals of an operation implementation in order to be able to compose it with another one then all is lost [13]. In addition, it is critical that the operations are lazy evaluated, meaning one can first declare the operations, compose them into a pipeline and only then execute the whole pipeline. Finally, the remote I/O operations have to be associative in order to be really composable (meaning that object updt1 and updt2 in listing 4 are equivalent). auto u p d t 2 = Open | w r t _ c l s ;

Rules of pipelining
Now, let us dive into the new XRootD declarative API and let us start the tour with the most important topic, which is composability. In XRootD client all the I/O operations can be composed with the | operator. We will call all the basic operations (remote counterparts of POSIX open, read, write, etc.) as primitive and all the results of composing two or more operations as composed. Another useful concept that we will need is the Pipeline utility class, it is a general-purpose polimorphic I/O operation pipeline wrapper. In other words, instances of Pipeline can store any I/O operation (primitive or composed). The last thing we need to know to create our first pipeline (see List. 5) is that each I/O operation needs a context (or a handle if you will), in most cases this will be the File or FileSystem object. Pipelines obey certain rules. First of all, as we mentioned before, the operations within a pipeline are associative. Secondly, defining a pipeline does not trigger it (in a sense, it is lazy evaluated). Thirdly, once executed, operations in the pipeline are performed strictly from left to right (in our example it is first the Open, then the Read, and then the Close). Finally, if during the execution an operation fails the pipeline stops, and subsequent operations are ignored.

Executing a pipeline
A pipeline can be executed either using the Async or WaitFor functions. The Pipeline object needs to transfer the ownership of the underlying operations pipeline to the executing routine. The Async function (as the name suggests) triggers asynchronous execution of the pipeline and returns a std::future to the final status that is the outcome of carrying out the I/O operations in the given pipeline (see List. 6).

Handlers
Any operation can (but does not have to) be assigned with a handler using the » operator. In a simple case the result of an operation can be directed to an instance of std::future (see List. 8).  Finally, an I/O operation can be handled also with a packaged_task and for backwards compatibility also with an instance of ResponseHandler (an XRootD4 style handler).

Forwarding values between handlers and operations
In order to facilitate forwarding values between handlers and operations the Fwd class has been provided. Fwd is an argument wrapper accepted by any I/O operation. Initially, a Fwd instance contains no value and can be assigned with one for example in an operation handler. To illustrate the usefulness of forwarding values, let us consider following example. Let us assume that there is a file (not to big) of unknown size and that there is a need to read the file with a single read request. In order to implement this scenario, we will forward two values from the handler of an Open operation that has the stat information of the given file as arguments to the Read operation (see List. 10).

Control directives
There are four control directives that allow to alter pipeline execution: Pipeline::Stop, Pipeline::Repeat, Pipeline::Ignore and Pipeline::Replace. All of those directives must be called from within an operation handler body. The Pipeline::Stop forces the pipeline to be stopped, the user may optionally provide the final status as an argument (defaults to success). The Pipeline::Repeat forces the current operation to be repeated (e.g. repeat read operation until the end-of-file, see List. 11). The Pipeline::Ignore makes the pipeline ignore an operation failure and resumes execution at the operation next in turn (e.g. ignore a read error and proceed to Close, see List. 11). The Pipeline:Replace is overloaded and can be used either to replace the current operation with a different one or to replace the whole pipeline. This facility allows to define composed I/O operations and whole pipelines in a recursive way (e.g. recursively try opening redundant file replicas, see List. 12).
Listing 12. Recursively try replicas.  routine has two implementations: first that uses the old XRootD4 OO asynchronous API (Zi-pArchiveReader) and second based on the new XRootD5 declarative API (ZipArchive) [17]. In order to compare the two implementations we applied some standard software metrics to both software routines.

Cyclomatic complexity
Cyclomatic complexity is a metric that allows to determine the number of linearly independent paths within a section of source code. Two paths are considered as linearly independent if and only if an edge exists that belongs to only one of those paths. [18] Cyclometric complexity M is resolved based on a control flow graph of the given section of source code. The cyclometric complexity of the ZIP archive open routine implemented using the XRootD4 OO API and XRootD5 declarative API have been determined respectively to M=14 based on the control flow graph shown in Figure 1 and to M=8 based on the control flow graph shown in Figure 2. The significantly lower cyclometric complexity of the implementation that employed the new declarative API implies easier code maintainability and is due to greatly simplified error handling. Finally, lower cyclometric complexity means there are fewer execution paths that need to be accounted for in the test suite.

Cohesion
Cohesion is an ordinal software metric that reflects the degree to which a software module or a class is unified around a central concept it serves. It is a measure of how strongly the encapsulated data are related and of how much the functionalities embedded in a single software module have in common. High cohesion is a desirable property because it is associated with several important characteristics of software like robustness, reusability and understandability. [19] [20] The fundamental goal of the ZIP archive class is to extract the information about data layout in the archive from the metadata and to serve the data itself to the end user. In the implementation employing the XRootD4 OO API, the ZIP metadata parsing functionality, the custom asynchronous callbacks providing operation chaining and core functionality of the class are strongly interleaved. One can easily notice a recurring pattern: an asynchronous operation is issued, then its result is interpreted as ZIP metadata, and then subsequently the custom completion handler chains another asynchronous operation. As a result, the XRootD4 OO API based implementation can be classified as having sequential cohesion. On the other hand, in the implementation based on XRootD5 declarative API it has been possible to extract ZIP metadata parsing from the ZIP archive class into a separate module providing this functionality. Moreover, there is no need for custom completion handlers that provide operation chaining as this functionality is by default available in the new API. As a result, the ZIP archive class is focused only at its core role and hence the implementation can be classified as having functional cohesion.
It is worth noticing that higher cohesion of the implementation employing the declarative API results in better reusability (extracted ZIP parsing functionality in separate module), more concise codebase (37% shorter compared to its counterpart) and enhanced code readability and maintainability (its counterpart needed 5 custom completion handler classes).

Coupling
Coupling is a software metric indicating how closely connected two (or more) routines or modules are. Low coupling often implies good software design, and if accompanied by high cohesion, leads to high readability and maintainability. [19] The problem of tight coupling of the ZIP archive class and the ZIP metadata parser that occurred in the implementation based on XRootD4 OO API (control coupling) has been fully resolved in the refactored version of the software that uses the new declarative API (data coupling).

Conclusions
The new declarative API introduced in XRootD5 facilitates functional software design and allows for a better-structured code, as confirmed by applying several software metrics. It gives a standard way of composing asynchronous I/O operations, which comes in handy especially when implementing complex remote data access schemes like erasure-coding. In addition, it provides a set of control directives that make it possible to dynamically alter a running I/O pipeline. The XRootD client declarative API is in line with modern C++ programming practices and facilitates use of valuable utilities like lambdas and std::futures in the context of the framework.
The new API provides a convenient and efficient way of doing asynchronous I/O in the present day without the need of falling back to slightly more heavyweight technologies like stackfull boost::fibers or having to wait for compiler support for C++20 (introduces coroutines) on target platforms.