X4Pro – universal, fully relational EXFOR database

. EXFOR library contains experimental reaction data compiled by the Nuclear Reaction Data Centers (NRDC) since 1970 and presenting data from more than 25,000 experiments in the EXFOR format with associated dictionaries and documentation. The IAEA-NDS provides nuclear data services primarily through computer systems built on relational databases and retrieval software. X4Pro is an extended EXFOR relational database designed for programmatic access to whole EXFOR content and distributed as local SQLite database with demo-codes in Python and Fortran.


Introduction
EXFOR (EXchange FORmat) was initially designed as a format for compilation and exchange of experimental reaction data between members of Nuclear Reaction Data Centers network. Using resulting EXFOR files, every data center was arranging own data storage and distribution to users separately. In 2005, NRDC merged EXFOR files from major data centers to a common EXFOR library [1] maintained in the IAEA-NDS and accompanied with dictionaries and documentation. At the same time, nuclear data centers have different policy in data storage and distribution (sometimes cooperating with each other [2]): it includes database and user interface implementation, IT technologies, deliverable data formats, etc.
The IAEA provides EXFOR data services mainly through Web retrieval system and on stand-alone packages (known as "EXFOR CD-ROMs") with database and retrieval software running on the user's computer. All these products are based on EXFOR Relational database with the same schema [2] and maintenance software running at the IAEA-NDS and NNDC on database management systems (DBMS): MySQL, SyBase, MariaDB, MS-Access. Recently developed standalone package X4Lite includes EXFOR Relational database adapted to SQLite, Java programs with GUI interface and non-interactive retrieval code generating output in C4 [3], C5, X4+, XML, JSON and other formats and working on Windows, Linux, MacOS. X4Pro is an extension of EXFOR Relational database.

Problems and tasks
Working with EXFOR data via GUI or API via Web or locally, users are limited by the functionality foreseen by software developers and any significant extensions usually need a lot of additional programming. Recent development in the area of EXFOR usage (model codes, * Corresponding author: v.zerkin@iaea.org evaluation technique, machine learning methods, data formats, programming languages and skills of new generation of physicists) require easy access to whole contents of EXFOR data and associated information from EXFOR dictionaries and external data sources like original publications and old monitor data. EXFOR relational database [2] presents EXFOR meta-data in tables accessible for SQL commands, but data points (i.e., numerical data) are stored as part of SUBENT in BLOB (binary large object). In order to be used, numerical data need to be extracted from BLOBs to EXFOR file, parsed and converted to a universal form (the same units, CM-Lab. system, etc.). So, every time when data are needed in the computational (comparable) form, we need to run parser-converter with whole machinery of dictionaries and additional info -it is hidden under Web/GUI implementation, but when a new user begins to program in such a system, it becomes inconvenient, inefficient and long. There are some other problems with parsers-converters, such as: compatibility with another computer languages, adaptation to different computer platforms, copyright policy, etc.
Obviously, it would be nice to store and directly access all data in a standard relational form without intermediate storage and operations, but there are some problems with rational representation of EXFOR data in a relational form associated with a large variety of the data and volume (now EXFOR has more than 500 types of data and 18 million data points).
Presenting EXFOR data in C4, C5, R33, JSON, XML forms usually works well for specific tasks and users' communities but have problems when trying to present whole EXFOR library. For example, the structure of C4/C5 works well for evaluators because of the use of ENDF compatible designation (MF:MT) but has a limited number of independent variables and finally covers from 60 to 80% of the entire library. JSON and XML presentations can have different hierarchy and sorting order not necessarily convenient for different applications. Anyway, for the moment there is no common decision about the recommended EXFOR output format. Storage of data points in relational database with direct SQL access could solve problems with hierarchy and sorting order because it would be simply prescribed in SQL SELECT command.
Another task is to provide the needed information and recipe for automatic renormalization of EXFOR data using new cross section standards and decay data and for data corrections shared by experts.
If these extensions would be implemented in a relational database, their usage could be arranged (maybe with some limitations) in any programming language having a driver allowing to execute SQL commands.

X4Pro concept and implementation
X4Pro was designed as an extension of EXFOR Relational database with: • data points in original and computational forms, • old and new monitor and decay data, • instructions for data corrections shared by experts. Problem with large variety of EXFOR data types is solved using JSON data type supported by modern relational databases (MySQL, MariaDB, SQLite, PostgreSQL, etc.) developing a multi-model approach since mid-2010s.
X4Pro is implemented in MariaDB and SQLite with 7 additional tables. Initially tables are created in MariaDB and then cloned to SQLite.
Data for renormalization are stored together with every data point for energy dependent monitor data and for whole dataset for decay data. Old monitor and decay data are taken from EXFOR and from Monitor-Archive, new data from IAEA-Standards [4], IRDFF-II [5] and ENSDF [6].
Currently automatic renormalization can be performed for the following data types: integral cross sections (SIG), angular distributions (DA), emission spectra (DE), double differential cross sections (DAE), fission yields (FY). Note: automatic renormalization can be performed using even only SQL commands and applied in any programming language, for example, in Fortran.
Instructions for experts' data modifications are currently stored in the database as Python modules to be loaded on the fly during data retrieval process. Although it is a language dependent component, it allows to avoid programming of own data modification interpreter as it is done in EXFOR Web retrieval system [2].

X4Pro trial distribution
Trial version of X4Pro [7] includes: • X4Pro/SQLite database, 4Gb (incomplete) • Fortran/C EXFOR data retrieval demo programs • retrieve DAE from EXFOR • retrieve SIG from EXFOR and produce C4 file • find and recalculate Legendre polynomials to DA • Python demo programs: • retrieve and plot EXFOR with remote ENDF data • renormalize and correct EXFOR data • recalculate EXFOR data (e.g., Ratio to SIG) • retrieve and plot original EXFOR covariance data • populating CouchDB database from X4Pro Interactive plotting in Python is done using packages Plotly [8] and Matplotlib [9] (see Fig. 1 -9). All codes have been tested in Windows, Linux, MacOS.

Conclusion
X4Pro is an extension of EXFOR Relational database used in nuclear data services of the IAEA-NDS and NNDC since 2004. X4Pro is implemented as set of tables using JSON data type in a multi-model database approach for presenting EXFOR data points in original and computational forms together with monitor and decay data from EXFOR files and external data sources.
X4Pro contains data allowing to renormalize EXFOR data automatically, and codes to modify data applying algorithms shared by experts.
X4Pro can be used from any programming language having SQL support without developing complex, language dependent EXFOR parsers-converters.
X4Pro distribution provides demo codes in Python and Fortran for retrieving data from local EXFOR and remote ENDF databases with advanced operations including automatic and experts' data corrections, recalculations and plotting.
X4Pro can significantly expand the horizons of EXFOR use, making full content of the EXFOR system available for programming access to everyone -from students with homework to professionals with advanced tasks.