Multi-criteria comparative evaluation of spallation reaction models

This paper presents an approach to a comparative evaluation of the predictive ability of spallation reaction models based on widely used, well-proven multiple-criteria decision analysis methods (MAVT/MAUT, AHP, TOPSIS, PROMETHEE) and the results of such a comparison for 17 spallation reaction models in the presence of the interaction of high-energy protons with natPb.


Introduction
In the last decade, high energy nuclear reactions have attracted increasing interest due to scientific problems and numerous applications, such as: creation of high-energy neutron sources, production of medical radioisotopes, radiation protection of space vehicles and accelerators.Solving these problems requires a large amount of nuclear data for a wide range of nuclides and energies up to several dozens of GeV.To obtain all data experimentally is not possible; therefore, it is necessary to develop analytical methods the accuracy of which should be verified by measurements performed under certain conditions [1][2][3][4][5].
There is a growing number of models and programs designed to simulate nuclear reactions in different energy ranges and mass numbers as well as criteria and algorithms to verify the adequacy of simulated nuclear reactions to full-scale experimental data [6][7][8].It should be noted that currently there are no universally accepted theoretical concepts and models that could satisfactorily explain the entire spectrum of the considered nuclear reactions.An evaluation of the predictive ability of calculation tools is based on the goodness-of-fit test.Due to lack of consensus among experts in the subject area on goodness measures and assessment procedures, the evaluation results may vary significantly.
This paper presents an alternative approach to evaluating the predictive ability of models based on the widely used, well-proven methods for discrete decision analysis (MAVT/MAUT, AHP, TOPSIS, PROMETHEE) and the results of such evaluations for the known spallation reaction models (Bertini/Dresner, Bertini/ABLA, ISABEL/Dresner, ISABEL/ABLA, INCL4/Dresner, INCL4/ABLA, CEM2k, CASCADE, CASCADE/ASF, CASCADEX-1.2 [9]) in the presence of the interaction of high-energy protons with nat Pb. a e-mail: korovinyu@mail.ru

Multiple-criteria decision analysis
Multiple-criteria decision analysis (MCDA) methods are aimed at supporting decision makers who have to deal with numerous and conflicting assessments and intend to highlight conflicts and find compromises in the decision making process [10,11].The MCDA problems consist of a finite number of alternatives explicitly known at the beginning of the decision support process.Each alternative is characterized by its performance in multiple criteria.The problem may be defined as finding the best alternative for a decision maker, or selecting a set of acceptable trade-off alternatives.
The MCDA methods provide an opportunity for an analysis of the predefined set of alternatives.MCDA is applicable to the following problem: given a set of alternatives and criteria for their assessment, it can be assumed that each alternative has been evaluated by each criterion either by experts' judgments or objective calculations.Then, it is necessary to derive a rule from the experts' preferences which will make it possible to rank the alternatives according to their values and identify the best among them.
A large number of MCDA methods have been developed to deal with different kinds of problems.The evaluations presented in this study were made using the following well-known and widely used MCDA methods: MAVT (Multi-attribute Value Theory), MAUT (Multi-attribute Utility Theory), TOPSIS (Technique for Order Preference by Similarity to the Ideal Solution), PROMETHEE (Preference Ranking Organization Method for Enrichment Evaluations), AHP (Analytic Hierarchy Process).

Deviation factors
To apply these methods to evaluations of the predictive ability of spallation reaction models in the presence of the interaction of high-energy protons with nuclei, the

Evaluation
of the integrated closeness to the experiment provided that the data can be very different.The index of a power reflects the acceptable compensation degree for small values of some parameters by large values of others.The higher the index, the greater the possible compensation degree.
Evaluation of the relative integral closeness to the experiment.
following reactions for nat Pb were selected.The selection of this set of nuclei is due to the fact that there is a large set of experimental data for nat Pb because it is considered as the basic one for a number of innovative reactor facility projects.
The experimental values were taken from the EXFOR database as well as databases used in the IAEA "Benchmark of Spallation Models" project [12][13][14].
Table 1 shows the deviation factors used in the present study.Table 2 contains the deviation factor values for the nat Pb(p,x) reaction.To evaluate the factors, 279 experimental values were selected for nat Pb residual crosssections at the projectile proton energy values in the range of 70-2600 MeV (see Fig. 1).
The problem is to rank the models in accordance with the values of the totality of the deviation factors.This requires aggregation which is ensured by the use of the MCDA methods.
The studies properly organized on the basis of the MCDA paradigm represent a process as not only technically operating with a set of mathematical methods and various analytical tools but also leading to a comprehensive understanding of the problem and its elaboration.MCDA does not provide a 'right solution'; in this regard, it would be appropriate to speak about a compromise or a trade-off solution, paying special attention to an analysis of the solution stability to the various methods used and their model parameters.

Ranking of models using different MCDA methods
The approach applied in this study consists of several different MCDA methods which may facilitate a thorough understanding, recognition and analysis of the problem, providing an additional sensitivity analysis of the obtained ranking results to the methods used that increase the confidence level of the study.
Application of a wide range of different methods may have a significant impact on subsequent decision making and help a decision maker more thoroughly understand and analyze the problem in order to achieve consistency in judgments and estimates.At the same time, it becomes necessary to examine the stability and robustness of the ranking results towards different assumptions.
Although the ranks of alternatives may vary for different MCDA methods, an analysis of the problem by different methods can play a significant role in the interactive process of understanding the problem and identifying its main features and demonstrate that different methods may provide noncontradictory results.
All the methods are implemented in their simplest form.The assumption of equivalence of all the deviation factors is accepted as the basic option.Changes of the deviation factors' significance (weights) were made within a sensitivity analysis.
Table 3 shows the models ranking results (ranks) using various methods for equal weight option and their combining into groups.As shown in Table 3, the use of various discrete decision analysis methods (MAVT/MAUT, AHP, TOPSIS, PROMETHEE) to evaluate the predictive ability of spallation reaction leads, despite some differences in model ranking, to well-coordinated and similar results.
Despite the fact that the models ranking results are somehow affected by the deviation factors' weights, there are stability regions where the ranking order is preserved in a wide range of weight value changes (Fig. 2).However, models of a certain group do not overrun its borders.
To obviate the necessity of defining the values of weighting factors for a multi-attribute model, a weight stochastic generation method has been implemented which makes it possible to rank models in the absence of information about the significance of certain deviation factors (weights values) as well as determine the model preference probability, i.e., integral performance characteristic of a computational model.
Figure 3 shows the model ranking results with due regard to the uncertainty in the weight values in a box plot format (95, 75, 50, 25, 5% percentiles are denoted; the models are ranked by the average scores, i.e., the values of a multi-attribute value function).The models ranking results based on this approach are consistent with the results obtained from the classical deterministic methods described above.The study demonstrates that taking into account the sensitivity analysis results, an additional alternative analysis using experts' judgments and the whole set of graphical and attributive information, it is possible to select the best models.
A multi-criteria approach to a comparative evaluation of high-energy nuclear reaction models as well as evaluated nuclear data obtained by using these models makes it possible to more finely differentiate various models with due account for experts' opinions, which makes an additional contribution to both the understanding of nuclear reaction mechanisms and preparation of a reliable nuclear data set.

Conclusion
The study has shown that if the MCDA methods are applied to evaluating the predictive ability of spallation reaction models, despite some differences in model ranking, the results obtained by using different methods turn out to give good fits.The study demonstrates that taking into account the sensitivity analysis results, an additional alternative analysis using experts' judgments and the whole set of geographical and attributive information, it becomes possible to select the best models.
The study was partly performed under the support from the Ministry of Education and Science of Russia (unique identifier of applied scientific project 11.9655.2017/bch).

Figure 3 .
Figure 3. Ranking of models for calculating the nat Pb(p,x) reaction.

Table 2 .
Deviation factors for the nat Pb(p,x) reaction.

Table 3 .
Ranking results for the nat Pb(p,x) reaction.
Figure 2. Linear weight approach to weights sensitivity analysis.