Nuclear data adjustment using Bayesian inference, diagnostics for model ﬁt and inﬂuence of model parameters

. The mathematical models used for nuclear data evaluations contain a large number of theoretical parameters that are usually uncertain. These parameters can be calibrated (or improved) by the information collected from integral / di ﬀ erential experiments. The Bayesian inference technique is used to utilize measurements for data assimilation. The Bayesian approximation is based on the least-square or Monte-Carlo approaches. In this process, the model parameters are optimized. In the adjustment process, it is essential to include the analysis related to the inﬂuence of model parameters on the adjusted data. In this work, some statistical indicators such as the concept of Cook’s distance; Akaike, Bayesian and deviance information criteria; e ﬀ ective degrees of freedom are developed within the CONRAD platform. Further, these indicators are applied to a test case of 155 Gd to evaluate and compare the inﬂuence of resonance parameters.


Introduction
In the modelling of a nuclear application (for example, a reactor design), nuclear data libraries are used extensively. These nuclear libraries contain all the information related to nuclear reaction and physics to be used in the modelling. These data are evaluated using nuclear reaction models and differential data. The evaluated data can also be improved by using integral experiments [1,2]. This data improvement process is based on a mathematical framework called the Bayesian inference, where the inclusion of experiments optimizes the model parameters. The Bayesian problem can be solved using analytical (least square analysis) or sampling-based (Monte-Carlo approach) methods [1].
In terms of the physical interpretation of a particular adjustment (i.e., by types of reaction, isotopes, and energy ranges), the adjustment process is still not convincing. The application domain of the adjusted data is not clearly understood. For the case of a new reactor design, it is not clear if the data which was adjusted with a specific set of integral/differential experiments is representative or the data should be adjusted with another set of experiments. In this regard, it is essential to evaluate the influence of input data (or model parameters) on the adjustment. Also, identifying the best distribution while fitting a data set is one of the biggest challenges for mathematicians.
To diagnose the adjustment and fit, there are several approaches (such as goodness of fit, generalizability) proposed in the literature. In terms of fit, it should not be the best fitting model but the best predicting model that can make accurate predictions. In the process of fitting a model to the given data, the preferred model should not be * Corresponding author: e-mail: cyrille.de-saint-jean@cea.fr evaluated based on its goodness of fit (GOF) but its generalizability. The goodness of fit measures root mean square of the deviance (GOF = i (obs i − predict i ) 2 /N). The GOF improves with more parameters.
A model's generalizability is its ability to fit all future data samples from the same process, not only the current data. A model's complexity is its flexibility to fit a wide range of data [5]. It depends on the number of model parameters (the degrees of freedom) and the functional form of the model. We are always interested in a model selection method accounting for the effects of complexity. Over-fitting usually undermines the predictive accuracy. Therefore, among the candidate models, the model that best captures the underlying regularities should be the preferred one.
There are several methods proposed in the literature to select the best model. Some popular selection methods are Akaike information criterion (AIC), Bayesian information criterion (BIC), deviance information criterion (DIC), Cross-validation, Bootstrap, Minimum description length, Cross entropy, etc. Information criteria such as AIC and BIC measure a model's generalizability (i.e., GOF and complexity) [6,9]. These information criteria find a tradeoff between the model simplicity and the GOF. For studying the influence of model parameters on the fit, the Cook's distance (CD) [4,7,8] can be used.
The main objectives of this work are to develop statistical indicators within the existing platform CONRAD [3] (a platform for nuclear data evaluation from CEA) to diagnose the influence of model parameters and model fit in the process of nuclear data adjustment and to demonstrate these indicators using a test case. This work is a continuation of what was presented in [4], with the addition that CD, AIC, BIC, DIC, and the effective number of degrees of freedom are discussed and applied to a nuclear data fit. Also, a complete set of new data is analyzed. Results are compared and discussed.

Data adjustment using Bayesian inference
In probability theory, the Bayes' theorem combines the probability distributions of the prior information and the measurements to provide the posterior distribution. Using the Bayes' theorem, a relationship between the prior, posterior, and the likelihood can be obtained. I.e., the prior is convoluted with the likelihood for the new set of experiments in order to obtain the posterior as expressed below.
The first two statistical moments (mean and covariance) of the posterior distribution can be obtained using a fitting procedure (e.g., generalized least square fit). In nuclear data assimilation and inverse problems, this is a very standard method for data adjustment. The mathematical details are presented in [1,4].

Model selection, measure of influential data, model complexity and fit
As mentioned above, there are several methods to account for the quality of the model and the influence of individual model parameters. In this section, we explore the influence of model parameters by using Cook's distance (Section 3.1) and by using reduced models (Section 3.2.)

Cook's distance for the influential parameter (or data)
The influence of a perticular parameter (or data set) on the overall regression fit can be understood in terms of the variation in the fit if a particular model parameter or a data set is not considered in the fitting process. A generalized formula for the Cook's distance is given in [7,8].
As previously discussed in [4], if σ p is the posterior data adjusted with all model parameters and σ i p is the posterior data when the i th parameter is not considered in the adjustment process, the Cook's distance can be written as: where M σ P is the covariance of the posterior parameters.

Model selection
The model selection process is an essential step in statistical analysis when the accuracy of the model used in the adjustment is of prime interest, especially for nuclear-related applications. Suppose, we have models M 1 , M 2 , .....M k such that where p is the probability of the model parameter x and X j is the parameter space associated with parameter x j of the model M j . To predict the relative quality of a statistical model and to select the best model out of all possible models, information criteria are often used. Information criteria estimate the relative quality of statistical models. The model minimizing the chosen criterion is selected as the best model. In information theory, the Akaike information criterion (AIC) is defined as: where n is the number of model parameters and ln(L) is the log of the maximum likelihood function of model M j . The likelihood is defined as the joint probability distribution of the evaluated values at the given observations. The maximum likelihood (L) corresponds to the likelihood computed with the optimized model parameters in the Bayesian approximation. BIC and DIC are also similar to AIC, but the penalty is harsher. Thus, BIC and DIC tend to choose a simpler model.
In information theory, the Bayesian information criterion and deviation information criterion are defined as: where n D is the effective degrees of freedom and k is the number of experimental data points used in Bayesian parameters adjustment. The model with the smallest values of these criteria will be considered as the best model. Using Kullback-Leibler residual information [9], the effective degrees of freedom n D can be estimated as: where M σ p and M σ p are the prior and posterior covariances. Information criteria are suitable for all types of problems related to model selection and are the most popular methods for model selection. More detailed information about model selection criterion, effective degrees of freedom and goodness of fit test for Bayesian posterior can be found in the article of Spiegelhalter, D. J. [9]. All these methods discussed above (Cook's distance, information criteria and effective degrees of freedom) are implemented in CONRAD and are demonstrated with a test case in the next section.

Test case and results
In this work, for the adjustment process, the time-offlight experiment for radiative capture of 155 Gd in the energy range of [1.4, 3.4] eV is taken as an experiment for the adjustment. In Figure 1, the experimental data (capture cross-section) at 2138 points in the energy range is shown. This experimental data is extracted from the CONRAD database. The uncertainty in the experimental data is 1% of the mean. Two peaks, first at 2.008 eV and second at 2.568 eV are observed. The corresponding resonance parameters (neutron width Γ n , radiation width Γ γ ) and associated uncertainties are shown in Table 1. These four parameters (Γ γ1 , Γ γ2 , Γ n1 , Γ n2 ) are considered here as prior parameters in the Bayesian inference for parameter adjustment. The case where all four parameters are used in the adjustment process can be considered as a model M. To compare information criteria with the Cook's distance, four sub-cases are created by removing one parameter in the adjustment. In this way, four different models M1 (with model parameters Γ γ2 , Γ n1 , Γ n2 ), M2 (with model parameters Γ γ1 , Γ γ2 , Γ n2 ), M3 (with model parameters Γ γ1 , Γ n1 , Γ n2 ) and M4 (with model parameters Γ γ1 , Γ γ2 , Γ n1 ) are used for parameters adjustment with Bayesian analysis. Further, Cook's distance, model selection criteria (AIC, BIC, and DIC), and effective degrees of freedom are computed. The results are tabulated in Table  2.   From Table 2, it can be observed that for each case, the values of information criteria (AIC, BIC, and DIC) are approximately the same. The main reason for this is that for these sub-cases, the number of model parameters (equal to 3) is very small in comparison to the log of likelihood. The largest Cook's distance in M3 suggests that Γ γ2 is a very important parameter for the fit. AIC, BIC, and DIC also suggest that if Γ γ2 is removed from the fit, the model is rejected (large values of information criteria). It can be also noticed from the Cook's distance that Γ n2 is not an important parameter in the fit as the model M4 (in which the parameter Γ n2 is removed) has the smallest Cook's distance. Model M4 with parameters Γ γ1 , Γ n1 , Γ γ2 is the best model out of the given options M1, M2, M3 and M4. All approaches (AIC, BIC, DIC, and Cook's distance) give the same conclusion.
The statistical indicators discussed in this article are currently developed and validated with a simplified test case here. It is expected that these indicators will be used as essential decision-making tools in enhancing the accuracy of the nuclear data adjustment and in selecting the experimental domain for new reactor design.

Conclusions
In this article, for 155 Gd, the influence of resonance parameters is evaluated using Cook's distance and is compared with other model selection techniques (AIC, BIC and DIC criteria) derived from information theory. Effective degrees of freedom is also estimated. It is concluded that all the approaches (AIC, BIC, DIC, and Cook's distance) discussed and developed in CONRAD give the same conclusion. Future work will concentrate on defining the best way to use these indicators to enhance the nuclear data assimilation process by reducing/discriminating the number of parameters.