Uncovering Where Compensating Errors Could Hide in ENDF/B-VIII.0

.


Introduction
Nuclear data libraries, among them the current U.S. nuclear data library ENDF/B-VIII.0[1], are adversely affected by compensating errors that lurk in unconstrained physics spaces [2][3][4].Unconstrained physics spaces arise between nuclear data observables of the same or different isotopes due to a combination of imprecise differential information and the integral nature of validation experiments.On the one hand, differential information (from theory and experiments) is often imprecise and may lead to evaluated uncertainties that are significant in terms of impact on application simulations.For instance, ENDF/B-VIII.0nuclear data uncertainties lead to a combined uncertainty of 1,025 pcm on the simulated effective neutron multiplication factor, k eff , of the Jezebel critical assembly [3]; this uncertainty on simulated k eff is between four to five times the range from prompt to delayed critical for this assembly.On the other hand, sub-sets of nuclear data libraries are validated with respect to integral experiments that often use several 100s to 1,000s of nuclear data to simulate a single output quantity.Consequently, these experiments cannot uniquely identify an error in any particular nuclear data value as being responsible for an observed difference between simulation and experiment.Freedom in both differential and integral information can permit widely varying combinations of nuclear data to si- * e-mail: dneudecker@lanl.govmultaneously fit within uncertainties of differential information and reliably predict experimentally measured integral quantities.However, only one combination of nuclear data is a true representation of the underlying physics; all other combinations suffer from compensating errors.
Compensating errors are not a concern for application simulations that are closely aligned with integral experiments used to validate the nuclear data.However, if an application extrapolates beyond available validation experiments, compensating errors could significantly adversely impact the predictive power of transport simulations. 1  If one knows where unconstrained physics spaces occur within nuclear data libraries, then new experiments could be built or new theory developed to impose further targeted constraints that reduce the compensating errors with the largest impact.
The EUCLID (Experiments Underpinned by Computational Learning for Improvements in Nuclear Data) LDRD-DR (Laboratory Directed Research & Development-Directed Research) project [7] developed a novel process that leverages machine learning and expert judgment to pin-point unconstrained physics spaces. 1 Also, historically, nuclear data have been primarily validated with respect to well-vetted ICSBEP critical-assembly benchmarks [5].Some nuclear data application areas do not have similar benchmark suites, and their simulations are more likely to be adversely impacted by compensating errors in nuclear data libraries.Therefore, the nuclear data and applications community highlighted the need for validation with responses beyond criticality [6].
A preliminary version of this process is described in Section 2 along with its input data in Section 3.1 and methods used in Section 4. Section 5 walks through two examples to show how unconstrained physics spaces are identified in the ENDF/B-VIII.0library.The examples relate to nuclear data for 9 Be and 239 Pu.The discussion in Section 6 focuses on how such results can be used for future developments but also on limitations of the preliminary process.Conclusions and an outlook are provided in Section 7.

Process to identify unconstrained physics spaces
The process to identify unconstrained physics spaces, depicted in Fig. 1, consists of 5 steps: 1. Machine learning (ML) methods are used to predict which nuclear data observables are likely related to bias in simulating integral responses.To this end, differences between simulations and integral observations are compiled, along with sensitivities of simulated results to the nuclear data inputs.These biases and sensitivities are inputs to the code RAFIEKI (Random Forest Integral Experiment Knowledge Interpreter) used in Refs.[4,8].RAFIEKI fits a random forest to predict bias and interprets it using the SHAP (SHapley Additive ex-Planations) metric [9,10] to provide a sorted list of nuclear data quantities that are most closely associated with bias.
2. Experts work through this prioritized list to identify combinations of nuclear data that could be jointly responsible for bias.An unconstrained physics space could exist between these nuclear data observables based on only integral information.

Input data
Three types of data are used for the process in Fig. 1 to identify unconstrained physics spaces: (1) integral data, (2) differential data, and (3) sensitivities.Sensitivities encode how simulated integral values change with input nuclear data.All input data are described below.

Integral experiment input
Four integral-experiment response types are used in step 1: • Effective neutron multiplication factors, k eff , for more than onethousand ICSBEP (International Criticality Safety Benchmark Evaluation Project) critical assemblies [5] are used.These data are the gold-standard for validating nuclear data; k eff values are provided with credible uncertainties as low as 0.05% [5], and data sets are carefully vetted as part of the ICSBEP review process.Therefore, the standardized bias (∆) between experiment (E) and simulated (C, for calculated) values is within ±2 for the large majority of assemblies shown in the top of Fig. 2. ∆ is measured in units of the standard deviation for combined xperimental and Monte Carlo uncertainties, δE and δC.
• The spread in ∆ is distinctly larger for the neutronleakage spectra [12] of LLNL pulsed-spheres (second plot from the top in Fig. 2).These experiments were not yet vetted to be benchmarks and questions remain about concrete surrounding the beamline and the modeling of the neutron source, etc. [13,14].These experiments measure neutrons leaked from a sphere of material containing few or only one isotope that was pulsed by a 14-MeV D+T neutron source situated in the center of the sphere.Nuclear data from 5-15 MeV, dependent on the thickness of the sphere and the level scheme of the nuclear data, can be validated with neutron-leakage spectra [13].Thus, LLNL pulsed sphere experiments distinctly extend the energy range of validation beyond what can be reached with k eff alone (up to approximately 5 MeV).
• Data on the effective delayed neutron fraction, β eff , of nine critical assemblies [5] are also considered (third plot from the top in Fig. 2); β eff can be used to tease out delayed versus total components of fission.However, the high total uncertainties (on the order of 8%) reduce the impact of these experiments considerably [15].• Reaction rates measured in the center of Jezebel, Govida, Flattop-HEU, and Flattop-Pu critical assemblies are also included in the analysis [16].They enable a more detailed analysis of the reactions measured or the prompt-fission neutron spectrum, and are more precise than LLNL pulsed-sphere neutron-leakage spectra and β eff .Also, the sensitivities to the average promptfission neutron multiplicity are negligibly small, allowing to factor out this component from the fission source term [16,17].However, fewer, and less biased, data points (bottom of Fig. 2) enter this analysis compared to k eff and pulsed-sphere neutron-leakage spectra.
The rich set of integral data beyond k eff is powerful because it extends the energy range of validation and encompasses different sensitivities of C to nuclear data.This variety translates into more effective constraints on nuclear data quantities-specifically, fewer degrees of freedom in correctly predicting E across all the experiments studied.
We take experimental and simulated values into account for each experiment in the form of the standardized error, ∆.Pulsed-sphere neutron-leakage spectra, β eff , and k eff values were simulated with MCNP ® Code Version 6.2 2 [18], while reaction rates were computed with SENSMG [19].All values of C were calculated with ENDF/B-VIII.0[1].

Differential data
In step 3 of Fig. 1, nuclear data are compared to differential experimental data from the EXFOR database [11] using EXFOR plotting tools [20].Many differential data have a one-to-one correspondence to a specific nuclear-data value and thus enable experts to directly assess the quality of 2 MCNP ® and Monte Carlo N-Particle ® are registered trademarks owned by Triad National Security, LLC, manager and operator of Los Alamos National Laboratory.Any third party use of such registered marks should be properly attributed to Triad National Security, LLC, including the use of the designation as appropriate.For the purposes of visual clarity, the registered trademark symbol is assumed for all references to MCNP within the remainder of this paper.nuclear data as well as how far specific values can move and stay within the spread and uncertainties of differential experimental data.

Sensitivities
In steps 1 and 4 of Fig. 1, sensitivities, S , of C with respect to nuclear data are used to identify unconstrained physics spaces.They encapsulate how C is impacted by changing specific nuclear data.Sensitivities for k eff were calculated via the KSEN card implemented in MCNP6.2 [18].Sensitivities for β eff were derived by a linear combination of adjoint-based k eff sensitivities with and without delayed neutrons [15].Brute-force methods were applied to compute S for LLNL pulsed-sphere neutron-leakage spectra that relied on perturbing nuclear data and computing via MCNP6.2 the change in C for each perturbation [13,15,21].SENSMG was used to obtain reaction rate sensitivities [16,19].

Methods
The question that is asked in step 1 of Fig. 1 is: "Which combinations of different nuclear data lead to bias in simulating integral experiments?"The tool RAFIEKI employed in Refs.[4,8] leverages machine learning to augment experts' ability in answering this question by predicting ∆ as a non-linear function of features via a random forest [22].These features correspond to sensitivities, S , of the four integral responses mentioned with respect to all pertinent nuclear data.Then, the SHAP metric [9,10] is applied to the random forest predictor to give a sorted list of which nuclear data are most important to predict bias.The higher the SHAP value is for a specific nuclear data observable, the more likely it is related to bias.
The random forest and SHAP metric are needed as we implicitly try to find which of more than 20,000 nuclear data are related to bias in simulating more than 6,000 experimental data points.One issue is that this problem is  highly underdetermined, as 6,000 equations (number of ∆) depend on 20,000 observables (S ).To further complicate matters, many nuclear data co-vary together in simulating C. For instance, even in the simplest experiment studied, such as a mono-isotopic pulsed sphere ( 27 Al, 16 O, etc.) or a bare critical assembly, several scattering, and if applicable, fission nuclear data are used to compute C. It was shown in Ref. [4,8] that due to this integral nature of the experiments, the random forest points to several, covarying, nuclear data observables as related to bias.Additionally, the relationship between nuclear data sensitivites and bias may not be causal.For instance, the bias might result from missing nuclear data or experimental shortcomings, but the random forest can only point to the features provided, and will therefore apportion bias only to nuclear data sensitivities.Therefore, rather than having RAFIEKI directly identify the source of bias, the SHAP results from RAFIEKI inform nuclear data experts about underlying sources of bias from the predictive observables.

Results
The process of Fig. 1 is showcased below by identifying whether there are unconstrained physics spaces in 239 Pu and 9 Be ENDF/B-VIII.0nuclear data.

The 239 Pu example
It is already known from Refs.[2,3] that unconstrained physics spaces exist between 239 Pu nuclear data observables.Hence, this example allows us to counter-check whether the process works.
In the first step, ∆ values shown in Fig. 2 and all sensitivities of these experiments with respect to pertinent nuclear data are fed into the random forest.The SHAP metric then lists which nuclear data are most important to predict bias for k eff , LLNL pulsed-sphere neutron-leakage spectra, β eff , and reaction rates.
In the second step, an expert judges which nuclear data are likely to co-vary together in predicting ∆.Here, we assumed that all 239 Pu (and 9 Be for the example below) nuclear data observables in the fast range co-vary together in simulating integral experiments.This assumption is supported for 239 Pu by SHAP values, shown in the top of Fig. 3, that are similar (and large) across different observables for the same energy range.For instance, the prompt-fission neutron spectrum, average total fission neutron multiplicity, inelastic and fission cross sections are all predicted to be strongly related to bias in the same energy range.The (n,2n) cross section becomes strongly related to bias at higher energy (above its threshold), where LLNL pulsed spheres enable validation of nuclear data.While the capture and elastic cross sections are less strongly related to predicting ∆, they are linked with the inelastic and fission cross sections through the total cross section.Therefore, the set of spectra, multiplicity, fission, elastic, inelastic, capture, and (n,2n) cross sections could all be related jointly to ∆.However, given the integral nature of the experiments used, it could be that nuclear data is incorrect for just one or a sub-set of those observables, while bias is assigned to all data that are correlated with each other through their joint use in simulating C. Thus an unconstrained physics space is postulated between them.
In the third step, one should try to reduce the number of observables that are deemed responsible for predicting ∆ by excluding some as not biased based on differential experimental data.However, differential experimental data for the subset of prompt-fission neutron spectrum, fission cross section, and average prompt-fission neutron multiplicity in Fig. 4 clearly show that there is significant freedom in moving these nuclear data within experimental uncertainties.Fewer experimental data exist for inelastic, elastic, capture, and (n,2n) cross sections.Therefore, all seven 239 Pu observables in the top of Fig. 3 are potentially biased.
In step four, one could exclude those observables related to bias where changes in the nuclear data within differential experimental data do not significantly impact the simulation of C.However, it is a well-known fact that small changes in the nuclear data of the observables in the top of Fig. 3, let alone changes within the spread of existing experimental data, significantly impact the simulation of k eff [23].Hence, all seven 239 Pu observables in the top of Fig. 3 remain as potentially leading to bias in predicting ∆ (conclusion in step 5).As this is more than one observable, it is unknown which specific nuclear data needs to change to reduce ∆.Consequently, an unconstrained physics space is diagnosed for these 239 Pu observables, as expected from Refs.[2,3] .
The 9 Be example The 9 Be example is different: RAFIEKI identifies two observables, (n,2n) and elastic cross sections in the fast range, as strongly related to bias, shown in the bottom of Fig. 3.In addition to that, the elastic cross section only plays a role in predicting ∆ above 14 MeV where only pulsed-sphere neutron-leakage spectra provide experimental information.Hence, the expert identifies in step 2 only one unique observable as strongly related to bias below 14 MeV, the (n,2n) cross section.Although you can inter-pret the RAFIEKI results as pointing to a problem in the (n,2n) cross section, one may not want to in this case.One should rather, more carefully, paraphrase the RAFIEKI results to conclude that a neutron-emission contribution of 9 Be is potentially suffering from biases.In fact, the 9 Be inelastic cross section is missing in ENDF/B-VIII.0nuclear data, and work is currently ongoing at LANL to include it [24].This example illustrates two important points.First, there is no unconstrained physics space for 9 Be below 14 MeV, because only one observable is highlighted as related to bias.Second, expert judgment is needed to interpret RAFIEKI results.While RAFIEKI would point to the (n,2n) cross section as biased, experts inform us that this is rather a clear indication for the need of additional ((n,inl) cross section) nuclear data.RAFIEKI cannot point to the (n,inl) cross section as related to bias because it is not available in ENDF/B-VIII.0,and, therefore, no sensitivities exist.However, the results did point us into the right direction (i.e., neutron-emission in the fast range).

Discussion
The process to identify unconstrained physics spaces was shown above to correctly pin-point where unconstrained physics spaces exist in ENDF/B-VIII.0,as well as being able to inform where individual nuclear data should be improved.This process is deemed preliminary because it relies on expert judgment in two cases: First, expert judgment comes into identifying which nuclear data are likely jointly related to bias.Right now, it is assumed that nuclear data of the same isotope co-vary together.However, it is expected that nuclear data across several isotopes would covary together in predicting ∆.For instance, 240 Pu appears, to at least a small percentage, in all Pu critical assemblies and LLNL pulsed spheres.Hence, when one predicts C of these experiments, 239 Pu and 240 Pu nuclear data would be expected to co-vary together and compensating errors could hide between nuclear data of those two isotopes.This issue compounds when one simulates k eff of thermal solution assemblies, where nuclear data of 1 H, 16 O, 14 N, components of steel, and major actinide nuclear data are all used jointly.Hence, compensating errors could occur between nuclear data of those isotopes.The current process does not account for that and might miss critical correlations between data.Therefore, work is ongoing at LANL to identify which nuclear data co-vary in simulating various integral responses, and build this part of the analysis into RAFIEKI.
Expert judgment comes also into play, when one assesses whether a change in nuclear data matters for simulating integral experiment responses.While this is straight-forward to assess for the examples used above, this is not clear-cut for minor actinides or isotopes appearing in structural materials.Tools such as FAUST and NDaST [25,26] allow one to systematically evaluate if small changes in nuclear data, within differential bounds, would impact the simulation of integral experiments.
This process can be generalized to other types of problems.For instance, one could apply the same process to identify where model parameters insufficiently describe cross sections or fission observables.To give an explicit example, fission event generators such as CGMF, FREYA, and FIFRELIN [27][28][29] predict prompt-fission neutron and gamma spectra, along with average prompt-fission neutron and gamma multiplicities based on pre-neutron emission fragments yields, Y(A), the average total kinetic energy of the fission fragments, T KE , etc.More precise experimental data exist for the average prompt-fission neutron multiplicities, ν p , and spectra, PFNS, which take the place of integral data in step 1 of Fig. 1, not only because of their precision, but also as ν p and PFNS predictions depend in a convoluted manner on Y(A), T KE , etc. Random forest and SHAP could help identify which model parameters linked to Y(A) and T KE are related to bias in simulating average prompt-fission neutron multiplicities and spectra, given sensitivities that link model parameters and output multiplicities and spectra.Experimental values for Y(A) and T KE take the place of differential data in step 3 of the process, while sensitivities of model parameters to ν p and PFNS need to be used in step 4. Through this, one could learn which parameter would need to be improved to better predict ν p and PFNS.The current problem has an advantage over applying this process to CGMF, FREYA, and FIFRELIN [27][28][29]: we can reasonably assume that our "model" linking simulations of integral data and nuclear data is correct, i.e., that the transport codes are able to predict all experimental input quantities correctly.This is not the case for CGMF, where model defects are known to affect the prediction of the PFNS.Still, applying this process can point the model developers to parameter spaces where weaknesses could hide.

Conclusions and outlook
The EUCLID (Experiments Underpinned by Computational Learning for Improvements in Nuclear Data) LDRD-DR (Laboratory Directed Research & Development-Directed Research) project [7] developed a process that enables identifying unconstrained physics spaces between several observables in nuclear data libraries.Unconstrained physics spaces arise between various nuclear data observables due to imprecise differential information (from theory and experiment) on each observable, and the integral nature of validation experiments that uses several observables at once to simulate one integral value.Due to a lack of constraints, several differing combinations of data can be found for the same nuclear data observables that obey the bounds of differential data and predict integral responses adequately.Only one combination of nuclear data can accurately represent nature; the others are riddled with compensating errors.It is important to identify unconstrained physics spaces and develop experiments (integral and differential) and theory with the explicit aim to bound them.
EUCLID's process to identify unconstrained physics spaces hinges on analyzing with machine learning methods (random forest, SHAP metric) which nuclear data observables are jointly responsible for predicting bias.We then down-select via expert judgment-using differential data and sensitivities of integral responses to nuclear data-those nuclear data observables that span an unconstrained physics space where compensating errors could hide in ENDF/B-VIII.0.We applied this procedure to identify unconstrained physics spaces in 239 Pu, where we already knew that an unconstrained physics space existed.But we also showcased an example ( 9 Be nuclear data), where the same process identified issues in the nuclear data.In the discussion, it was highlighted that the same process can be applied in other subject areas, where one deals with uncertainties across inter-dependent and heterogeneous data.
The EUCLID project applied this process to several isotopes of ENDF/B-VIII.0,and selected from that 239 Pu ν p , PFNS, elastic, inelastic, capture, and fission cross sections in the fast range as a combination of nuclear data that is likely affected by compensating errors.We will now proceed to design a combination of integral experiments to reduce these compensating errors [7].
Another course of future work is addressing the shortcomings in the preliminary process: Expert judgment is used in defining which nuclear data observables are jointly related to bias in simulating integral experiments.Right now, SHAP only gives us a list of which nuclear data are most important in predicting bias, but does not tell us the combination of nuclear data that lead to bias in simulating integral responses.While expert's input is crucial in correctly identifying the issues at hand in nuclear data, it is difficult for humans to assess correctly which combinations of nuclear data co-vary together when simulating integral responses due to the high dimensionality of the problem (6,000 experiments depending on 20,000 nuclear data).However, work is ongoing to pair principal component analysis-a technique made to tackle such problems-with random forests to identify combinations of nuclear data that jointly predict bias.Also, the tool FAUST [25] will be used to consistently assess the impact of proposed changes in nuclear data.
Lastly, only four different integral responses were included in the analysis so far; more responses have been studied by EUCLID [16].To calculate the necessary sensitivities of various integral responses, new sensitivity calculation capabilities have been implemented in the MCNP code [30].In the near future, five more responses (reactivity coefficients, neutron-leakage spectra of critical assemblies, Rossi-alpha, Feynman-Y, and count rate of subcritical assemblies [31][32][33]) will be used for this type of analysis.Once these new responses and principal component analysis are implemented into the process to identify unconstrained physics spaces, a journal article will follow.

Figure 1 .
Figure 1.The preliminary process to identify unconstrained physics spaces.

2 +
C 2 of reaction rate

Figure 2 .
Figure2.The standardized bias, ∆, of integral data used for the analysis here is shown for the following responses: k eff , LLNL pulsed-sphere neutron-leakage spectra, β eff , and reaction rates in critical assemblies.

Figure 3 .
Figure 3. SHAP results are shown in color for 239 Pu (top) and 9 Be (bottom) nuclear data.White color indicates a SHAP value of 0, meaning this reaction is not predicted to be related to bias.

Figure 4 .
Figure 4.The spread in differential experimental data of the 239 Pu prompt-fission neutron spectrum at incident-neutron energies of 1.5 MeV, fission cross section, and average prompt-fission neutron multiplicity, ν p , is compared to ENDF/B-VIII.0nuclear data.