MULTILEVEL CMFD ACCELERATION METHODS FOR WHOLE CORE REACTOR SIMULATION IN VERA

Ongoing efforts are being made to improve the performance of MPACT as the deterministic neutron transport solver in the Virtual Environment for Reactor Analysis (VERA). As other parts of the code have been improved, the coarse mesh finite difference method (CMFD) has come to take up a significant portion of the runtime. Multilevel-in-energy CMFD and multilevel-in-space CMFD solvers have been used to improve CMFD solver performance. A new multilevel-in-space-and-energy CMFD solver is being introduced that combines components of these two methods. W-Cycles and partial W-Cycles are being investigated to further improve the efficiency of the multilevel-in-energy CMFD solver. The performance of these methods is demonstrated on full core reactor physics problems of interest to VERA.


INTRODUCTION
To support the use of the Virtual Environment for Reactor Analysis (VERA), which is being developed by the Consortium for the Advanced Simulation of Light Water Reactors (CASL), development efforts have been devoted to improving the performance of various components of VERA. The work presented here focuses on accelerating the transport iterations by improving the coarse mesh finite difference (CMFD) solution scheme in MPACT [1], the deterministic transport solver in VERA. MPACT solves the multigroup three-dimensional full core neutron transport problem using the 2D/1D method, coupling together the 2D radial method of characteristics (MOC) and the 1D axial NEM-P3 solution using transverse leakage. The CMFD solver accelerates the overall convergence of the eigenvalue problem, and it also provides stability to the 2D/1D convergence. Typically, there is an MOC plane for each axial node, but the subplane method [2][3] allows solvers to be refined independently, and it also allows multiple axial nodes to exist within each MOC plane. Improving the performance of the CMFD solver is the focus of some effort, as it has come to take up a large component of the runtime as other components have been improved ( e.g., MOC [4], subgroup [5], and cross section processing).
A multilevel-in-energy (MLE) CMFD method has been implemented in MPACT [6]. It was formulated by recursively applying the CMFD equations on progressively coarser energy grids. This can have various numbers of levels, with the coarsest energy grid being two groups. This method showed improvement over single level CMFD. A multilevel-in-space (MLS) CMFD method was also added, with spatial coarsening defined by quarter assembly-collapse and subplane collapse on successive levels [7][8][9]. This paper presents a new multilevel-in-space-and-energy (MLS&E) solver for accelerating the CMFD solution in MPACT. This method combines the previously mentioned MLE and MLS CMFD capabilities by performing coarsening in both energy and space on each level. The accuracy and performance of these new multilevel solvers in various combinations is compared for target problems of interest to CASL. The combined MLS&E CMFD solver can reduce runtime compared to power iterations and the solvers with MLE or MLS individually. The paper will also investigate more complex variations of the multigrid cycles in order to further improve the efficiency of each of these multilevel solvers. These variations include performing W-cycles and partial W-cycles for cases in which the post-sweep solves are skipped [10][11][12][13]. These cycles can potentially reduce the total solve time by reducing the number of outer iterations without significantly increasing the work in the CMFD solver. These multilevel CMFD solvers and multigrid cycles can improve the efficacy and reduce the computational time requirements of MPACT and VERA.

Multilevel in Space+Energy CMFD
An MLS&E CMFD method was developed by combining the mechanisms of both the MLS and MLE CMFD methods. Coarsening is performed in both space and energy on each level. For MLE CMFD, the default number of levels is three as that is the most effective structure for most cases. The MLS CMFD is also broken down into three levels, as shown in Figure 1. Level 1 is the original CMFD mesh, which is pin homogenized radially, and the nodal discretization is homogenized axially. Level 2 is coarsened by collapsing radially to quarter assemblies. Level 3 is the coarsest level, with the quarter assembly collapsed radially and subplane regions collapsed axially if subplanes are used in the model. The MLS&E CMFD coarsens in both space and energy on each level. Thus, the finest grid on Level 1 is the original energy grid and the pin-homogenized spatial mesh. The coarsest grid on Level 3 is 2 energy groups, along with a subplane and a quarter assembly collapsed mesh. The MLS&E CMFD equations are solved using the same algorithm that is used for the MLE and MLS CMFD. The hierarchy of CMFD equations is solved with a single V-Cycle on each outer iteration. The V-Cycle is illustrated on three levels by the first cycle in Figure 1. The spatial restriction is performed usinga typical CMFD flux-volume homogenization. Restriction in space is done with a similar fluxenergy homenization. On each level except the coarsest, a pre-sweep is performed in the form of a fixed source solve. The eigenvalue problem is solved on the coarsest level. A fixed source problem is again solved on all but the coarsest level as a post-sweep using the updated fission source.

Multigrid Cycles
A new variation of this method was developed by using W-Cycles on each outer iteration. A W-Cycle with two multigrid iterations is shown in Figure 2. This type of cycle gives a more tightly converged CMFD solution which can accelerate the outer iterations. Another multigrid iteration scheme that was investigated was partial V-Cycles (pV-Cycle) [8]. This cycle is similar to the standard V-Cycle, except that the post-sweep fixed source problem is skipped. The solution from the eigenvalue problem and fixed source pre-sweep problems is still used to update the fine mesh solution, but without the additional postsweeps. This approach can also be applied to W-Cycle to create partial W-Cycles (pW-Cycles). Using these pV-and pW-Cycles can reduce the number of fixed-source CMFD problems that need to be solved while still solving the eigenvalue problem. A pV-Cycle and a pW-Cycle with two multigrid iterations are show in Figure 2.

RESULTS
The performance of these multilevel CMFD methods was tested on VERA Progression Problems [14]. Problem 5 consists of a Watts Bar Nuclear Unit 1 full core with Westinghouse 17×17-type fuel assemblies depicted in Figure 3. Problem 9 is the same reactor configuration as Problem 5, but with Cycle 1 depletion.  Table I compares the performance of the multilevel methods for Progression Problem 5 run on 406 cores. The first two columns indicate the type of multilevel method and the number of groups on each energy grid. The first row is the standard CMFD method. The third column indicates the number of transport outer iterations. The next two columns give the total time to solve the problem and the total time spent in the CMFD solver. The last two columns give the total speedup and the speedup in the CMFD problem compared to the standard CMFD solver. On each outer iteration the standard CMFD solver iterates on the CMFD problem until the residual is reduced by two orders of magnitude. The MLCMFD solvers simply perform a single V-cycle on each outer iteration. It can be seen that both the MLE and MLS CMFD solvers significantly reduce the number of outer iterations and the CMFD time compared to the standard CMFD, thus reducing the total time by about half. This indicates that the MLCMFD solvers do a better job reducing the error in the CMFD problem and require less work than the standard CMFD. The MLS&E CMFD also reduces the CMFD time, as well as further reducing the number of outer iterations, resulting in the fastest run time. Figure 4 plots the convergence of the fission source for the various CMFD solvers. It can be seen that the standard CMFD solver quickly reaches a slow converging error mode. The multilevel CMFD solvers have much faster convergence rates, with MLS&E CMFD converging the fastest.   fewest outer iterations and the shortest CMFD time, making it the fastest running method. Figure 5 shows the runtime for each depletion statepoint for Problem 9 Watts Bar Cycle 1 with 32 statepoints and 441 effective full power days of depletion. The MLS&E CMFD solver has the shortest runtime for the majority of the statepoints, even when the standard CMFD solver slows down on the later states.   Tables III and IV. The runtime for W-Cycles are shown in Table III for 2-4 energy levels, and the results with standard V-Cycles are included for comparison. Using the W-Cycles reduces the number of outer iterations, but it also increases the CMFD time per iteration. The effect of the W-Cycles is most noticeable for the cases with two energy levels since it has the fewest number of levels in which the CMFD equations are solved. Although the W-Cycles did reduce the number of outer iterations, the increase in the CMFD work meant that they did not reduce the total run time compared to V-Cycles for this problem.  Table IV shows the runtime of pW-Cycles for MLE CMFD with pW-Cycles and 2-4 energy levels. The pV-Cycles did not work to converge the iterations for this problem. For MLE CMFD with two energy levels, the pW-Cycles have fewer outer iterations than the standard V-Cycles on two levels. However, for the pW-Cycles on three or four levels, there was no significant reduction in the number of outer iterations or the total runtime.

CONCLUSIONS
A new MLS&E CMFD solver was developed in MPACT. This solver accelerates the outer iterations and reduces the work in the CMFD problem. The MLS&E CMFD solver took about half the runtime of standard CMFD and was shown to run faster than the standard CMFD and the MLE and MLS CMFD solvers separately. New multigrid iterative algorithms were investigated for the MLE CMFD solver. The W-Cycles reduced the number of outer iterations but did not consistently reduce the runtime. The pW-Cycles also accelerated the outer iterations for some cases but did not give consistent runtime improvements. With these improvements, MPACT continues to aggressively address performance considerations, making it a more powerful, attractive tool for industry users.