Adaptive angle and parallel multigrid for deterministic shielding problems

Traditionally, solver technology and the space/angle discretisations are intimately linked; sweep-based (wavefront) methods are typically used with DG FEM in space and Sn in angle to solve the Boltzmann-transport equation. These parallelise well (scaling to >100,000 cores) on structured grids, however achieving good scaling on unstructured grids is still an open problem. This talk will focus on alternate space/angle discretisations and solver technology we have been developing within the Applied Modelling and Computation Group (AMCG) at Imperial College. These approaches enables the use of traditional angular discretisations like Pn, Sn, along with new approaches based on linear and haar wavelets. We can use these angular discretisations to perform regular and goal-based anisotropic adaptivity in angle, focusing resolution in important directions. We have also been developing multigrid solver technology which does not require sweep-based methods, allowing the possibility of excellent scaling on unstructured grids.


Introduction
In this work, we focus on the steady-state, first-order integro-differential form of the BTE, given by where ψ(r, Ω, E) is the angular flux at spatial position r, in direction Ω and at energy E. The total and scatter macroscopic cross sections are given by Σ t and Σ s , respectively, with S e the source term.We provide a brief outline of our discretisation of (1), below, taken from [1].In short, we use a sub-grid scale FEM on an unstructured grid to provide a high-order, stable discretisation.For the sub-grid scale formulation, we decompose the solution as ψ = φ + θ, where φ and θ are the solutions on the "coarse" and "fine" scales, respectively.We approximate the solution on the coarse scale with a a e-mail: s.dargaville@imperial.ac.uk b e-mail: andrew.buchan@imperial.ac.uk c e-mail: r.smedley-stevenson@imperial.ac.uk d e-mail: Paul.Smith3@amecfw.com e e-mail: c.pain@imperial.ac.uk continuous finite element representation spanned by η N basis functions, while on the fine scale we use a discontinuous representation spanned by η Q basis functions (like [2]).Doing this, we can recover the system where S Φ and S Θ are the discretised source and Φ and Θ are vectors containing the coefficients of the coarse and fine discretised solutions, φi, j and θi, j , respectively.This is a Schur complement of the block D, or Equation 3 has the same number of DOF as the continuous problem, but now involves the inversion of D. As is typical in Schur complements, we apply approximations to D to increase its sparsity.The inversion of D can be performed element by element (in practice, given the approximations applied, D −1 is often small enough to be stored in memory).Solving (3) can therefore be considered equivalent to solving the continuous problem modified by a local, element-based stabilisation term.The general nature of this discretisation makes it easy to apply different (adaptive) angular discretisations.

Anisotropic angular adaptivity
Goal-based adaptive technology requires solving an adjoint problem, which allows the targeting of resolution where necessary to reduce the error in a goal, like the average flux over a region/surface (see [3][4][5], also see [6][7][8][9] for more examples of the use of angular adaptivity).Figures 1 and 2 show the results of using goal-based angular adaptivity using linear wavelets to discretise in angle.Figure 1b shows that a given error in the average flux of the "green" region show in Figure 1a, is achieved with around an order of magnitude less unknowns than a uniform S n calculation.This is because the goal-based angular adaptivity has focused angular resolution only in the duct region, as shown in Figure 2a. Figure 2b shows that this angular resolution is applied anisotropically on the sphere.

Multigrid solver
Multigrid has been used with great success in radiation transport, applied to the spatial, angular and energy domains, along with combinations of the three.This started with S n discretisations in angle, with multigrid applied both in angle [10,11] and space [12][13][14][15].Spherical harmonics (P n ) have also been used, again with multigrid performed in both space [16][17][18][19] and angle [20].Stepping away from purely spatial or angular multigrid, mixed space/angle [21,22] and space/angle/energy [23] multigrid schemes with S n have been explored recently.Multigrid in energy has also been used to improve the parallel performance of DENOVO [24].
We have been developing matrix-free multigrid solvers for the adapted discretisations described above.With this solver, we can precondition with spatial, angular or mixed space/angle multigrid.This is with the goal of improving the scaling on unstructured grids, when compared with purely sweep-based solvers.Table 1 shows that applying uniform angular resolution with level 4 Haar wavelets (corresponds to 256 angles, see [5]), results in 80% strong scaling efficiency on 6144 cores.
(a) Geometry of the 3D "duct" problem.Red region is a source, with the "goal" set as the average flux over the green region.The duct is a vacuum, surrounded by a scattering region with the bulk a heavy absorber.Figure 2: Anisotropic goal-based adaptivity with linear wavelets applied to 3D "duct" shielding problem [4,5] (b) Convergence of the error in the average flux in the goal region using adaptive linear wavelets.

Figure 1 :
Figure1: Problem geometry and convergence results for a 3D "duct" shielding problem[4,5] Position of wavelets at Node 2 shown in Figure 2a.Most of the wavelets are focused in the direction pointing down the duct region.