ABSTRACT. PETSc proposes a variety of parallel linear solvers. For some linear systems, even ones that are symmetric positive definite, it is still a topic of active research to find a solver that is efficient and scalable. Elliptic problems with heterogeneous coefficients are an example of such systems.

I will present the balancing domain decomposition method as well as its multipreconditioned variant which is more robust and can tackle numerically hard problems. This method can be used in an adaptive way where only some iterations are multipreconditioned. This is enough to ensure good convergence. A crucial feature of the method is that the adaptive procedure is automatic and there is only one parameter that must be chosen by the user. I will also present numerical results demonstrating the efficiency of the method.

ABSTRACT. PRIME (Phase Resolving Simulation Environment) is an Euler-Lagrange code for the simulation of disperse objects in a continuous fluid. It is being developed at the TU Dresden Chair of Fluid Mechanics since 2006, where it was and is employed in a variety of research projects. These cover a wide range of physics starting from single-phase flows including the transport of passive and active scalars, as in the calculation of heat convection, to problems involving disperse multiphase flows and fluid-structure interaction. The disperse phase is geometrically resolved, in general, and coupled to the fluid by an Immersed-Boundary-Method which enables phase-resolving simulations of arbitrary objects such as sediment particles. Other investigations have covered ascending bubbles, even with varying shape, and the behavior of entire clusters thereof, also considering the influence of electromagnetic fields, in case these rise in a liquid metal. Geometrically unresolved small particles are available as well, so as to simulate three-phase flows with fluid, bubbles and particles, for example. A recent extension to highly flexible blade-like structures reaches out into the field of strongly coupled fluid-structure interaction. All simulations are large scale with O(10^5 – 10^9) grid points of the continuous phase. Beyond validating and synchronizing all these development efforts, a key challenge lies in the maintenance of a stable and performant code. PETSc and HYPRE have proven to be reliable pillars of PRIME, allowing us to focus on multi-physics extensions while seeing high speed-up and scale-up in our simulations. In this talk we provide a brief application-driven overview over the numerical models and the physical applications. We aim at sharing our experiences with PETSc, including successes and difficulties, and discuss planned extensions.

ABSTRACT. TPLS (Two Phase Level Set) is a high performance, finite difference progam for the simulation of two phase liquid/liquid or liquid/gas flows. It is available via SourceForge under a BSD licence. It has been under development for a number of years and continues to be developed. TPLS 3.0, which was released in November 2017, uses PETSc to manage the 3D decomposition of the domain and also makes use of PETSc's linear solvers. The program has been validated using the Rayleigh–Taylor instability and has been run with density ratios up to 1,000 and domain sizes of up to 45 million points. The run time configuration of the program is described and some performance results are given.

ABSTRACT. We design a high-order implicit–explicit weighted non-oscillatory (IMEX-WENO) scheme for the solution of the population age density model with the nonlinear mortality rate as well as fertility rate. The mortality rate is the combination of natural mortality that comes to be unbounded at maximum age and bounded external mortality rate that includes external resources as well as seasonality. The existence of global terms in the mortality function as well as in the boundary condition is the main technical complication which provides high nonlinearity for the model equation. We carefully construct a numerical scheme in such way, that the high-order accuracy is maintained in the global terms and boundary condition. The performance of the designed scheme is shown by comparing with the exact solution of the examples considered.

ABSTRACT. Parallel algebraic multigrid (AMG) lacks scalability in both the setup and solve phases due to large costs associated with inter-process communication. We present an open source parallel AMG solver, RAPtor, that was developed for readability as well as parallel performance. Optimizations to the setup phase reduce MPI collective requirements, yielding improved scalability. Furthermore, node-aware communication, in which data is aggregated on node before sent as a single inter-node message, is implemented throughout both the setup and solve phases of RAPtor, greatly increasing the strong scalability of each phase as well as the solver as a whole.

ABSTRACT. PERMON (Parallel, Efficient, Robust, Modular, Object-oriented, Numerical) is a scalable software toolbox for solution of quadratic programming (QP) problems. It is based on PETSc, and it follows its highly successful design.

QP problems arise in various disciplines including elasto-plasticity, contact problems with friction, support vector machines, and many others.

The main functionality is contained in the PermonQP module. It provides a framework for the solution of QP problems. It includes data structures, transformations (e.g. dualization), algorithms, and supporting functions for QP. PermonQP can use all solvers and interfaces to the external packages provided by PETSc as well as our implementations of algorithms such as SMALBE (SemiMonotonic Augmented Lagrangian algorithm for Bound and Equality constraints) or MPRGP (Modified Proportioning with Reduced Gradient Projections).

Among the presented QP applications will be the solution of a model 3D linear elasticity contact problem using Total Finite Element Tearing and Interconnecting (TFETI) and its novel Projector-avoiding variant implemented in PermonFLLOP. The classical (T)FETI operator consists of a generalized inverse of the stiffness matrix scaled from both left and right by a corse problem (CP) projector. The solution of CP represents a bottleneck in the FETI methods parallelization. The main idea of the Projector-avoiding TFETI method lays in a cheap way to project this generalized inverse onto the range of the stiffness matrix. This projection creates Moore-Penrose pseudoinverse that is much better conditioned than the classical TFETI operator. Not only we obtain a system with improved eigenvalue distribution, but we also do not need to multiply this matrix by CP projectors. Thus avoiding some of the CP projector applications. Our tests of the Project-avoiding method show about 1.7 speedup over standard TFETI on problems with more than 1.25 billion of unknowns computed on up to 15625 cores.

Another application will be machine learning using linear Support Vector Machines (SVMs) implemented in the PermonSVM module. The quality of the solution, as well as the scalability of the QP solvers, will be demonstrated on benchmarks from the LIBSVM collection and datasets related to the Pfam protein database used in the Exascale Compound Activity Prediction (ExCAPE) project.

ABSTRACT. The optimization of problems governed by partial differential equations is inherently computationally expensive. If the original simulation requires a number of N timesteps then the optimization requires $2Nm$, where $m$ is the total number of iterations up to convergence towards a local mininum/maximum. It is therefore crucial to operate at the limit of accuracy and efficiency. In this regards we explore the impact of matrix-free highly vectorized spatial discretizations, checkpoint reduction, and multigrid approaches. The problem under consideration, two-dimensional Burger's equation, encompasses all the difficulties that cripple PDE-constrained optimization problems, such as ill-conditioning, lack of global extremum, restrictive stability bounds as well as a strong coupling between forward and adjoint solves.

ABSTRACT. In this talk we present a unified approach how to introduce PETSc to a software architecture where multiple applications communicate with each other by means of MPI-directives. This introduction is motivated by the requirement to provide efficient numerical algorithms and data structures for the AC² code family that is developed by the GRS. This code family is used to simulate the behavior of nuclear reactors during operation, transients, design basis and beyond design basis accidents up to radioactive releases to the environment.

To improve the overall load balance and to compensate for possible shortage of system resources (e. g. on portable computers) the approach allows for a shared use of the processes that are dedicated to numerical computations. It is also possible to establish numerical logic that works on data from different applications. The talk puts special emphasis on the techniques that where developed to create and configure an MPI-communicator scheme that provides the aforementioned features.

ABSTRACT. Various strategies will be presented to extend the scalability of existing multigrid or domain decomposition linear solvers using appropriate recycling strategies or block methods, i.e., by treating multiple right-hand sides simultaneously. Furthermore, I will talk about the design of an efficient matrix-free finite element solver for the radiative transport equation and my journey into the beautiful world of MATSHELLs.

ABSTRACT. DMStag is a new implementation of PETSc's DM class. It represents a topologically-Cartesian cell complex based on a rectangular grid of line segments, quadrilaterals, or hexahedra, along with routines to manipulate interacting fields with respect to these top-level elements ands lower-dimensional cells in the complex. The intended use is for finite volume or DEC schemes, on regular grids. DMStag is intermediate between DMDA and DMPlex; it represents a structured grid like DMDA, but supports different strata like DMPlex. It also supports a notion of compressed coordinates, allowing for efficiency associated with knowing that coordinates are regularly-spaced in one or more dimensions, or describable as a product of 1- or 2-dimensional coordinate grids. Finally, it offers compatibility checks to ensure consistency in the use of multiple DMs representing the same physical domain. Existing PETSc-based codes operating with staggered grids must use DMPlex, optimized for unstructured meshes, multiple DMDA objects, or a single DMDA with multiple degrees of freedom. Using DMDA in these ways pushes the burden of maintaining index conventions and ensuring consistent parallelization to the user, as well as making it more difficult to experiment with different data layouts. The DMStag API aims to make working with a staggered grid-based code, for instance a finite volume Stokes solver, at least as easy as working with codes based on collocated quantities on a regular grid.

ABSTRACT. Underworld2 is an open-source, particle-in-cell finite element code tuned for large-scale geodynamics simulations. The hybrid numerical algorithms enable the tracking of historic information, via the lagrangian points, while the structured computational mesh allows for efficient solving methods, such as multigrid. Underworld has been used extensively to study the dynamics of subductions zones ( Moresi et al., 2014 ); instabilities of the lithosphere ( Beall et al., 2017 ) and dynamics of convecting planetary mantles ( Farrington et al., 2014 ). Common to all these studies is the description of material properties (velocity, temperature, density, viscosity, etc) varying both in space and time and often functions of each other. These varying properties feed into the numerical algorithms as coefficients for the pertinent discretised PDEs, defining the linear algebra solution and then being updated accordingly. Underworld offers a versatile and customisable system to define such material properties called `uw.function`, a programmable pointwise operator object. `uw.function`s are composed in a dynamic and user friendly python environment and perform on-the-fly evaluations in a computationally efficient C/C++ API.

In this case study Underworld’s `uw.function` is prototyped with Petsc’s PetscDS object which requires a pointwise evaluation of discretized PDE integrands. This study showcases how an existing project can leverage Petsc to enhance its existing numerical capability with the added benefit of significantly reducing a legacy codebase. Future possibilities of this prototyped are examined and discussed.

ABSTRACT. We developed a CFD code using PETSc's DMPlex. The code is an entropy stable discontinuous spectral collocation implementation developed for the solution of the compressible Navier Stokes equations. The presented work supports unstructured high order grids, it supports tensor product elements directly and simplex elements through partition. The entropy stability property of the method allows for stable solution at high order (over p=15). Our interest on solving at such high order is for locality of information and reduced communication. We observed that at a sufficiently high order we can successfully hide the cost of communication allowing for scalability up to thousands of processors.

ABSTRACT. An anisotropic mesh adaptive shallow water solver is created for tsunami modelling, building upon work in the Firedrake and Thetis projects. The former provides efficient, finite element solving software and the latter specialises to coastal, estuarine and ocean modelling applications, on unstructured meshes. A variety of a posteriori error estimates are considered, including those based on reconstructed Hessian information, residual approximations and implicit error estimates. Numerical solutions of the adjoint shallow water equations are considered for weighting such error estimates, providing a goal-oriented approach.

This work enables the efficient generation of numerical solutions to tsunami wave propagation problems using the shallow water equations. The case study of the 2011 Tohoku tsunami is considered, for which leading tsunami waves reached the Japanese coast in just ten minutes. Results indicate that, under mesh adaptivity, computational cost can be reduced, whilst retaining a sufficiently high accuracy in terms of the relative error in computing an objective functional of interest. Through establishing an efficient approach to numerical tsunami simulation, sufficient warning could be provided in future scenarios, allowing for evacuation and damage mitigation in coastal areas determined most at risk.

ABSTRACT. The Multipoint Flux Mixed Finite Element (MFMFE) method is a way to robustly discretize a variable-coefficient Poisson operator using cell-centered variables. We can generalize it to other differential operators by saying that the generalized MFMFE method is the local elimination of a nodal ansatz codifferential by a sparse approximate inverse (SPAI) mass matrix. This poster shows three useful recent extensions in this area: consistent SPAI definition for nonconforming meshes, MFMFE for linear elasticity, and MFMFE for Stokes equations.

ABSTRACT. Code generation is an increasingly popular technique for the development of complicated finite element discretisations of partial differential equations. In this poster, we show new developments in the Firedrake finite element framework that allow the code generation pipeline to exploit structure inherent to some finite elements. This includes, in particular, sum factorisation on tensor product cells for the full family of $Q^-$ spaces.

This is achieved by providing a more expressive interface between the form compiler and the library providing the implementation of finite element tabulation. This is encapsulated in a new finite element library, FInAT, that explicitly communicates the structure of elements. New form compiler algorithms are introduced that exploit this structure.