Impactful PRISM research
The Platform for Research in Simulation Methods (PRISM, https://prism.ac.uk) is a major research programme based at Imperial encompassing activity in computational science. The main outputs of PRISM are frameworks for encoding composable mathematical abstractions as software, making advances in numerical mathematics practically applicable to a wide range of high-performance real-world applications across industry and in government. PRISM software is used to benchmark modelling technology in industrial fluid mechanics at McLaren Formula 1 and Rolls Royce, is in operational use by DUG Technologies to provide seismic imaging services to many oil and gas majors, is the prototyping platform for the advanced numerics in the Met Office’s next generation climate and weather model, and forms the core of the Finnish Meteorological Institute’s development of a new operational capability for sea level forecasting.
Computer simulations of fluids and solids underpin much of modern engineering, as well as forming the basis of the weather and climate forecasts that inform daily life and national policy. Advances in the numerical mathematics underpinning these simulations can enable higher fidelity simulations at larger scale. However advanced techniques are also harder to implement, with the result that sophisticated mathematics is often only applied to toy problems while real applications are limited to simpler but less capable techniques. The contribution of PRISM is to create software abstractions for the fundamental mathematics describing advanced numerical algorithms which can be composed together by scientists and engineers to create bespoke simulation capabilities. With PRISM-created software, the sophistication of the numerical discretisations and solvers available to scientists and engineers is no longer limited to the (typically simple) set that they can implement themselves. This mathematical composability of operators lets users easily make substantial changes to the algorithm, to discover the best approach for their current science challenge and available hardware. PRISM researchers take this automation and composability approach much further than other systems, so that many advanced approaches which are often regarded as too difficult in practice become straightforward. Two EPSRC Platform Grants with classifications in the mathematical topics of Continuum Mechanics and Fluid Mechanics, have supported PRISM in the development of a suite of computational modelling software, which has been widely applied in industry and public sector science and engineering.
We now describe the software encapsulated by PRISM. Firedrake [1] and Devito [2] replace the hand-derivation of the computation by computer algebra compiler processes which automatically apply the required series of symbolic transformations. This results in the automatic generation of the program which conducts the calculations. The scientist or engineer writes equations, and the computation occurs automatically, in parallel, achieving much higher computational performance than typical hand coding. Firedrake provides this capability for a vast range of finite element methods, while Devito provides high order finite difference capabilities for the wave equations critical to the oil and gas industries. The ability to effectively exploit complex finite element discretisations has underpinned numerics research led by Cotter [3] into compatible finite element methods which effectively capture the dynamics of atmospheric circulation. This work has provided an accurate numerical scheme for weather and climate prediction which avoids the catastrophic resolution singularities at the Earth’s poles which plague the previous methods.
PyFR [4] is a compressible flow simulation framework using high order flux reconstruction techniques to exploit modern massively parallel hardware such as GPUs. PyFR’s domain specific language separates the specification of the flux reconstruction algorithm from its implementation, which is generated by a specialised compiler. This enables the code to exploit the different brands and generations of GPUs available on different supercomputers without rewriting the core software.
Nektar++ [5] employs inheritance and template metaprogramming to enable the application of finite element techniques which combine spatial and polynomial refinement. This delivers unmatched levels of accuracy at the cost of complexity which would be infeasible to deliver in a more conventional software framework.
PRISM software and the underpinning innovative numerical methods have been widely adopted by industrial and public sector users. The impacts are as diverse as the beneficiaries: whole new weather and climate simulation capabilities, commercial advantages in the seismic Impact imaging sector, and competitive edge in Formula 1 racing to name but a few. In the following, we detail examplar impacts of each PRISM component.
The UK Met Office uses Firedrake as the principal dynamics prototyping platform for its next generation weather and climate simulation system. The Firedrake high productivity development environment has enabled Cotter’s advanced compatible finite element discretisations (and related solvers) to be demonstrated at scale as a solution to the Met Office’s current parallel performance limitations. The Met Office decision to go ahead with the new model development, and to base it on these numerics, rests on Firedrake and the research it underpinned. “The capability of compatible FEM in the GungHo dynamical core is a key element of our strategy in developing a model that will be fit for the mid-life upgrade of our new supercomputer. This is currently expected to be in 2025 and up to £1.2 billion investment has been approved for this new machine.” The Met Office has now committed to a 180 FTE years project to turn these decisions into an improved forecasting system for the UK.
Firedrake plays a similar role for EDF’s simulations in support of their geological storage of nuclear waste. The ability to rapidly combine sophisticated linear solvers and preconditioners has enabled them to overcome the scaling limitations of their current approaches, and they have committed resources to implement the improvements in their production code.
The Finnish Meteorological Institute have built a coastal ocean modelling system, Thetis, directly using Firedrake. They use the 3D version of this model to develop their understanding of the complex internal processes that characterise the coastal ocean. The 2D version of the model enables sea level simulation over a wide area and has been chosen as the basis for their next generation operational forecasting capability. “We would not be developing simulation capabilities at this level without Firedrake.”
DUG Technologies Ltd (a public company with market capitalisation A$112.40M as 27th January 2021) uses Devito in production. DUG provides seismic imaging services to the oil and gas industry. Seismic imaging is a computational process which creates images of subsurface structures from acoustic survey data. Accurate seismic imaging enables fewer survey wells to be drilled, reducing the financial and environmental cost of this activity. Seismic imaging is a petascale problem in which the ability to rapidly translate numerical advances into operational use is key to the competitiveness of this technology. The agility that the code generation approach provides is the reason DUG adopted Devito. DUG have rebuilt the core computational engine for their DUG Wave waveform inversion software around Devito, achieving a factor of 10 improvement in performance and scalability. Over the past year, DUG Wave has been running continuously using Devito on computing resources equivalent to the 100-200 range of the world’s Top500 supercomputer 2020 list. The code now scales to tens of petaflops, translating to a cost reduction of millions of dollars for their customers. “Without Devito, the same amount of processing with previous technology would have exceeded our total compute capacity.”
McLaren Formula 1 Racing uses Nektar++ to guide their race car design. Their standard tool is a Reynolds Averaged Navier Stokes (RANS) solver code which is fast but has limited applicability in terms of flow conditions (particularly during breakdown of vortical flows which are a key component of maintaining aerodynamics performance of racing cars). Nektar++ is a high-fidelity tool that has much broader applicability. Competition regulations prevent the direct application of high-fidelity tools during the race car season, so it is critical to understand the conditions where RANS loses reliability. McLaren use Nektar++ to develop protocols to decide whether given RANS simulations are reliable or not, and therefore whether the results suggest useful improvements to make to the car’s aerodynamic shape.
Rolls Royce uses Nektar++ for similar purposes in their next-generation turbomachinery design. Their RANS code is used as a crucial part of their design cycle, and they compare RANS predictions with those made by Nektar++ in order to guide their design process.
Zenotech, based in Bristol, delivers market-leading high-performance computing tool and consultancy services for businesses in the aerospace, automotive, civil and renewable energy sectors. Zenotech has added a flux reconstruction solver based on PyFr to its proprietary zCFD codebase, complementing the existing finite volume solver. The resulting high order solver is being used by Zenotech to simulate aircraft parts and in acoustic problems for the rail sector.
[1] Rathgeber F, Ham DA, Mitchell L, Lange M, Luporini F, McRae AT, Bercea GT, Markall GR and Kelly PH (2017) Firedrake: automating the finite element method by composing abstractions. ACM Transactions on Mathematical Software (TOMS), 43(3), p.24, doi:10.1145/2998441.
[2] Louboutin M, Lange M, Luporini F, Kukreja N, Witte PA, Herrmann FJ, Velesko P, and Gorman GJ (2019) “Devito (v3. 1.0): an embedded domain-specific language for finite differences and geophysical exploration.” Geoscientific Model Development 12(3): 1165-1187, doi:10.5194/gmd-12-1165-2019.
[3] Natale A, Shipton J, and Cotter CJ (2016) “Compatible finite element spaces for geophysical fluid dynamics.” Dynamics and Statistics of the Climate System 1(1), doi:10.1093/climsys/dzw005.
[4] Witherden FD, Farrington AM, and Vincent PE (2014), “PyFR: An Open Source Framework for Solving Advection-Diffusion Type Problems on Streaming Architectures using the Flux Reconstruction Approach,” Computer Physics Communications, 185 (11), p. 3028-3040, doi:10.1016/j.cpc.2014.07.011.
[5] Cantwell CD, Moxey D, Comerford A, Bolis A, Rocco G, Mengaldo G, De Grazia D, Yakovlev S, Lombard J-E, Ekelschot D, Jordi B, Xu H, Mohamied Y, Eskilsson C, Nelson B, Vos P, Biotto C, Kirby RM, and Sherwin SJ, “Nektar++ : an open-source spectral/hp element framework,” Computer physics communications, 192, p. 205–219, 2015, doi:10.1016/j.cpc.2015.02.008.
