ExCALIBUR

ExCALIBUR is the UK Exascale preparation scheme, providing hardware and software effort in order to prepare the UK HPC community for forthcoming Exascale services.

COSMA hosts several enabling hardware prototype systems funded by ExCALIBUR, which are hosted as part of the HPC Hardware Laboratory:

  1. DINE - a BlueField-2 (previously BlueField-1) enabled HPC cluster

  2. AMD MI GPU systems

  3. A Rockport network fabric, using a switchless 6D torus topology

  4. CXL-prototype composable RAM systems

  5. DWAVE quantum annealing credits

To use these systems, please contact cosma support.

AMD GPU systems

COSMA has a MI100 GPU in server ga004.

THere are also six AMD MI50 GPUs hosted in the ga003 server. These are interlinked with a 4x and 2x InfinityFabric link between the GPUs, to enable direct data transfer. However, unless you specifically need this, please use the newer MI100 system instead.

Relevant software is in /opt/rocm* and the AMD AOCC compiler is available as a module (module load aocc)

The rocm_smi.py command will provide information about the GPUs.

These systems are available for general use. If you use them, feedback would be welcome, in particular around performance, and the software environment.

DWAVE quantum annealing credits

From 1st June 2023 for a 24 month period, we have arranged access to DWAVE quantum annealing systems.

Please contact cosma-support if you wish to access this.

This is available to any of the UK community.

ExCALIBUR software projects at Durham

There are several ExCALIBUR software projects (finished and ongoing) at Durham, investigating Exascale software techniques to allow codes to run effectively on Exascale systems. These are:

  1. Clawpack enabled Exahype for heterogeneous hardware

  2. Massively Parallel Particle Hydrodynamics for Engineering and Astrophysics

  3. Particles At Exascale (PAX)

  4. Quantum Enhanced and Verified Exascale Computing (QEVEC)

MPPHEA

Massively Parallel Particle Hydrodynamics for Engineering and Astrophysics

SPH (smoothed particle hydrodynamics), and Lagrangian approaches to hydrodynamics in general, are a powerful approach to hydrodynamics problems. In this scheme, the fluid is represented by a large number of particles. Each particle tracks a Lagrangian element, moving with the flow. The scheme does not require a predefined grid making it very suitable for tracking flows with moving boundaries, particularly flows with a free surfaces, and problems that involve the mixing of different fluids, flows with physically active elements or large dynamic range. The method was originally developed to study stellar collisions, but is widely adopted in engineering and cosmology because of it flexibility, adaptivity and natural multi-physics capability. The range of applications of the method is growing rapidly and is being adopted by a rapidly growing range of commercial companies including Airbus, Unilever, Shell, EDF, Michelin and Renault.

The widespread use of SPH, and its potential for adoption across a wide range of science domains, make it a priority use case for the Excalibur project. Massively parallel simulations with billion to hundreds of billions of particles will revolutionise our understanding of the Universe and will empower engineering applications of unprecedented scale, ranging from the end-to-end simulation of transients (such as bird strike) in a jet engines, to simulation of tsunami waves over-running a series of defensive walls.

The ExCALIBUR project brings together a wide range of experience and knowledge, from Engineering, Computer Science and Astrophysics. The group will identify the limitations of current approaches and will develop proof of concept solutions that will set out the path to SPH simulations in the exa-scale limit.The working group bring together two state of the art codes. SWIFT and DualSPHysics:

  1. SWIFT: a state-of-the-art SPH fluid dynamics solver that uses a novel Fine-grained task parallelism approach

  2. DualSPHysics: a state of the art GPU enabled SPH code.

The team

PI:

Richard Bower (Durham)

Co-Is:

Alastair Basden (Durham)

Tobias Weinrzierl (Durham)

Carlos Frenk (Durham)

Benedict Rogers (Manchester)

Georgios Fourtakas (Manchester)

Scott Kay (Manchester)

Robert Crain (Liverpool John Moores)

Tom de Vuyst (Hertfordshire)

Pablo Lauren-Aguilar (Exeter)

Dave Acreman (Exeter)

Matthieu Schaller (Leiden)

Workgroup meeting minutes

Unedited meeting minutes

Resources

ExCALIBUR Training

A code performance workshop series (7 monthly workshops) was run in 2021, and we are looking at repeating this in the future.

ExCALIBUR Cross-cutting research

The Quantum Enhanced and Verified Exascale Computing (QEVEC) project has been funded as part of the ExCALIBUR cross-cutting research call.