High Performance Multigrid on Current Large Scale Parallel Computers

1 downloads 0 Views 717KB Size Report
Feb 26, 2008 - Performance. Adaptive Mesh Refinement. High Performance Multigrid on Current Large. Scale Parallel Computers. Tobias Gradl, Ulrich Rüde.
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

High Performance Multigrid on Current Large Scale Parallel Computers Tobias Gradl, Ulrich R¨ ude Lehrstuhl f¨ ur Systemsimulation Universit¨ at Erlangen-N¨ urnberg

2008-02-26

Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

What is Multigrid?

I

Has nothing to do with Grid computing

I

General methodology: respect the different scales of a problem

I

Useful e. g. for solving elliptic PDEs I I I

convergence rate independent of problem size asymptotically optimal complexity ⇒ algorithmic scalability efficient parallelization — if one knows how to do it

Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Multigrid: V-Cycle Goal: solve Ah u h = f h using a hierarchy of grids Relax on Ah u h = f h

Correct u h ← u h + e h

Residual r h = f h − Ah u h Restrict f H = IhH r h

Interpolate e h = IHh u H

Solve AH u H = f H by recursion

Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Combining FE and MG Parallelization

Combining FE and MG: not straightforward FE: mesh may be unstructured. MG: what nodes to remove for coarsening?

Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Combining FE and MG Parallelization

Combining FE and MG: not straightforward FE: mesh may be unstructured. MG: what nodes to remove for coarsening? Hierarchical Hybrid Grids start with the coarse grid!

Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Combining FE and MG Parallelization

Combining FE and MG: not straightforward FE: mesh may be unstructured. MG: what nodes to remove for coarsening? Hierarchical Hybrid Grids start with the coarse grid!

Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Combining FE and MG Parallelization

Combining FE and MG: not straightforward FE: mesh may be unstructured. MG: what nodes to remove for coarsening? Hierarchical Hybrid Grids start with the coarse grid!

Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Combining FE and MG Parallelization

Combining FE and MG: not straightforward FE: mesh may be unstructured. MG: what nodes to remove for coarsening? Hierarchical Hybrid Grids start with the coarse grid!

Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Combining FE and MG Parallelization

Combining FE and MG: not straightforward FE: mesh may be unstructured. MG: what nodes to remove for coarsening? Hierarchical Hybrid Grids start with the coarse grid!

Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Combining FE and MG Parallelization

Combining FE and MG: not straightforward FE: mesh may be unstructured. MG: what nodes to remove for coarsening? Hierarchical Hybrid Grids start with the coarse grid!

⇒ same stencil for all points within a patch Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Combining FE and MG Parallelization

Combining FE and MG: not straightforward FE: mesh may be unstructured. MG: what nodes to remove for coarsening? Hierarchical Hybrid Grids start with the coarse grid!

⇒ same stencil for all points within a patch Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Combining FE and MG Parallelization

HHG properties

Advantages I

Multigrid is straightforward

I

Very memory efficient 1011 unknowns are possible

I

Very fast

Limitation I

Coarse input mesh needed

Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Combining FE and MG Parallelization

HHG on parallel computers

P0

P0

P1

P1

Mesh is split up at coarsest level → Vertices, Edges, Faces, Volumes Faciliates parallelization for message passing infrastructures (distributed memory parallel computers)

Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Testing on HLRB II

HLRB II at Leibniz-Rechenzentrum M¨ unchen

I

9728 CPUs (1.6 GHz Intel Itanium 2)

I

56.5 Tflop/s peak performance (rank 15 on TOP500 list)

I

38 Tbytes of main memory

Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Performance on HLRB II Processors 64 504 2040 4080 6120 8152 9170

Unknowns (×106 ) 2 147.5 16 911.4 68 451.0 136 902.1 205 353.1 273 535.7 307 694.1

Avg. time per V -cycle (sec) 4.93 5.44 5.60 5.68 6.33 7.43 * 7.75 *

Time to solution (||r || < 10−6 · ||r0 ||) 59.2 65.3 67.2 68.2 76.0 89.2 93.0

*: including high-density partitions

T. Gradl, U. R¨ ude: Massively Parallel Multilevel Finite Element Solvers on the Altix 4700. inSiDE 5(2), 2007. Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Exploring other computer architectures

DEISA*: Access to Europe’s largest computers, support with “enabling work”. Project: Test HHG on a variety of architectures I

Summer 2008

I

100 000 CPU hours

I

Combine with application

* Distributed European Infrastructure for Supercomputing Applications

Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Two approaches Refinement with hanging nodes

Adaptive refinement: motivation Varying mesh density required by I

geometry (e. g. walls of a room)

Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Two approaches Refinement with hanging nodes

Adaptive refinement: motivation Varying mesh density required by I

geometry (e. g. walls of a room)

I

physics (e. g. singularities)

Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Two approaches Refinement with hanging nodes

Two approaches I

Red-green → conforming grids

Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Two approaches Refinement with hanging nodes

Two approaches I

Red-green → conforming grids

Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Two approaches Refinement with hanging nodes

Two approaches I

Red-green → conforming grids

Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Two approaches Refinement with hanging nodes

Two approaches I

Red-green → conforming grids

Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Two approaches Refinement with hanging nodes

Two approaches I

Red-green → conforming grids

I

Hanging nodes → non-conforming grids

Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Two approaches Refinement with hanging nodes

Comparison: red-green vs. hanging nodes

Both have (dis-)advantages (some of them HHG specific): I

Red-green: impossible for purely quadrilateral/hexahedral grids

I

Hanging nodes: too large refined areas

I

Red-green: more elements on coarsest level

The ideal HHG grid has few coarse level elements. ⇒ Use refinement with hanging nodes whenever possible.

Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Two approaches Refinement with hanging nodes

Mathematical foundation Unknowns: 

uh

 i

  for h < hi , i = 1..n = Ihhi u hi i

Residual: AH u H = f H solved up to discretization error ⇔ r H = 0   r h = f h − Ah u h = f h − Ah IHh u H !

r H = IhH r h = 0

Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Two approaches Refinement with hanging nodes

Refinement with hanging nodes in HHG Au = f , r = f − Au, compact basis functions

Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Two approaches Refinement with hanging nodes

Refinement with hanging nodes in HHG Au = f , r = f − Au, compact basis functions Uniform refinement: only one boundary layer

Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Two approaches Refinement with hanging nodes

Refinement with hanging nodes in HHG Au = f , r = f − Au, compact basis functions Uniform refinement: only one boundary layer Adaptive refinement: two boundary layers

Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Two approaches Refinement with hanging nodes

Refinement with hanging nodes in HHG Au = f , r = f − Au, compact basis functions Uniform refinement: only one boundary layer Adaptive refinement: two boundary layers 1. Smooth u

Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Two approaches Refinement with hanging nodes

Refinement with hanging nodes in HHG Au = f , r = f − Au, compact basis functions Uniform refinement: only one boundary layer Adaptive refinement: two boundary layers 1. Smooth u 2. Compute & restrict r

Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Two approaches Refinement with hanging nodes

Implementation

Become flexible, but stay fast. I

Preserve structured regions ...or at least...

I

Treat unstructured regions efficiently.

I

Avoid additional communication ...or at least...

I

Preserve communication locality.

Work in progress...

Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co

Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement

Thank you for your attention! Any questions?

Tobias Gradl, Ulrich R¨ ude

High Performance Multigrid on Current Large Scale Parallel Co