Feb 26, 2008 - Performance. Adaptive Mesh Refinement. High Performance Multigrid on Current Large. Scale Parallel Computers. Tobias Gradl, Ulrich Rüde.
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
High Performance Multigrid on Current Large Scale Parallel Computers Tobias Gradl, Ulrich R¨ ude Lehrstuhl f¨ ur Systemsimulation Universit¨ at Erlangen-N¨ urnberg
2008-02-26
Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
What is Multigrid?
I
Has nothing to do with Grid computing
I
General methodology: respect the different scales of a problem
I
Useful e. g. for solving elliptic PDEs I I I
convergence rate independent of problem size asymptotically optimal complexity ⇒ algorithmic scalability efficient parallelization — if one knows how to do it
Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Multigrid: V-Cycle Goal: solve Ah u h = f h using a hierarchy of grids Relax on Ah u h = f h
Correct u h ← u h + e h
Residual r h = f h − Ah u h Restrict f H = IhH r h
Interpolate e h = IHh u H
Solve AH u H = f H by recursion
Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Combining FE and MG Parallelization
Combining FE and MG: not straightforward FE: mesh may be unstructured. MG: what nodes to remove for coarsening?
Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Combining FE and MG Parallelization
Combining FE and MG: not straightforward FE: mesh may be unstructured. MG: what nodes to remove for coarsening? Hierarchical Hybrid Grids start with the coarse grid!
Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Combining FE and MG Parallelization
Combining FE and MG: not straightforward FE: mesh may be unstructured. MG: what nodes to remove for coarsening? Hierarchical Hybrid Grids start with the coarse grid!
Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Combining FE and MG Parallelization
Combining FE and MG: not straightforward FE: mesh may be unstructured. MG: what nodes to remove for coarsening? Hierarchical Hybrid Grids start with the coarse grid!
Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Combining FE and MG Parallelization
Combining FE and MG: not straightforward FE: mesh may be unstructured. MG: what nodes to remove for coarsening? Hierarchical Hybrid Grids start with the coarse grid!
Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Combining FE and MG Parallelization
Combining FE and MG: not straightforward FE: mesh may be unstructured. MG: what nodes to remove for coarsening? Hierarchical Hybrid Grids start with the coarse grid!
Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Combining FE and MG Parallelization
Combining FE and MG: not straightforward FE: mesh may be unstructured. MG: what nodes to remove for coarsening? Hierarchical Hybrid Grids start with the coarse grid!
⇒ same stencil for all points within a patch Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Combining FE and MG Parallelization
Combining FE and MG: not straightforward FE: mesh may be unstructured. MG: what nodes to remove for coarsening? Hierarchical Hybrid Grids start with the coarse grid!
⇒ same stencil for all points within a patch Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Combining FE and MG Parallelization
HHG properties
Advantages I
Multigrid is straightforward
I
Very memory efficient 1011 unknowns are possible
I
Very fast
Limitation I
Coarse input mesh needed
Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Combining FE and MG Parallelization
HHG on parallel computers
P0
P0
P1
P1
Mesh is split up at coarsest level → Vertices, Edges, Faces, Volumes Faciliates parallelization for message passing infrastructures (distributed memory parallel computers)
Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Testing on HLRB II
HLRB II at Leibniz-Rechenzentrum M¨ unchen
I
9728 CPUs (1.6 GHz Intel Itanium 2)
I
56.5 Tflop/s peak performance (rank 15 on TOP500 list)
I
38 Tbytes of main memory
Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Performance on HLRB II Processors 64 504 2040 4080 6120 8152 9170
Unknowns (×106 ) 2 147.5 16 911.4 68 451.0 136 902.1 205 353.1 273 535.7 307 694.1
Avg. time per V -cycle (sec) 4.93 5.44 5.60 5.68 6.33 7.43 * 7.75 *
Time to solution (||r || < 10−6 · ||r0 ||) 59.2 65.3 67.2 68.2 76.0 89.2 93.0
*: including high-density partitions
T. Gradl, U. R¨ ude: Massively Parallel Multilevel Finite Element Solvers on the Altix 4700. inSiDE 5(2), 2007. Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Exploring other computer architectures
DEISA*: Access to Europe’s largest computers, support with “enabling work”. Project: Test HHG on a variety of architectures I
Summer 2008
I
100 000 CPU hours
I
Combine with application
* Distributed European Infrastructure for Supercomputing Applications
Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Two approaches Refinement with hanging nodes
Adaptive refinement: motivation Varying mesh density required by I
geometry (e. g. walls of a room)
Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Two approaches Refinement with hanging nodes
Adaptive refinement: motivation Varying mesh density required by I
geometry (e. g. walls of a room)
I
physics (e. g. singularities)
Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Two approaches Refinement with hanging nodes
Two approaches I
Red-green → conforming grids
Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Two approaches Refinement with hanging nodes
Two approaches I
Red-green → conforming grids
Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Two approaches Refinement with hanging nodes
Two approaches I
Red-green → conforming grids
Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Two approaches Refinement with hanging nodes
Two approaches I
Red-green → conforming grids
Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Two approaches Refinement with hanging nodes
Two approaches I
Red-green → conforming grids
I
Hanging nodes → non-conforming grids
Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Two approaches Refinement with hanging nodes
Comparison: red-green vs. hanging nodes
Both have (dis-)advantages (some of them HHG specific): I
Red-green: impossible for purely quadrilateral/hexahedral grids
I
Hanging nodes: too large refined areas
I
Red-green: more elements on coarsest level
The ideal HHG grid has few coarse level elements. ⇒ Use refinement with hanging nodes whenever possible.
Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Two approaches Refinement with hanging nodes
Mathematical foundation Unknowns:
uh
i
for h < hi , i = 1..n = Ihhi u hi i
Residual: AH u H = f H solved up to discretization error ⇔ r H = 0 r h = f h − Ah u h = f h − Ah IHh u H !
r H = IhH r h = 0
Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Two approaches Refinement with hanging nodes
Refinement with hanging nodes in HHG Au = f , r = f − Au, compact basis functions
Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Two approaches Refinement with hanging nodes
Refinement with hanging nodes in HHG Au = f , r = f − Au, compact basis functions Uniform refinement: only one boundary layer
Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Two approaches Refinement with hanging nodes
Refinement with hanging nodes in HHG Au = f , r = f − Au, compact basis functions Uniform refinement: only one boundary layer Adaptive refinement: two boundary layers
Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Two approaches Refinement with hanging nodes
Refinement with hanging nodes in HHG Au = f , r = f − Au, compact basis functions Uniform refinement: only one boundary layer Adaptive refinement: two boundary layers 1. Smooth u
Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Two approaches Refinement with hanging nodes
Refinement with hanging nodes in HHG Au = f , r = f − Au, compact basis functions Uniform refinement: only one boundary layer Adaptive refinement: two boundary layers 1. Smooth u 2. Compute & restrict r
Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Two approaches Refinement with hanging nodes
Implementation
Become flexible, but stay fast. I
Preserve structured regions ...or at least...
I
Treat unstructured regions efficiently.
I
Avoid additional communication ...or at least...
I
Preserve communication locality.
Work in progress...
Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co
Outline Multigrid The HHG Framework Performance Adaptive Mesh Refinement
Thank you for your attention! Any questions?
Tobias Gradl, Ulrich R¨ ude
High Performance Multigrid on Current Large Scale Parallel Co