Oct 27, 2017 - Ariane 6 conception (2012-2020). > more than 50% of the aero-budget in experimental studies. > 70% of the studies using large numerical ...
Alexandre Limare AMR in FLUSEPA Octree method Execution flow
Adaptive refinement of intersecting meshes, in a high-order Finite Volume framework, for aeropropulsion applications PhD Defense - Saclay
Hangingnodes 2-1 Balancing
High-order redistribution Problem description Validation
Load balancing Industrial test cases Martel bench
Alexandre Limare GAMMA3 - UTT - Inria ArianeGroup Supervisors Pierre Brenner - ArianeGroup Houman Borouchaki - UTT
27th October 2017
Simplified after body flow
Conclusion
1 / 46
Alexandre Limare
Context
AMR in FLUSEPA Octree method Execution flow
Aim: Understand and simulate complex physical phenomena more accurately
Hangingnodes 2-1 Balancing
High-order redistribution Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Ariane 5 design phase (1987-1996) > Stage separation − Reduced models > Take-off ignition overpressure wave ⇒ Aerothermal databases: wind tunnel tests
Ariane 6 conception (2012-2020) > more than 50% of the aero-budget in experimental studies > 70% of the studies using large numerical simulation tools
Conclusion
2 / 46
Alexandre Limare AMR in FLUSEPA Octree method Execution flow Hangingnodes
Context Fluid Mechanics for space launchers • Complex violent unsteady phenomena (aerodynamic efforts, turbulence,
chemistry, dispersed phases influence) • Flows about bodies in relative motion
2-1 Balancing
High-order redistribution Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Conclusion
Figure: Boosters separation (FLUSEPA-ArianeGroup) Figure: Missile stage separation, re-entry of the Huygens probe 3 / 46
Alexandre Limare AMR in FLUSEPA Octree method Execution flow
Context FLUSEPATM1 • 30 years old in-house CFD tool • Euler, Navier-Stokes, turbulence modelling (RANS, URANS, HRL)
Hangingnodes 2-1 Balancing
High-order redistribution Problem description Validation
Load balancing Industrial test cases
(a) A5 Ignition OverPressure wave at take-off
Martel bench Simplified after body flow
Conclusion
(b) Propeller simulation
Numerical formulation • Unstructured cell-centered 3D Finite-Volume (Godunov + k-exact reconstruction) • Adaptive explicit integration scheme • Overset grids by conservative geometric intersections > Input cells: hexa, prism, penta and tets 1
Registered trademark in France with number 134009261 4 / 46
Alexandre Limare AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution
Overset grids Mesh overlapping strategy > Simple part of bodies meshed independently > Large displacements, meshes adapted to attached boundary layers and bow shocks > Motion calculated by integration of the aerodynamic & propulsive loads for each body with (6-DoF model)
Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Conclusion
(c) Detail of the multi component mesh
(d) Assembled
Figure: Multi-level intersecting meshes
5 / 46
Alexandre Limare AMR in FLUSEPA
Overset grids Mesh intersections
Octree method
• Based on priority levels
Execution flow
• Geometric Intersection (no interpolations) > Intersection faces directly used for fluxes calculations
Hangingnodes 2-1 Balancing
• Residual cells are assembled (CFL)
High-order redistribution Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Conclusion
Figure: Multi-level intersecting meshes 6 / 46
Alexandre Limare AMR in FLUSEPA
Overset grids Mesh intersections
Octree method
• Based on priority levels
Execution flow
• Geometric Intersection (no interpolations) > Intersection faces directly used for fluxes calculations
Hangingnodes 2-1 Balancing
• Residual cells are assembled (CFL)
High-order redistribution Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Conclusion
Figure: Multi-level intersecting meshes 6 / 46
Alexandre Limare AMR in FLUSEPA
Overset grids Mesh intersections
Octree method
• Based on priority levels
Execution flow
• Geometric Intersection (no interpolations) > Intersection faces directly used for fluxes calculations
Hangingnodes 2-1 Balancing
• Residual cells are assembled (CFL)
High-order redistribution Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Conclusion
Figure: Multi-level intersecting meshes 6 / 46
Alexandre Limare AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description Validation
Adaptive explicit solver (Slave) Considered competitive compared to an implicit solver for unsteady simulations
Principles • compute each cell near its maximum physical time step (CFL condition) > maximum allowed time step computed for each cell > each cell j is sorted according to its time level L: 2L dtmin . ≤ dtj < 2L+1 dtmin and put into a time level > each time level and its set of cells has its own time step (2L dtmin ) • only fluxes interpolation for boundaries between temporal levels (still
conservative) • during one iteration, multiple sub-iterations
Load balancing
time
Industrial test cases Martel bench Simplified after body flow
2∆tmin ∆tmin
Conclusion
class 0
class 1 Figure: Initial state
7 / 46
Alexandre Limare AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description Validation
Adaptive explicit solver (Slave) Considered competitive compared to an implicit solver for unsteady simulations
Principles • compute each cell near its maximum physical time step (CFL condition) > maximum allowed time step computed for each cell > each cell j is sorted according to its time level L: 2L dtmin . ≤ dtj < 2L+1 dtmin and put into a time level > each time level and its set of cells has its own time step (2L dtmin ) • only fluxes interpolation for boundaries between temporal levels (still
conservative) • during one iteration, multiple sub-iterations
Load balancing
time
Industrial test cases Martel bench Simplified after body flow
2∆tmin ∆tmin
Conclusion
class 0
class 1
Figure: Sub-iteration 1
7 / 46
Alexandre Limare AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description Validation
Adaptive explicit solver (Slave) Considered competitive compared to an implicit solver for unsteady simulations
Principles • compute each cell near its maximum physical time step (CFL condition) > maximum allowed time step computed for each cell > each cell j is sorted according to its time level L: 2L dtmin . ≤ dtj < 2L+1 dtmin and put into a time level > each time level and its set of cells has its own time step (2L dtmin ) • only fluxes interpolation for boundaries between temporal levels (still
conservative) • during one iteration, multiple sub-iterations → cost per cell varies
Load balancing
time
Industrial test cases Martel bench Simplified after body flow
2∆tmin ∆tmin
Conclusion
class 0
class 1
Figure: Sub-iteration 2
7 / 46
Alexandre Limare AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution
Industrial goals
Why AMR ? Reliable: • Work within the validity domain of each physical model • Limit error-prone user interventions • Capture the phenomena involved and needed
Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Conclusion
Efficient: • Automate time-consuming tasks (input generation) • Integrate clever error-correcting systems • High Performance Computing properties
Sustainable: • Retrocompatibility with available modules • Hardware independent
8 / 46
Alexandre Limare
Outline
AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description Validation
1 AMR in FLUSEPA
Octree method Execution flow Hanging-nodes 2-1 Balancing 2 High-order redistribution
Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
3 Load balancing 4 Industrial test cases
Martel bench Simplified after body flow
Conclusion
5 Conclusion
9 / 46
Alexandre Limare
Outline
AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description Validation
1 AMR in FLUSEPA
Octree method Execution flow Hanging-nodes 2-1 Balancing 2 High-order redistribution
Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
3 Load balancing 4 Industrial test cases
Martel bench Simplified after body flow
Conclusion
5 Conclusion
10 / 46
Alexandre Limare
Octree method
AMR in FLUSEPA Octree method Execution flow
⇒Octree-AMR : Large error correction and simple implementation
Hangingnodes 2-1 Balancing
High-order redistribution Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Full hexa meshes Adaptation method chosen: Octree per cell with a 2 − 1 per node balancing:
Advantages • simple algorithmic method • FLUSEPA’s solver allows general
polyhedra • low memory overhead due to AMR
structures (15 %) Figure: Octree method
Conclusion
10 / 46
Alexandre Limare AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution
Octree Requirements : • High-order, conservative, redistribution method of the conservative variables • Adaptation of the intersection algorithms • Geometric properties properly calculated • Treatment of "non-conform" zones • Load balancing
Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Conclusion
Figure: Non-conform zone
11 / 46
Alexandre Limare
Structural modification of FLUSEPA
AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description
Turbulence modeling
Eulerian dispersed phases
Chemical reactions modeling
Liquid-gaz interfaces
K-exact reconstruction
Adaptive explicit
Flux calculation
Implicit
Source/coupling terms calculation
Input/Output Topology/geometry Temporel integration Spatial discretisation Physical modeling Reduced models Mesh adaptation
• • • • • • •
Sub-domain partitioning
Input/Output
Mesh displacement/deformation
Validation
Load balancing
Mesh adaptation sensors
Mesh deformation
Industrial test cases
AMR refinement/coarsening
Coherent flow structure capturing
Martel bench Simplified after body flow
Fusion of too covered cells
Engine pilot Mechanical bonds Structure kinematic
Radiation • Structures • Fluid/Particules
Split + projection of the variables
Structure deformation
Structure thermal response
Geometric characteristics
Conclusion
Figure: Schematic view of FLUSEPA’s modules 12 / 46
Alexandre Limare
Structural modification of FLUSEPA
AMR in FLUSEPA Octree method
Turbulence modeling
Eulerian dispersed phases
Execution flow
Chemical reactions modeling
Liquid-gaz interfaces
• • • • • • •
Hangingnodes 2-1 Balancing
High-order redistribution
K-exact reconstruction
Adaptive explicit
Flux calculation
Implicit
Source/coupling terms calculation
Input/Output Topology/geometry Temporel integration Spatial discretisation Physical modeling Reduced models Mesh adaptation
SLAVE MASTER
Input/Output
Problem description Validation
Mesh adaptation sensors
Sub-domain partitioning
Load balancing
Mesh displacement/deformation
Industrial test cases
Mesh deformation
Martel bench Simplified after body flow
Conclusion
AMR refinement/coarsening
Coherent flow structure capturing
Fusion of too covered cells
Engine pilot Mechanical bonds Structure kinematic
Radiation • Structures • Fluid/Particules
Split + projection of the variables
Structure deformation
Structure thermal response
Geometric characteristics
12 / 46
Alexandre Limare
Execution flow
AMR in FLUSEPA Octree method Execution flow
Master
Hangingnodes 2-1 Balancing
High-order redistribution Problem description
Slave SOLVER Slave SOLVER
Validation
Load balancing
Slave SOLVER
Industrial test cases
Slave
Martel bench Simplified after body flow
Time
Conclusion
Figure: Mesh adaptation algorithm
13 / 46
Alexandre Limare
Execution flow
AMR in FLUSEPA Octree method
MPI message
Execution flow
Master
Hangingnodes 2-1 Balancing
High-order redistribution Problem description
Slave SOLVER Slave SOLVER
Validation
Load balancing Industrial test cases
Slave SOLVER AMR sensor Slave
Martel bench Simplified after body flow
Time
Conclusion
Figure: Mesh adaptation algorithm
13 / 46
Alexandre Limare
Execution flow
AMR in FLUSEPA Octree method
MPI message
Execution flow
Master
Hangingnodes 2-1 Balancing
High-order redistribution Problem description
Cells to be refined/coarsened
Slave SOLVER Slave SOLVER
Validation
Load balancing Industrial test cases
Slave SOLVER AMR sensor Slave
Martel bench Simplified after body flow
Time
Conclusion
Figure: Mesh adaptation algorithm
13 / 46
Alexandre Limare
Execution flow
AMR in FLUSEPA Octree method
MPI message Topology update
Execution flow
AMR & Intersection
Hangingnodes 2-1 Balancing
High-order redistribution Problem description
Cells to be refined/coarsened
Slave SOLVER Slave SOLVER
Validation
Load balancing Industrial test cases
Master
Slave SOLVER AMR sensor Slave
Martel bench Simplified after body flow
Time
Conclusion
Figure: Mesh adaptation algorithm
13 / 46
Alexandre Limare
Execution flow
AMR in FLUSEPA Octree method
MPI message Topology update
Execution flow
AMR & Intersection
Hangingnodes 2-1 Balancing
High-order redistribution Problem description
Cells to be refined/coarsened
Differential topology
Slave SOLVER Slave SOLVER
Validation
Load balancing Industrial test cases
Master
Slave SOLVER AMR sensor Slave
Martel bench Simplified after body flow
Time
Conclusion
Figure: Mesh adaptation algorithm
13 / 46
Alexandre Limare
Execution flow
AMR in FLUSEPA Octree method
MPI message Topology update
Execution flow
AMR & Intersection
Hangingnodes 2-1 Balancing
High-order redistribution Problem description
Master
Cells to be refined/coarsened
Differential topology
Slave SOLVER
H-O Redist.
Slave SOLVER
H-O Redist.
Validation
Load balancing Industrial test cases
Slave SOLVER
H-O Redist.
AMR sensor
Redistribution of the variables of the modified cells
Slave
Martel bench Simplified after body flow
Time
Conclusion
Figure: Mesh adaptation algorithm
13 / 46
Alexandre Limare
Execution flow
AMR in FLUSEPA Octree method
MPI message Topology update Load balancing
Execution flow
AMR & Intersection
Hangingnodes 2-1 Balancing
High-order redistribution Problem description
Cells to be refined/coarsened
Differential topology
Slave SOLVER
H-O Redist.
Slave SOLVER
H-O Redist.
Validation
Load balancing Industrial test cases
Master
Slave SOLVER
H-O Redist.
AMR sensor
Redistribution of the variables of the modified cells
Slave
Martel bench Simplified after body flow
Time
Conclusion
Figure: Mesh adaptation algorithm
13 / 46
Alexandre Limare
Execution flow
AMR in FLUSEPA Octree method
MPI message Topology update Load balancing
Execution flow
AMR & Intersection
Hangingnodes 2-1 Balancing
High-order redistribution Problem description
Cells to be refined/coarsened
SOLVER
Full new topology Local indexing
Industrial test cases
Slave
H-O Redist.
Slave SOLVER
H-O Redist.
Validation
Load balancing
Master
Slave SOLVER
H-O Redist.
AMR sensor
Redistribution of the variables of the modified cells
Slave
Martel bench Simplified after body flow
Time
Conclusion
Figure: Mesh adaptation algorithm
13 / 46
Alexandre Limare
Execution flow
AMR in FLUSEPA
Idle time
Octree method
MPI message Topology update Load balancing
Execution flow
AMR & Intersection
Hangingnodes 2-1 Balancing
High-order redistribution Problem description
Cells to be refined/coarsened
SOLVER
Full new topology Local indexing
Industrial test cases
Slave
H-O Redist.
Slave SOLVER
H-O Redist.
Validation
Load balancing
Master
Slave SOLVER
H-O Redist.
AMR sensor
Redistribution of the variables of the modified cells
Slave
Martel bench Simplified after body flow
Time
Conclusion
Figure: Mesh adaptation algorithm
13 / 46
Alexandre Limare
Octree
AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description Validation
Load balancing Industrial test cases
Master • Full topology available (even
the covered cells) • Full tree kept • Original mesh cannot be
coarsened Figure: Master tree
Martel bench Simplified after body flow
Conclusion
14 / 46
Alexandre Limare
Octree
AMR in FLUSEPA
P
Octree method
P
Execution flow
Tα
Hangingnodes 2-1 Balancing
High-order redistribution
(a) Tα to be refined
Problem description
Tα,4
Tα,3
Tα,1
Tα,2
(b) Tα,i after refinement Tα,3 is totally covered
Validation
Load balancing
Figure: Refinement of a partially covered cell Tα , covering polyhedron P, shaded zone
Industrial test cases Martel bench Simplified after body flow
Conclusion
Slave • Partial topology (covered
cells are removed) • Differential modification • No tree kept Figure: Temporary slave tree 15 / 46
Alexandre Limare AMR in FLUSEPA Octree method
The original solver of FLUSEPA relies on a general description of the polyhedra for the geometric intersections, the treatment of hanging nodes implies few new developments
Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description Validation
Ω1 12
3
4
5
6
7
M2
Load balancing Industrial test cases Martel bench
Figure: Ω1 cell cut by a higher priority grid M2 , local numbering of the obtained faces
Simplified after body flow
Conclusion
16 / 46
Alexandre Limare
Description of the polyhedra per cell
AMR in FLUSEPA Octree method Execution flow Hangingnodes
Intersections calculation rely on a definition of conform closed surfaces > Each cell is represented as a closed surface It is conform and AMR use must ensure that each surface stays conform
2-1 Balancing
High-order redistribution Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Conclusion
Figure: Basic hexahedron
17 / 46
Alexandre Limare
Description of the polyhedra per cell
AMR in FLUSEPA Octree method Execution flow Hangingnodes
Intersections calculation relies on a definition of conform closed surfaces > Each cell is represented as a closed surface It is conform and AMR use must ensure that each surface stays conform
2-1 Balancing
High-order redistribution Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Conclusion
Figure: Basic hexahedron originally treated as a dodecahedron
17 / 46
Alexandre Limare
Description of the polyhedra per cell
AMR in FLUSEPA Octree method Execution flow Hangingnodes
Intersections calculation relies on a definition of conform closed surfaces > Each cell is represented as a closed surface It is conform and AMR use must ensure that each surface stays conform
2-1 Balancing
High-order redistribution Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Conclusion
Figure: Dodecahedron definition is not sufficient for varying level of refinement
17 / 46
Alexandre Limare
Description of the polyhedra per cell
AMR in FLUSEPA Octree method Execution flow Hangingnodes
Intersections calculation relies on a definition of conform closed surfaces > Each cell is represented as a closed surface It is conform and AMR use must ensure that each surface stays conform
2-1 Balancing
High-order redistribution Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Conclusion
Figure: New face definition
17 / 46
Alexandre Limare
Triangulation per face
AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Conclusion
Figure: Full library of the possible faces with regards to the existing hanging nodes
divided faces nbpolyhedra
0 4096
1 1536
2 480
3 480
4 30
5 6
6 1
Table: Number of possible polyhedra considering the number of type "2" faces
total : 6629 cases per hexahedron → Mesh is constrained so as to limit the number of possible cases 18 / 46
Alexandre Limare
2 − 1 balancing of the meshes
AMR in FLUSEPA Octree method Execution flow Hangingnodes
0
2-1 Balancing
High-order redistribution Problem description Validation
Load balancing
0 Ni 1 0
0 0
Industrial test cases Martel bench Simplified after body flow
(a) Balanced unstructured grid before refinement. In red: the cell having to be refined. In green: the level of refinement of the cells in the valence ball of Ni
Conclusion
Figure: Example of a mesh adaptation with 2 − 1 per node balancing rule for the node Ni
Intra-mesh cell size progression constraint, what about the intersections ? 19 / 46
Alexandre Limare
2 − 1 balancing of the meshes
AMR in FLUSEPA Octree method Execution flow Hangingnodes
0
2-1 Balancing
High-order redistribution Problem description Validation
Load balancing
1
0
1 Ni 1
0
Ni 0
0
2
1
1 1
Industrial test cases Martel bench Simplified after body flow
(a) Balanced unstructured grid before refinement. In red: the cell having to be refined. In green: the level of refinement of the cells in the valence ball of Ni
(b) Balanced mesh after refinement
Conclusion
Figure: Example of a mesh adaptation with 2 − 1 per node balancing rule for the node Ni
Intra-mesh cell size progression constraint, what about the intersections ? 19 / 46
Alexandre Limare AMR in FLUSEPA Octree method Execution flow Hangingnodes
Cell size progression constraint Adaptation sensor • minimum cell size in the 1st
neighborhood U
> local grid resolution adapted in the intersection area
2-1 Balancing
High-order redistribution
MOVIE
Problem description
Figure: Advected turbulent structure
Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Conclusion
(a) 3D view
(b) 2D view
Figure: Grids colored by priority levels 20 / 46
Alexandre Limare AMR in FLUSEPA Octree method Execution flow Hangingnodes
Cell size progression constraint Adaptation sensor • minimum cell size in the 1st
neighborhood U
> local grid resolution adapted in the intersection area
2-1 Balancing
High-order redistribution Problem description Validation
Load balancing
MOVIE too expansive for moving grids ! → compatibility must only be enforced when needed
Figure: Advected turbulent structure
Industrial test cases Martel bench Simplified after body flow
Conclusion
(a) 3D view
(b) 2D view
Figure: Grids colored by priority levels 20 / 46
Alexandre Limare
Outline
AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description Validation
1 AMR in FLUSEPA
Octree method Execution flow Hanging-nodes 2-1 Balancing 2 High-order redistribution
Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
3 Load balancing 4 Industrial test cases
Martel bench Simplified after body flow
Conclusion
5 Conclusion
21 / 46
Alexandre Limare
High-order redistribution
AMR in FLUSEPA Octree method
q
q
q
Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description
xi
Validation
Load balancing
(a) Piecewise constant (1rst-order)
xi (b) Piecewise linear (2nd-order)
xi (c) Piecewise parabolic (3rd-order)
Industrial test cases Martel bench Simplified after body flow
Conclusion
Figure: 1D-mesh, influence of the order of reconstruction on a given reference function
In 1D: a) q = a b) q = a + bX c) q = a + bY + cX 2
21 / 46
Alexandre Limare
Influence of the order of the redistribution
AMR in FLUSEPA Octree method
q
q
q
Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description
xi
Validation
Load balancing
(a) Piecewise constant (1rst order)
xi (b) Piecewise linear (2nd-order)
xi (c) Piecewise parabolic (3rd-order)
Industrial test cases Martel bench Simplified after body flow
Conclusion
Figure: Example of a consistent redistribution method for the variables after a cell division
In 3D: a) q = a b) q = a + bX + cY + dZ c) q = a + bY + cY + dZ + eX 2 + fY 2 + gZ 2 + hXY + iXZ + jYZ
22 / 46
Validation 1rst step: Cartesian grids
Alexandre Limare AMR in FLUSEPA Octree method Execution flow Hangingnodes
Taylor expansion gives a truncation error : ε = Chn + D∆t p
2-1 Balancing
High-order redistribution Problem description Validation
Load balancing
with D∆t p Chn All grids are cartesian with a fixed space step ∆x = ∆y = h. Therefore: log ε = n log h + C , where C is a constant
Industrial test cases Martel bench Simplified after body flow
All cases are run with a 3rd-order spatial accurate scheme. ⇒ only the order of the redistribution of the variables varies
Conclusion
23 / 46
Alexandre Limare
Steady vortex
AMR in FLUSEPA Octree method
Adaptation sensor : pressure gradient
Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Conclusion
(a) Pressure field after refinement
(b) Mesh obtained after refinement
24 / 46
Alexandre Limare
Steady vortex
AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description
(c) Mesh 1, first-order (d) Mesh 1, third-order redistribution projection
Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Conclusion
(e) Mesh 3, first-order (f) Mesh 3, third-order projection projection
Figure: First derivative of the velocity, Frœbenius norm
Figure: L2 -error as a function of the space discretization step
25 / 46
Alexandre Limare
Advected vortex + Intersection
AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description Validation
Load balancing
Figure: Initial field of the temperature and zoom of the intersection zone
Industrial test cases Martel bench Simplified after body flow
Conclusion
∆x = ∆y 0.2142 0.1500 0.1153
Points per vortex 11 15 20
Figure: L2 −error as a function of the space discretization step, mesh without intersection
Comparison the first derivative of the velocity of the 2nd background grid
26 / 46
Alexandre Limare
Outline
AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description Validation
1 AMR in FLUSEPA
Octree method Execution flow Hanging-nodes 2-1 Balancing 2 High-order redistribution
Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
3 Load balancing 4 Industrial test cases
Martel bench Simplified after body flow
Conclusion
5 Conclusion
27 / 46
Alexandre Limare
MPI partitioning / Load balancing
AMR in FLUSEPA Octree method
Shared & Distributed memory architecture
Execution flow
slow network
Hangingnodes 2-1 Balancing
High-order redistribution
Shared memory
Shared memory
Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Figure: Initial mesh
Conclusion
How to have a proper spatial partitioning of the mesh according to the hardware used for the calculation and have a nice load balancing ?
27 / 46
Alexandre Limare
MPI partitioning / Load balancing
AMR in FLUSEPA Octree method
Shared & Distributed memory architecture
Execution flow
slow network
Hangingnodes 2-1 Balancing
High-order redistribution
Shared memory
Shared memory
Problem description Validation
Load balancing
calculation domain
Industrial test cases Martel bench Simplified after body flow
Figure: Initial mesh
Conclusion
How to have a proper spatial partitioning of the mesh according to the hardware used for the calculation and have a nice load balancing ?
27 / 46
Alexandre Limare
MPI partitioning / Load balancing
AMR in FLUSEPA Octree method
Shared & Distributed memory architecture
Execution flow
slow network
Hangingnodes 2-1 Balancing
High-order redistribution
Shared memory
Shared memory
Problem description Validation
Load balancing
calculation domain
Industrial test cases Martel bench Simplified after body flow
Figure: Initial mesh
Conclusion
How to have a proper spatial partitioning of the mesh according to the hardware used for the calculation and have a nice load balancing ?
27 / 46
Alexandre Limare AMR in FLUSEPA
MPI partitioning / Load balancing Meshes are moving & modified by the AMR
Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Conclusion
28 / 46
Alexandre Limare AMR in FLUSEPA
MPI partitioning / Load balancing Meshes are moving & modified by the AMR
Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Conclusion
When AMR is active, partitioning is done on the (reconstructed) initial mesh with weighted cost per cells due to AMR :
X
C (Ω) =
|{z}
C( Ωi ) ,
Ωi ⊂Ω / |Ωi |6=0
original cell
|
{z
}
non covered daughter cells
where : C (Ωi ) ' 2Li , with : Li the temporal level of Ωi . No weight is associated with the faces
28 / 46
Alexandre Limare
MPI partitioning / Load balancing
AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Conclusion
(e) assembled cells
(f) cells with the same mother
Some cells need to be on the same calculation unit
2 rules (often conflicting): • assembled cells are on the same slave • cells created by AMR with the same root cell are on the same slave
⇒ iterative algorithm
29 / 46
Alexandre Limare AMR in FLUSEPA
MPI partitioning with an adaptive explicit solver / Load balancing
Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description Validation
(g) Cost per cell on the initial reconstructed mesh
Load balancing Industrial test cases Martel bench Simplified after body flow
Conclusion
(h) AMR mesh colored by slave number
Figure: Mesh partitioning with regards to the adaptive explicit solver 30 / 46
Alexandre Limare
Time spent per task
AMR in FLUSEPA
Idle time
Octree method
MPI message Topology update Load balancing
Execution flow
AMR & Intersection
Hangingnodes 2-1 Balancing
High-order redistribution Problem description
Cells to be refined/coarsened
SOLVER
Full new topology Local indexing
Industrial test cases
Slave
H-O Redist.
Slave SOLVER
H-O Redist.
Validation
Load balancing
Master
Slave SOLVER
H-O Redist.
AMR sensor
Redistribution of the variables of the modified cells
Slave
Martel bench Simplified after body flow
Time
Conclusion
Figure: Execution flow
31 / 46
Alexandre Limare
Time spent per task for one global iteration
AMR in FLUSEPA Octree method
solver 36%
Execution flow Hangingnodes 2-1 Balancing
High-order redistribution
comm MPI 51%
Problem description Validation
AMR kernel 8%
Load balancing
Sensor 0%
Industrial test cases
HOR 0%
Intersection 5%
Martel bench Simplified after body flow
Figure: Cost per task for one global iteration (Ignition Overpressure test case)
Conclusion
nb cells 1 × 107
σsolver 13%
cells created / iteration 1 × 106
coarsened 9.5 × 105
Table: Additional information (IOP) 32 / 46
Alexandre Limare
Outline
AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description Validation
1 AMR in FLUSEPA
Octree method Execution flow Hanging-nodes 2-1 Balancing 2 High-order redistribution
Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
3 Load balancing 4 Industrial test cases
Martel bench Simplified after body flow
Conclusion
5 Conclusion
33 / 46
Alexandre Limare
Martel bench
AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Figure: Picture of the martel bench
Conclusion
Figure: Detail of the initial meshes
hmin 4.3×10−3
cell init 1 × 106
max 8×106
equivalent resolution ∼ 1.7 × 108
Mach 4 33 / 46
Alexandre Limare
Martel bench
AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description Validation
Maximum cell number: 8×106 MOVIE
Load balancing Industrial test cases Martel bench Simplified after body flow
Conclusion
Figure: Meshes during calculation colored by pressure gradient levels, exponential scale
34 / 46
Alexandre Limare
Martel bench
AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description
Figure: Position of the probes
Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Conclusion
(a) Probe L3
(b) Probe 5D
(c) Probe 10D
Figure: Pressure as a function of time. Comparison between a FLUSEPA calculation with and without AMR 35 / 46
Alexandre Limare AMR in FLUSEPA Octree method Execution flow Hangingnodes
Simplified after body flow • Experimentally studied by Deprés et al. and Meliga et al. • Numerically studied by Pont • Re = 1.2 × 106 , turbulence modelling : DDES • ATAC project (Aerodynamique des Tuyeres et Arriere-Corps)
2-1 Balancing
High-order redistribution Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Conclusion
(a) Sketch of the axisymmetric backward facing step
(b) Initial mesh, 6 level of priority
Figure: Case setup
36 / 46
Alexandre Limare
Simplified after body flow
AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Figure: Adapted mesh
Conclusion
37 / 46
Alexandre Limare
Simplified after body flow
AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description Validation
Load balancing
MOVIE
Industrial test cases Martel bench Simplified after body flow
Conclusion
38 / 46
Alexandre Limare
Simplified after body flow
AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description Validation
Load balancing
MOVIE
Industrial test cases Martel bench Simplified after body flow
Conclusion
39 / 46
Alexandre Limare
Axisymmetric backward facing step
AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Conclusion
40 / 46
Alexandre Limare
Outline
AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description Validation
1 AMR in FLUSEPA
Octree method Execution flow Hanging-nodes 2-1 Balancing 2 High-order redistribution
Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
3 Load balancing 4 Industrial test cases
Martel bench Simplified after body flow
Conclusion
5 Conclusion
41 / 46
Alexandre Limare
Conclusion
AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution
Mesh adaptation so far... • Ready-to-use AMR module integrated in the FLUSEPA code, works with the
explicit temporal adaptative solver with Euler and Navier-Stokes simulations, and different turbulence models RANS, URANS, RANS/LES
Problem description
• Only full hexa meshes
Validation
• Moving meshes
Load balancing Industrial test cases
• High-order conservative redistribution of the variables • Distributed memory parallelism (MPI)
Martel bench Simplified after body flow
Octree-AMR is only the first part of a global mesh adaptation strategy !
Conclusion
41 / 46
Alexandre Limare
Perspectives
AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution
(a) k-d tree hexa refinement ([LH10])
Problem description
(b) 2D curvilinear meshes, treatment of curved walls [LH08]
Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
discontinuity bi-section interfaces
FLUID 1 FLUID 2
Conclusion
(c) Bi-section of a cell containing a strong discontinuity
(d) Coupling of a bi-section method with a quad-tree one
Figure: Future developments
42 / 46
Alexandre Limare
Toward a new mesh adaptation strategy for industrial studies
AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution
Capture of coherent structures
Problem description
• shocks
Validation
• shear layers
Load balancing
• vortices
Industrial test cases
• ...
Martel bench Simplified after body flow
Conclusion
⇒ copy and cloning of the mesh containing a well-defined coherent structure Figure: Calculation domain with created patches for wing vortices and the bow shock
43 / 46
Alexandre Limare
Execution flow
AMR in FLUSEPA
Idle time
Octree method
MPI message Topology update Load balancing
Execution flow
AMR & Intersection
Hangingnodes 2-1 Balancing
High-order redistribution Problem description
Cells to be refined/coarsened
SOLVER
Full new topology Local indexing
Industrial test cases
Slave
H-O Redist.
Slave SOLVER
H-O Redist.
Validation
Load balancing
Master
Slave SOLVER
H-O Redist.
AMR sensor
Redistribution of the variables of the modified cells
Slave
Martel bench Simplified after body flow
Time
Conclusion
Figure: Mesh adaptation algorithm
44 / 46
Alexandre Limare
Time spent per task
AMR in FLUSEPA Octree method
solver 36%
Execution flow Hangingnodes 2-1 Balancing
comms gains 48%
High-order redistribution Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Conclusion
comm MPI 3% Sensor 0%
HOR 0%
AMR kernel 0% Intersection gains 5% Intersection 0%
AMR gains 8%
Figure: Cost per task for one global iteration (Ignition Overpressure test case)
At each iteration : • Mesh is fully sent ⇒ differential topology could be send instead (95 % reduction) • Intersection calculation initialized at each iteration ⇒ Differential intersection, 90% reduction 45 / 46
Alexandre Limare AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution
Thank you for your attention
Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Conclusion
46 / 46
S. Deck. Alexandre Limare
Simulation numérique des charges latérales instationnaires sur des configurations de lanceur. PhD thesis, 2002. R. E. Harris and B. Williams.
AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description Validation
Load balancing
Ventus: An overset adaptive cartesian simulation framework for moving boundary problems. In 21st AIAA Computational Fluid Dynamics Conference, page 2866, 2013. J. C. Jouhaud. Méthode d’Adaptation de Maillages Structurés par Enrichissement. PhD thesis, University of Bordeaux I, 1997. T. Leicht and R. Hartmann. Anisotropic mesh refinement for discontinuous galerkin methods in two-dimensional aerodynamic flow simulations. International Journal for Numerical Methods in Fluids, 56(11):2111–2138, 2008. T. Leicht and R. Hartmann. Error estimation and anisotropic mesh refinement for 3d laminar aerodynamic flow simulations. Journal of Computational Physics, 229(19):7344–7360, 2010. P. Moreau, J. Labbe, F. Dupoirieux, and R. Borghi. Experimental and numerical study of a turbulent recirculation zone with combustion. In Turbulent Shear Flows 5, pages 337–346. Springer, 1987.
Industrial test cases Martel bench Simplified after body flow
Conclusion
46 / 46
Alexandre Limare
Hanging nodes
AMR in FLUSEPA Octree method Execution flow Hangingnodes
Definition (Hanging nodes) A hanging node is a node lying at the surface of the several elements and not referenced by the canonical structure element → node of all of these elements.
2-1 Balancing
High-order redistribution Problem description Validation
N6
N3 T1 N4
N5 Q1
T2 N2
Load balancing Industrial test cases Martel bench Simplified after body flow
Conclusion
N1
Figure: Example of a hanging node
No hanging nodes : ⇒ Modify the definition of the hexahedra, triangulation per face considering all possible cases
46 / 46
Alexandre Limare
Supersonic forward facing step
AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description
Mach 3 Initial cell number : 8.4 × 103 Final : 4 × 104
ρ 1.4
ρu 4.2
ρE 8.8
ρν 0
Validation
Load balancing
Table: Calculation input Figure: Schematic description of the case studied [Jou97]
Industrial test cases Martel bench Simplified after body flow
MOVIE
Conclusion
46 / 46
Alexandre Limare
Supersonic forward facing step
AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution
(a) Mesh at t = 3s (FLUSEPA)
(b) Harris and Williams simulation, five level of refinement [HW13]
(c) Pressure field at t = 3s (FLUSEPA)
(d) Jouhaud’s results [Jou97]
Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Conclusion
Figure: Comparison of different simulations of the same test case
→ Identical shock position in all simulations → Differences in the last reflection 46 / 46
Alexandre Limare
Backward facing step Calculation inputs:
AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
Similar to a after-body flow of a space launcher:
(a) Geometry description
High-order redistribution Problem description Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Conclusion
(b) Physical description of the flow and hybrid RANS/LES turbulence modelling
G3
G2 G1
10 h
G4
G5
(c) Mesh used for the calculation. In cyan: wall grids, in magenta: background grids, in green: the AMR grid
46 / 46
Alexandre Limare AMR in FLUSEPA Octree method
Backward facing step Re 4 × 104
Cell init 4 × 105
Useful part 2.3 ∼ 2.8 × 106
Pont 3.6 × 106
Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description
(d) U field and probes positions
Validation
Load balancing Industrial test cases Martel bench Simplified after body flow
Conclusion
(e) Urms
Figure: Statistical fields
MOVIE 46 / 46
Alexandre Limare
Backward facing step
AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing
High-order redistribution Problem description Validation
Figure: AMR results for
Load balancing
U U∞
as a function y at x = 0 (experience [MLDB87])
Industrial test cases Martel bench Simplified after body flow
Conclusion
Urms as a function of y at U∞ x = 0(experience [MLDB87]) Figure: AMR results
Urms as a function of y U∞ à x = 0 HRL models study, comparison with Moreau’s [MLDB87] and Deck’s [Dec02] results 46 / 46 Figure: Pont’s results of