Adaptive refinement of intersecting meshes, in a high ...

5 downloads 0 Views 23MB Size Report
Oct 27, 2017 - Ariane 6 conception (2012-2020). > more than 50% of the aero-budget in experimental studies. > 70% of the studies using large numerical ...
Alexandre Limare AMR in FLUSEPA Octree method Execution flow

Adaptive refinement of intersecting meshes, in a high-order Finite Volume framework, for aeropropulsion applications PhD Defense - Saclay

Hangingnodes 2-1 Balancing

High-order redistribution Problem description Validation

Load balancing Industrial test cases Martel bench

Alexandre Limare GAMMA3 - UTT - Inria ArianeGroup Supervisors Pierre Brenner - ArianeGroup Houman Borouchaki - UTT

27th October 2017

Simplified after body flow

Conclusion

1 / 46

Alexandre Limare

Context

AMR in FLUSEPA Octree method Execution flow

Aim: Understand and simulate complex physical phenomena more accurately

Hangingnodes 2-1 Balancing

High-order redistribution Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Ariane 5 design phase (1987-1996) > Stage separation − Reduced models > Take-off ignition overpressure wave ⇒ Aerothermal databases: wind tunnel tests

Ariane 6 conception (2012-2020) > more than 50% of the aero-budget in experimental studies > 70% of the studies using large numerical simulation tools

Conclusion

2 / 46

Alexandre Limare AMR in FLUSEPA Octree method Execution flow Hangingnodes

Context Fluid Mechanics for space launchers • Complex violent unsteady phenomena (aerodynamic efforts, turbulence,

chemistry, dispersed phases influence) • Flows about bodies in relative motion

2-1 Balancing

High-order redistribution Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Conclusion

Figure: Boosters separation (FLUSEPA-ArianeGroup) Figure: Missile stage separation, re-entry of the Huygens probe 3 / 46

Alexandre Limare AMR in FLUSEPA Octree method Execution flow

Context FLUSEPATM1 • 30 years old in-house CFD tool • Euler, Navier-Stokes, turbulence modelling (RANS, URANS, HRL)

Hangingnodes 2-1 Balancing

High-order redistribution Problem description Validation

Load balancing Industrial test cases

(a) A5 Ignition OverPressure wave at take-off

Martel bench Simplified after body flow

Conclusion

(b) Propeller simulation

Numerical formulation • Unstructured cell-centered 3D Finite-Volume (Godunov + k-exact reconstruction) • Adaptive explicit integration scheme • Overset grids by conservative geometric intersections > Input cells: hexa, prism, penta and tets 1

Registered trademark in France with number 134009261 4 / 46

Alexandre Limare AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution

Overset grids Mesh overlapping strategy > Simple part of bodies meshed independently > Large displacements, meshes adapted to attached boundary layers and bow shocks > Motion calculated by integration of the aerodynamic & propulsive loads for each body with (6-DoF model)

Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Conclusion

(c) Detail of the multi component mesh

(d) Assembled

Figure: Multi-level intersecting meshes

5 / 46

Alexandre Limare AMR in FLUSEPA

Overset grids Mesh intersections

Octree method

• Based on priority levels

Execution flow

• Geometric Intersection (no interpolations) > Intersection faces directly used for fluxes calculations

Hangingnodes 2-1 Balancing

• Residual cells are assembled (CFL)

High-order redistribution Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Conclusion

Figure: Multi-level intersecting meshes 6 / 46

Alexandre Limare AMR in FLUSEPA

Overset grids Mesh intersections

Octree method

• Based on priority levels

Execution flow

• Geometric Intersection (no interpolations) > Intersection faces directly used for fluxes calculations

Hangingnodes 2-1 Balancing

• Residual cells are assembled (CFL)

High-order redistribution Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Conclusion

Figure: Multi-level intersecting meshes 6 / 46

Alexandre Limare AMR in FLUSEPA

Overset grids Mesh intersections

Octree method

• Based on priority levels

Execution flow

• Geometric Intersection (no interpolations) > Intersection faces directly used for fluxes calculations

Hangingnodes 2-1 Balancing

• Residual cells are assembled (CFL)

High-order redistribution Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Conclusion

Figure: Multi-level intersecting meshes 6 / 46

Alexandre Limare AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description Validation

Adaptive explicit solver (Slave) Considered competitive compared to an implicit solver for unsteady simulations

Principles • compute each cell near its maximum physical time step (CFL condition) > maximum allowed time step computed for each cell > each cell j is sorted according to its time level L: 2L dtmin . ≤ dtj < 2L+1 dtmin and put into a time level > each time level and its set of cells has its own time step (2L dtmin ) • only fluxes interpolation for boundaries between temporal levels (still

conservative) • during one iteration, multiple sub-iterations

Load balancing

time

Industrial test cases Martel bench Simplified after body flow

2∆tmin ∆tmin

Conclusion

class 0

class 1 Figure: Initial state

7 / 46

Alexandre Limare AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description Validation

Adaptive explicit solver (Slave) Considered competitive compared to an implicit solver for unsteady simulations

Principles • compute each cell near its maximum physical time step (CFL condition) > maximum allowed time step computed for each cell > each cell j is sorted according to its time level L: 2L dtmin . ≤ dtj < 2L+1 dtmin and put into a time level > each time level and its set of cells has its own time step (2L dtmin ) • only fluxes interpolation for boundaries between temporal levels (still

conservative) • during one iteration, multiple sub-iterations

Load balancing

time

Industrial test cases Martel bench Simplified after body flow

2∆tmin ∆tmin

Conclusion

class 0

class 1

Figure: Sub-iteration 1

7 / 46

Alexandre Limare AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description Validation

Adaptive explicit solver (Slave) Considered competitive compared to an implicit solver for unsteady simulations

Principles • compute each cell near its maximum physical time step (CFL condition) > maximum allowed time step computed for each cell > each cell j is sorted according to its time level L: 2L dtmin . ≤ dtj < 2L+1 dtmin and put into a time level > each time level and its set of cells has its own time step (2L dtmin ) • only fluxes interpolation for boundaries between temporal levels (still

conservative) • during one iteration, multiple sub-iterations → cost per cell varies

Load balancing

time

Industrial test cases Martel bench Simplified after body flow

2∆tmin ∆tmin

Conclusion

class 0

class 1

Figure: Sub-iteration 2

7 / 46

Alexandre Limare AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution

Industrial goals

Why AMR ? Reliable: • Work within the validity domain of each physical model • Limit error-prone user interventions • Capture the phenomena involved and needed

Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Conclusion

Efficient: • Automate time-consuming tasks (input generation) • Integrate clever error-correcting systems • High Performance Computing properties

Sustainable: • Retrocompatibility with available modules • Hardware independent

8 / 46

Alexandre Limare

Outline

AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description Validation

1 AMR in FLUSEPA

Octree method Execution flow Hanging-nodes 2-1 Balancing 2 High-order redistribution

Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

3 Load balancing 4 Industrial test cases

Martel bench Simplified after body flow

Conclusion

5 Conclusion

9 / 46

Alexandre Limare

Outline

AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description Validation

1 AMR in FLUSEPA

Octree method Execution flow Hanging-nodes 2-1 Balancing 2 High-order redistribution

Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

3 Load balancing 4 Industrial test cases

Martel bench Simplified after body flow

Conclusion

5 Conclusion

10 / 46

Alexandre Limare

Octree method

AMR in FLUSEPA Octree method Execution flow

⇒Octree-AMR : Large error correction and simple implementation

Hangingnodes 2-1 Balancing

High-order redistribution Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Full hexa meshes Adaptation method chosen: Octree per cell with a 2 − 1 per node balancing:

Advantages • simple algorithmic method • FLUSEPA’s solver allows general

polyhedra • low memory overhead due to AMR

structures (15 %) Figure: Octree method

Conclusion

10 / 46

Alexandre Limare AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution

Octree Requirements : • High-order, conservative, redistribution method of the conservative variables • Adaptation of the intersection algorithms • Geometric properties properly calculated • Treatment of "non-conform" zones • Load balancing

Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Conclusion

Figure: Non-conform zone

11 / 46

Alexandre Limare

Structural modification of FLUSEPA

AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description

Turbulence modeling

Eulerian dispersed phases

Chemical reactions modeling

Liquid-gaz interfaces

K-exact reconstruction

Adaptive explicit

Flux calculation

Implicit

Source/coupling terms calculation

Input/Output Topology/geometry Temporel integration Spatial discretisation Physical modeling Reduced models Mesh adaptation

• • • • • • •

Sub-domain partitioning

Input/Output

Mesh displacement/deformation

Validation

Load balancing

Mesh adaptation sensors

Mesh deformation

Industrial test cases

AMR refinement/coarsening

Coherent flow structure capturing

Martel bench Simplified after body flow

Fusion of too covered cells

Engine pilot Mechanical bonds Structure kinematic

Radiation • Structures • Fluid/Particules

Split + projection of the variables

Structure deformation

Structure thermal response

Geometric characteristics

Conclusion

Figure: Schematic view of FLUSEPA’s modules 12 / 46

Alexandre Limare

Structural modification of FLUSEPA

AMR in FLUSEPA Octree method

Turbulence modeling

Eulerian dispersed phases

Execution flow

Chemical reactions modeling

Liquid-gaz interfaces

• • • • • • •

Hangingnodes 2-1 Balancing

High-order redistribution

K-exact reconstruction

Adaptive explicit

Flux calculation

Implicit

Source/coupling terms calculation

Input/Output Topology/geometry Temporel integration Spatial discretisation Physical modeling Reduced models Mesh adaptation

SLAVE MASTER

Input/Output

Problem description Validation

Mesh adaptation sensors

Sub-domain partitioning

Load balancing

Mesh displacement/deformation

Industrial test cases

Mesh deformation

Martel bench Simplified after body flow

Conclusion

AMR refinement/coarsening

Coherent flow structure capturing

Fusion of too covered cells

Engine pilot Mechanical bonds Structure kinematic

Radiation • Structures • Fluid/Particules

Split + projection of the variables

Structure deformation

Structure thermal response

Geometric characteristics

12 / 46

Alexandre Limare

Execution flow

AMR in FLUSEPA Octree method Execution flow

Master

Hangingnodes 2-1 Balancing

High-order redistribution Problem description

Slave SOLVER Slave SOLVER

Validation

Load balancing

Slave SOLVER

Industrial test cases

Slave

Martel bench Simplified after body flow

Time

Conclusion

Figure: Mesh adaptation algorithm

13 / 46

Alexandre Limare

Execution flow

AMR in FLUSEPA Octree method

MPI message

Execution flow

Master

Hangingnodes 2-1 Balancing

High-order redistribution Problem description

Slave SOLVER Slave SOLVER

Validation

Load balancing Industrial test cases

Slave SOLVER AMR sensor Slave

Martel bench Simplified after body flow

Time

Conclusion

Figure: Mesh adaptation algorithm

13 / 46

Alexandre Limare

Execution flow

AMR in FLUSEPA Octree method

MPI message

Execution flow

Master

Hangingnodes 2-1 Balancing

High-order redistribution Problem description

Cells to be refined/coarsened

Slave SOLVER Slave SOLVER

Validation

Load balancing Industrial test cases

Slave SOLVER AMR sensor Slave

Martel bench Simplified after body flow

Time

Conclusion

Figure: Mesh adaptation algorithm

13 / 46

Alexandre Limare

Execution flow

AMR in FLUSEPA Octree method

MPI message Topology update

Execution flow

AMR & Intersection

Hangingnodes 2-1 Balancing

High-order redistribution Problem description

Cells to be refined/coarsened

Slave SOLVER Slave SOLVER

Validation

Load balancing Industrial test cases

Master

Slave SOLVER AMR sensor Slave

Martel bench Simplified after body flow

Time

Conclusion

Figure: Mesh adaptation algorithm

13 / 46

Alexandre Limare

Execution flow

AMR in FLUSEPA Octree method

MPI message Topology update

Execution flow

AMR & Intersection

Hangingnodes 2-1 Balancing

High-order redistribution Problem description

Cells to be refined/coarsened

Differential topology

Slave SOLVER Slave SOLVER

Validation

Load balancing Industrial test cases

Master

Slave SOLVER AMR sensor Slave

Martel bench Simplified after body flow

Time

Conclusion

Figure: Mesh adaptation algorithm

13 / 46

Alexandre Limare

Execution flow

AMR in FLUSEPA Octree method

MPI message Topology update

Execution flow

AMR & Intersection

Hangingnodes 2-1 Balancing

High-order redistribution Problem description

Master

Cells to be refined/coarsened

Differential topology

Slave SOLVER

H-O Redist.

Slave SOLVER

H-O Redist.

Validation

Load balancing Industrial test cases

Slave SOLVER

H-O Redist.

AMR sensor

Redistribution of the variables of the modified cells

Slave

Martel bench Simplified after body flow

Time

Conclusion

Figure: Mesh adaptation algorithm

13 / 46

Alexandre Limare

Execution flow

AMR in FLUSEPA Octree method

MPI message Topology update Load balancing

Execution flow

AMR & Intersection

Hangingnodes 2-1 Balancing

High-order redistribution Problem description

Cells to be refined/coarsened

Differential topology

Slave SOLVER

H-O Redist.

Slave SOLVER

H-O Redist.

Validation

Load balancing Industrial test cases

Master

Slave SOLVER

H-O Redist.

AMR sensor

Redistribution of the variables of the modified cells

Slave

Martel bench Simplified after body flow

Time

Conclusion

Figure: Mesh adaptation algorithm

13 / 46

Alexandre Limare

Execution flow

AMR in FLUSEPA Octree method

MPI message Topology update Load balancing

Execution flow

AMR & Intersection

Hangingnodes 2-1 Balancing

High-order redistribution Problem description

Cells to be refined/coarsened

SOLVER

Full new topology Local indexing

Industrial test cases

Slave

H-O Redist.

Slave SOLVER

H-O Redist.

Validation

Load balancing

Master

Slave SOLVER

H-O Redist.

AMR sensor

Redistribution of the variables of the modified cells

Slave

Martel bench Simplified after body flow

Time

Conclusion

Figure: Mesh adaptation algorithm

13 / 46

Alexandre Limare

Execution flow

AMR in FLUSEPA

Idle time

Octree method

MPI message Topology update Load balancing

Execution flow

AMR & Intersection

Hangingnodes 2-1 Balancing

High-order redistribution Problem description

Cells to be refined/coarsened

SOLVER

Full new topology Local indexing

Industrial test cases

Slave

H-O Redist.

Slave SOLVER

H-O Redist.

Validation

Load balancing

Master

Slave SOLVER

H-O Redist.

AMR sensor

Redistribution of the variables of the modified cells

Slave

Martel bench Simplified after body flow

Time

Conclusion

Figure: Mesh adaptation algorithm

13 / 46

Alexandre Limare

Octree

AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description Validation

Load balancing Industrial test cases

Master • Full topology available (even

the covered cells) • Full tree kept • Original mesh cannot be

coarsened Figure: Master tree

Martel bench Simplified after body flow

Conclusion

14 / 46

Alexandre Limare

Octree

AMR in FLUSEPA

P

Octree method

P

Execution flow



Hangingnodes 2-1 Balancing

High-order redistribution

(a) Tα to be refined

Problem description

Tα,4

Tα,3

Tα,1

Tα,2

(b) Tα,i after refinement Tα,3 is totally covered

Validation

Load balancing

Figure: Refinement of a partially covered cell Tα , covering polyhedron P, shaded zone

Industrial test cases Martel bench Simplified after body flow

Conclusion

Slave • Partial topology (covered

cells are removed) • Differential modification • No tree kept Figure: Temporary slave tree 15 / 46

Alexandre Limare AMR in FLUSEPA Octree method

The original solver of FLUSEPA relies on a general description of the polyhedra for the geometric intersections, the treatment of hanging nodes implies few new developments

Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description Validation

Ω1 12

3

4

5

6

7

M2

Load balancing Industrial test cases Martel bench

Figure: Ω1 cell cut by a higher priority grid M2 , local numbering of the obtained faces

Simplified after body flow

Conclusion

16 / 46

Alexandre Limare

Description of the polyhedra per cell

AMR in FLUSEPA Octree method Execution flow Hangingnodes

Intersections calculation rely on a definition of conform closed surfaces > Each cell is represented as a closed surface It is conform and AMR use must ensure that each surface stays conform

2-1 Balancing

High-order redistribution Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Conclusion

Figure: Basic hexahedron

17 / 46

Alexandre Limare

Description of the polyhedra per cell

AMR in FLUSEPA Octree method Execution flow Hangingnodes

Intersections calculation relies on a definition of conform closed surfaces > Each cell is represented as a closed surface It is conform and AMR use must ensure that each surface stays conform

2-1 Balancing

High-order redistribution Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Conclusion

Figure: Basic hexahedron originally treated as a dodecahedron

17 / 46

Alexandre Limare

Description of the polyhedra per cell

AMR in FLUSEPA Octree method Execution flow Hangingnodes

Intersections calculation relies on a definition of conform closed surfaces > Each cell is represented as a closed surface It is conform and AMR use must ensure that each surface stays conform

2-1 Balancing

High-order redistribution Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Conclusion

Figure: Dodecahedron definition is not sufficient for varying level of refinement

17 / 46

Alexandre Limare

Description of the polyhedra per cell

AMR in FLUSEPA Octree method Execution flow Hangingnodes

Intersections calculation relies on a definition of conform closed surfaces > Each cell is represented as a closed surface It is conform and AMR use must ensure that each surface stays conform

2-1 Balancing

High-order redistribution Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Conclusion

Figure: New face definition

17 / 46

Alexandre Limare

Triangulation per face

AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Conclusion

Figure: Full library of the possible faces with regards to the existing hanging nodes

divided faces nbpolyhedra

0 4096

1 1536

2 480

3 480

4 30

5 6

6 1

Table: Number of possible polyhedra considering the number of type "2" faces

total : 6629 cases per hexahedron → Mesh is constrained so as to limit the number of possible cases 18 / 46

Alexandre Limare

2 − 1 balancing of the meshes

AMR in FLUSEPA Octree method Execution flow Hangingnodes

0

2-1 Balancing

High-order redistribution Problem description Validation

Load balancing

0 Ni 1 0

0 0

Industrial test cases Martel bench Simplified after body flow

(a) Balanced unstructured grid before refinement. In red: the cell having to be refined. In green: the level of refinement of the cells in the valence ball of Ni

Conclusion

Figure: Example of a mesh adaptation with 2 − 1 per node balancing rule for the node Ni

Intra-mesh cell size progression constraint, what about the intersections ? 19 / 46

Alexandre Limare

2 − 1 balancing of the meshes

AMR in FLUSEPA Octree method Execution flow Hangingnodes

0

2-1 Balancing

High-order redistribution Problem description Validation

Load balancing

1

0

1 Ni 1

0

Ni 0

0

2

1

1 1

Industrial test cases Martel bench Simplified after body flow

(a) Balanced unstructured grid before refinement. In red: the cell having to be refined. In green: the level of refinement of the cells in the valence ball of Ni

(b) Balanced mesh after refinement

Conclusion

Figure: Example of a mesh adaptation with 2 − 1 per node balancing rule for the node Ni

Intra-mesh cell size progression constraint, what about the intersections ? 19 / 46

Alexandre Limare AMR in FLUSEPA Octree method Execution flow Hangingnodes

Cell size progression constraint Adaptation sensor • minimum cell size in the 1st

neighborhood U

> local grid resolution adapted in the intersection area

2-1 Balancing

High-order redistribution

MOVIE

Problem description

Figure: Advected turbulent structure

Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Conclusion

(a) 3D view

(b) 2D view

Figure: Grids colored by priority levels 20 / 46

Alexandre Limare AMR in FLUSEPA Octree method Execution flow Hangingnodes

Cell size progression constraint Adaptation sensor • minimum cell size in the 1st

neighborhood U

> local grid resolution adapted in the intersection area

2-1 Balancing

High-order redistribution Problem description Validation

Load balancing

MOVIE too expansive for moving grids ! → compatibility must only be enforced when needed

Figure: Advected turbulent structure

Industrial test cases Martel bench Simplified after body flow

Conclusion

(a) 3D view

(b) 2D view

Figure: Grids colored by priority levels 20 / 46

Alexandre Limare

Outline

AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description Validation

1 AMR in FLUSEPA

Octree method Execution flow Hanging-nodes 2-1 Balancing 2 High-order redistribution

Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

3 Load balancing 4 Industrial test cases

Martel bench Simplified after body flow

Conclusion

5 Conclusion

21 / 46

Alexandre Limare

High-order redistribution

AMR in FLUSEPA Octree method

q

q

q

Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description

xi

Validation

Load balancing

(a) Piecewise constant (1rst-order)

xi (b) Piecewise linear (2nd-order)

xi (c) Piecewise parabolic (3rd-order)

Industrial test cases Martel bench Simplified after body flow

Conclusion

Figure: 1D-mesh, influence of the order of reconstruction on a given reference function

In 1D: a) q = a b) q = a + bX c) q = a + bY + cX 2

21 / 46

Alexandre Limare

Influence of the order of the redistribution

AMR in FLUSEPA Octree method

q

q

q

Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description

xi

Validation

Load balancing

(a) Piecewise constant (1rst order)

xi (b) Piecewise linear (2nd-order)

xi (c) Piecewise parabolic (3rd-order)

Industrial test cases Martel bench Simplified after body flow

Conclusion

Figure: Example of a consistent redistribution method for the variables after a cell division

In 3D: a) q = a b) q = a + bX + cY + dZ c) q = a + bY + cY + dZ + eX 2 + fY 2 + gZ 2 + hXY + iXZ + jYZ

22 / 46

Validation 1rst step: Cartesian grids

Alexandre Limare AMR in FLUSEPA Octree method Execution flow Hangingnodes

Taylor expansion gives a truncation error : ε = Chn + D∆t p

2-1 Balancing

High-order redistribution Problem description Validation

Load balancing

with D∆t p  Chn All grids are cartesian with a fixed space step ∆x = ∆y = h. Therefore: log ε = n log h + C , where C is a constant

Industrial test cases Martel bench Simplified after body flow

All cases are run with a 3rd-order spatial accurate scheme. ⇒ only the order of the redistribution of the variables varies

Conclusion

23 / 46

Alexandre Limare

Steady vortex

AMR in FLUSEPA Octree method

Adaptation sensor : pressure gradient

Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Conclusion

(a) Pressure field after refinement

(b) Mesh obtained after refinement

24 / 46

Alexandre Limare

Steady vortex

AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description

(c) Mesh 1, first-order (d) Mesh 1, third-order redistribution projection

Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Conclusion

(e) Mesh 3, first-order (f) Mesh 3, third-order projection projection

Figure: First derivative of the velocity, Frœbenius norm

Figure: L2 -error as a function of the space discretization step

25 / 46

Alexandre Limare

Advected vortex + Intersection

AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description Validation

Load balancing

Figure: Initial field of the temperature and zoom of the intersection zone

Industrial test cases Martel bench Simplified after body flow

Conclusion

∆x = ∆y 0.2142 0.1500 0.1153

Points per vortex 11 15 20

Figure: L2 −error as a function of the space discretization step, mesh without intersection

Comparison the first derivative of the velocity of the 2nd background grid

26 / 46

Alexandre Limare

Outline

AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description Validation

1 AMR in FLUSEPA

Octree method Execution flow Hanging-nodes 2-1 Balancing 2 High-order redistribution

Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

3 Load balancing 4 Industrial test cases

Martel bench Simplified after body flow

Conclusion

5 Conclusion

27 / 46

Alexandre Limare

MPI partitioning / Load balancing

AMR in FLUSEPA Octree method

Shared & Distributed memory architecture

Execution flow

slow network

Hangingnodes 2-1 Balancing

High-order redistribution

Shared memory

Shared memory

Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Figure: Initial mesh

Conclusion

How to have a proper spatial partitioning of the mesh according to the hardware used for the calculation and have a nice load balancing ?

27 / 46

Alexandre Limare

MPI partitioning / Load balancing

AMR in FLUSEPA Octree method

Shared & Distributed memory architecture

Execution flow

slow network

Hangingnodes 2-1 Balancing

High-order redistribution

Shared memory

Shared memory

Problem description Validation

Load balancing

calculation domain

Industrial test cases Martel bench Simplified after body flow

Figure: Initial mesh

Conclusion

How to have a proper spatial partitioning of the mesh according to the hardware used for the calculation and have a nice load balancing ?

27 / 46

Alexandre Limare

MPI partitioning / Load balancing

AMR in FLUSEPA Octree method

Shared & Distributed memory architecture

Execution flow

slow network

Hangingnodes 2-1 Balancing

High-order redistribution

Shared memory

Shared memory

Problem description Validation

Load balancing

calculation domain

Industrial test cases Martel bench Simplified after body flow

Figure: Initial mesh

Conclusion

How to have a proper spatial partitioning of the mesh according to the hardware used for the calculation and have a nice load balancing ?

27 / 46

Alexandre Limare AMR in FLUSEPA

MPI partitioning / Load balancing Meshes are moving & modified by the AMR

Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Conclusion

28 / 46

Alexandre Limare AMR in FLUSEPA

MPI partitioning / Load balancing Meshes are moving & modified by the AMR

Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Conclusion

When AMR is active, partitioning is done on the (reconstructed) initial mesh with weighted cost per cells due to AMR :

X

C (Ω) =

|{z}

C( Ωi ) ,

Ωi ⊂Ω / |Ωi |6=0

original cell

|

{z

}

non covered daughter cells

where : C (Ωi ) ' 2Li , with : Li the temporal level of Ωi . No weight is associated with the faces

28 / 46

Alexandre Limare

MPI partitioning / Load balancing

AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Conclusion

(e) assembled cells

(f) cells with the same mother

Some cells need to be on the same calculation unit

2 rules (often conflicting): • assembled cells are on the same slave • cells created by AMR with the same root cell are on the same slave

⇒ iterative algorithm

29 / 46

Alexandre Limare AMR in FLUSEPA

MPI partitioning with an adaptive explicit solver / Load balancing

Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description Validation

(g) Cost per cell on the initial reconstructed mesh

Load balancing Industrial test cases Martel bench Simplified after body flow

Conclusion

(h) AMR mesh colored by slave number

Figure: Mesh partitioning with regards to the adaptive explicit solver 30 / 46

Alexandre Limare

Time spent per task

AMR in FLUSEPA

Idle time

Octree method

MPI message Topology update Load balancing

Execution flow

AMR & Intersection

Hangingnodes 2-1 Balancing

High-order redistribution Problem description

Cells to be refined/coarsened

SOLVER

Full new topology Local indexing

Industrial test cases

Slave

H-O Redist.

Slave SOLVER

H-O Redist.

Validation

Load balancing

Master

Slave SOLVER

H-O Redist.

AMR sensor

Redistribution of the variables of the modified cells

Slave

Martel bench Simplified after body flow

Time

Conclusion

Figure: Execution flow

31 / 46

Alexandre Limare

Time spent per task for one global iteration

AMR in FLUSEPA Octree method

solver 36%

Execution flow Hangingnodes 2-1 Balancing

High-order redistribution

comm MPI 51%

Problem description Validation

AMR kernel 8%

Load balancing

Sensor 0%

Industrial test cases

HOR 0%

Intersection 5%

Martel bench Simplified after body flow

Figure: Cost per task for one global iteration (Ignition Overpressure test case)

Conclusion

nb cells 1 × 107

σsolver 13%

cells created / iteration 1 × 106

coarsened 9.5 × 105

Table: Additional information (IOP) 32 / 46

Alexandre Limare

Outline

AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description Validation

1 AMR in FLUSEPA

Octree method Execution flow Hanging-nodes 2-1 Balancing 2 High-order redistribution

Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

3 Load balancing 4 Industrial test cases

Martel bench Simplified after body flow

Conclusion

5 Conclusion

33 / 46

Alexandre Limare

Martel bench

AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Figure: Picture of the martel bench

Conclusion

Figure: Detail of the initial meshes

hmin 4.3×10−3

cell init 1 × 106

max 8×106

equivalent resolution ∼ 1.7 × 108

Mach 4 33 / 46

Alexandre Limare

Martel bench

AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description Validation

Maximum cell number: 8×106 MOVIE

Load balancing Industrial test cases Martel bench Simplified after body flow

Conclusion

Figure: Meshes during calculation colored by pressure gradient levels, exponential scale

34 / 46

Alexandre Limare

Martel bench

AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description

Figure: Position of the probes

Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Conclusion

(a) Probe L3

(b) Probe 5D

(c) Probe 10D

Figure: Pressure as a function of time. Comparison between a FLUSEPA calculation with and without AMR 35 / 46

Alexandre Limare AMR in FLUSEPA Octree method Execution flow Hangingnodes

Simplified after body flow • Experimentally studied by Deprés et al. and Meliga et al. • Numerically studied by Pont • Re = 1.2 × 106 , turbulence modelling : DDES • ATAC project (Aerodynamique des Tuyeres et Arriere-Corps)

2-1 Balancing

High-order redistribution Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Conclusion

(a) Sketch of the axisymmetric backward facing step

(b) Initial mesh, 6 level of priority

Figure: Case setup

36 / 46

Alexandre Limare

Simplified after body flow

AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Figure: Adapted mesh

Conclusion

37 / 46

Alexandre Limare

Simplified after body flow

AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description Validation

Load balancing

MOVIE

Industrial test cases Martel bench Simplified after body flow

Conclusion

38 / 46

Alexandre Limare

Simplified after body flow

AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description Validation

Load balancing

MOVIE

Industrial test cases Martel bench Simplified after body flow

Conclusion

39 / 46

Alexandre Limare

Axisymmetric backward facing step

AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Conclusion

40 / 46

Alexandre Limare

Outline

AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description Validation

1 AMR in FLUSEPA

Octree method Execution flow Hanging-nodes 2-1 Balancing 2 High-order redistribution

Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

3 Load balancing 4 Industrial test cases

Martel bench Simplified after body flow

Conclusion

5 Conclusion

41 / 46

Alexandre Limare

Conclusion

AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution

Mesh adaptation so far... • Ready-to-use AMR module integrated in the FLUSEPA code, works with the

explicit temporal adaptative solver with Euler and Navier-Stokes simulations, and different turbulence models RANS, URANS, RANS/LES

Problem description

• Only full hexa meshes

Validation

• Moving meshes

Load balancing Industrial test cases

• High-order conservative redistribution of the variables • Distributed memory parallelism (MPI)

Martel bench Simplified after body flow

Octree-AMR is only the first part of a global mesh adaptation strategy !

Conclusion

41 / 46

Alexandre Limare

Perspectives

AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution

(a) k-d tree hexa refinement ([LH10])

Problem description

(b) 2D curvilinear meshes, treatment of curved walls [LH08]

Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

discontinuity bi-section interfaces

FLUID 1 FLUID 2

Conclusion

(c) Bi-section of a cell containing a strong discontinuity

(d) Coupling of a bi-section method with a quad-tree one

Figure: Future developments

42 / 46

Alexandre Limare

Toward a new mesh adaptation strategy for industrial studies

AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution

Capture of coherent structures

Problem description

• shocks

Validation

• shear layers

Load balancing

• vortices

Industrial test cases

• ...

Martel bench Simplified after body flow

Conclusion

⇒ copy and cloning of the mesh containing a well-defined coherent structure Figure: Calculation domain with created patches for wing vortices and the bow shock

43 / 46

Alexandre Limare

Execution flow

AMR in FLUSEPA

Idle time

Octree method

MPI message Topology update Load balancing

Execution flow

AMR & Intersection

Hangingnodes 2-1 Balancing

High-order redistribution Problem description

Cells to be refined/coarsened

SOLVER

Full new topology Local indexing

Industrial test cases

Slave

H-O Redist.

Slave SOLVER

H-O Redist.

Validation

Load balancing

Master

Slave SOLVER

H-O Redist.

AMR sensor

Redistribution of the variables of the modified cells

Slave

Martel bench Simplified after body flow

Time

Conclusion

Figure: Mesh adaptation algorithm

44 / 46

Alexandre Limare

Time spent per task

AMR in FLUSEPA Octree method

solver 36%

Execution flow Hangingnodes 2-1 Balancing

comms gains 48%

High-order redistribution Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Conclusion

comm MPI 3% Sensor 0%

HOR 0%

AMR kernel 0% Intersection gains 5% Intersection 0%

AMR gains 8%

Figure: Cost per task for one global iteration (Ignition Overpressure test case)

At each iteration : • Mesh is fully sent ⇒ differential topology could be send instead (95 % reduction) • Intersection calculation initialized at each iteration ⇒ Differential intersection, 90% reduction 45 / 46

Alexandre Limare AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution

Thank you for your attention

Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Conclusion

46 / 46

S. Deck. Alexandre Limare

Simulation numérique des charges latérales instationnaires sur des configurations de lanceur. PhD thesis, 2002. R. E. Harris and B. Williams.

AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description Validation

Load balancing

Ventus: An overset adaptive cartesian simulation framework for moving boundary problems. In 21st AIAA Computational Fluid Dynamics Conference, page 2866, 2013. J. C. Jouhaud. Méthode d’Adaptation de Maillages Structurés par Enrichissement. PhD thesis, University of Bordeaux I, 1997. T. Leicht and R. Hartmann. Anisotropic mesh refinement for discontinuous galerkin methods in two-dimensional aerodynamic flow simulations. International Journal for Numerical Methods in Fluids, 56(11):2111–2138, 2008. T. Leicht and R. Hartmann. Error estimation and anisotropic mesh refinement for 3d laminar aerodynamic flow simulations. Journal of Computational Physics, 229(19):7344–7360, 2010. P. Moreau, J. Labbe, F. Dupoirieux, and R. Borghi. Experimental and numerical study of a turbulent recirculation zone with combustion. In Turbulent Shear Flows 5, pages 337–346. Springer, 1987.

Industrial test cases Martel bench Simplified after body flow

Conclusion

46 / 46

Alexandre Limare

Hanging nodes

AMR in FLUSEPA Octree method Execution flow Hangingnodes

Definition (Hanging nodes) A hanging node is a node lying at the surface of the several elements and not referenced by the canonical structure element → node of all of these elements.

2-1 Balancing

High-order redistribution Problem description Validation

N6

N3 T1 N4

N5 Q1

T2 N2

Load balancing Industrial test cases Martel bench Simplified after body flow

Conclusion

N1

Figure: Example of a hanging node

No hanging nodes : ⇒ Modify the definition of the hexahedra, triangulation per face considering all possible cases

46 / 46

Alexandre Limare

Supersonic forward facing step

AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description

Mach 3 Initial cell number : 8.4 × 103 Final : 4 × 104

ρ 1.4

ρu 4.2

ρE 8.8

ρν 0

Validation

Load balancing

Table: Calculation input Figure: Schematic description of the case studied [Jou97]

Industrial test cases Martel bench Simplified after body flow

MOVIE

Conclusion

46 / 46

Alexandre Limare

Supersonic forward facing step

AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution

(a) Mesh at t = 3s (FLUSEPA)

(b) Harris and Williams simulation, five level of refinement [HW13]

(c) Pressure field at t = 3s (FLUSEPA)

(d) Jouhaud’s results [Jou97]

Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Conclusion

Figure: Comparison of different simulations of the same test case

→ Identical shock position in all simulations → Differences in the last reflection 46 / 46

Alexandre Limare

Backward facing step Calculation inputs:

AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

Similar to a after-body flow of a space launcher:

(a) Geometry description

High-order redistribution Problem description Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Conclusion

(b) Physical description of the flow and hybrid RANS/LES turbulence modelling

G3

G2 G1

10 h

G4

G5

(c) Mesh used for the calculation. In cyan: wall grids, in magenta: background grids, in green: the AMR grid

46 / 46

Alexandre Limare AMR in FLUSEPA Octree method

Backward facing step Re 4 × 104

Cell init 4 × 105

Useful part 2.3 ∼ 2.8 × 106

Pont 3.6 × 106

Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description

(d) U field and probes positions

Validation

Load balancing Industrial test cases Martel bench Simplified after body flow

Conclusion

(e) Urms

Figure: Statistical fields

MOVIE 46 / 46

Alexandre Limare

Backward facing step

AMR in FLUSEPA Octree method Execution flow Hangingnodes 2-1 Balancing

High-order redistribution Problem description Validation

Figure: AMR results for

Load balancing

U U∞

as a function y at x = 0 (experience [MLDB87])

Industrial test cases Martel bench Simplified after body flow

Conclusion

Urms as a function of y at U∞ x = 0(experience [MLDB87]) Figure: AMR results

Urms as a function of y U∞ à x = 0 HRL models study, comparison with Moreau’s [MLDB87] and Deck’s [Dec02] results 46 / 46 Figure: Pont’s results of