Using approximate secant equations in limited

Using approximate secant equations in limited memory methods for multilevel unconstrained optimization by S. Gratton1 , V. Malmedy2,3 and Ph. L. Toint3 Report 09/18

16 November 2009

1

ENSEEIHT-IRIT, 2, rue Camichel, 31000 Toulouse, France Email: [email protected] 2

Fund for Scientific Research, rue d’Egmont, Brussels, Belgium. Email: [email protected] 3

Department of Mathematics, FUNDP-University of Namur, 61, rue de Bruxelles, B-5000 Namur, Belgium. Email: [email protected]

Using approximate secant equations in limited memory methods for multilevel unconstrained optimization Serge Gratton

Vincent Malmedy

Philippe L. Toint

16 November 2009 Abstract The properties of multilevel optimization problems defined on a hierarchy of discretization grids can be used to define approximate secant equations, which describe the second-order behaviour of the objective function. Following earlier work by Gratton and Toint (2009), we introduce a quasi-Newton method (with a linesearch) and a nonlinear conjugate gradient method that both take advantage of this new second-order information. We then present numerical experiments with these methods and formulate recommendations for their practical use.

Keywords nonlinear optimization · multilevel problems · quasi-Newton methods · nonlinear conjugate gradient methods · limited-memory algorithms.

Mathematics Subject Classification (2000) 65K05 · 65K10 · 90C06 · 90C26 · 90C30 · 90C53.

1

Introduction

Many optimization problems in science and engineering exhibit a hierarchical structure, especially when they are derived from the discretization of underlying continuous applications on grids of varying size. Inspired in part by multigrid techniques in linear algebra, numerical methods for the efficient resolution of such problems have been considered by various authors (see Fisher, 1998, Nash, 2000, Oh, Milstein, Bouman and Webb, 2003, Gratton, Sartenaer and Toint, 2008b, Gratton, Mouffe, Toint and Weber-Mendon¸ca, 2008a, for instance). While globally efficient, many of the proposed methods still have difficulties in that estimating the local curvature of the considered objective function might be costly, especially on finer grids where the number of variables is large, even if the associated matrices often have a sparsity pattern reflecting the grid structure or are only considered in operator form. These difficulties have been addressed, for large but otherwise unstructured optimization problems, by iterative methods such as limited-memory quasi-Newton techniques (see for instance, the L-BFGS method of Liu and Nocedal, 1989), in which the Hessian matrix is assembled at each iteration as a product of a modest number of low-rank updates. More recently, Gratton and Toint (2009) have proposed an extension of this approach to problems with multilevel grid structure, where the hierarchy of grids is used to generate additional curvature information. This proposal however includes many variants and leaves the door open as to which of these is preferable in terms of numerical reliability and efficiency. It is the purpose of this paper to explore this issue and make specific algorithmic recommendations. The paper is organized as follows. A statement of the problem and more detailed review of the considered methods is proposed in Section 2, while algorithmic variants of these method are discussed in Section 3. Our numerical experiments are then presented in Section 4 and conclusions are finally drawn in Section 5. The detailed numerical results are reported in appendix for reference purposes.

1

Draft version (16 November 2009) — definitely not for circulation

2

2

The problem

We consider the minimization of a smooth nonlinear objective function f from IRn to IR: min f (x).

(2.1)

x∈IRn

Quasi-Newton methods solve this problem in an iterative way that involves the construction, around the current iterate xk ∈ IRn , of a second-order model of the objective function of the form mk (xk + s) = f (xk ) + hgk , si +

1 2

hs, Bk si ,

def

where gk = ∇x f (xk ), and Bk is a symmetric (often positive-definite) approximation of the Hessian matrix ∇xx f (xk ), capturing the information about the curvature of the objective function around xk . If positive-definiteness of the matrix Bk is maintained throughout the iterations, the search direction at iteration k is then computed as dk = −Bk−1 gk ,

(2.2)

and a linesearch is performed along this direction to ensure global convergence (see Dennis and Schnabel, 1983, pages 116-125). In this process, the new approximate Hessian matrix Bk+1 is typically updated from the previous one, such that the secant equation Bk+1 sk = yk

(2.3)

holds, where sk = xk+1 −xk and yk = gk+1 −gk . This condition arises from the mean value theorem for vector-valued functions, which implies that (2.3) is satisfied by the mean Hessian on the interval [xk , xk+1 ]. The pair (sk , yk ) is said to be the secant pair associated with equation (2.3). To avoid def

the cost of solving a linear system in (2.2), the inverse matrix Hk = Bk−1 is often recurred instead of Bk . The most famous updating process of this kind is the BFGS formula sk ykT yk sTk sk sT Hk+1 = I − T (2.4) Hk I − T + T k sk yk sk yk sk yk which was developed by Broyden (1970), Fletcher (1970), Goldfarb (1970), and Shanno (1970). It readily follows from this formula that Hk+1 remains positive definite if Hk is positive definite and sTk yk > 0,

(2.5)

a condition that one can always enforce in the linesearch procedure if the objective function is bounded below (again see Dennis and Schnabel, 1983, pages 120, 208). As indicated above, we are especially interested in the resolution of multilevel (unconstrained) optimization problems, that is problems that are defined at several levels of accuracy. We denote these levels with index i from the finest (i = r) to the coarsest (i = 0) descriptions. Prolongation operators Pi and restriction operators Ri are given to go from level i−1 to level i and vice versa. At their finest level, such problems often have a large number of variables, making the explicit storage of the (dense) matrices Bk or Hk impractical. Gratton and Toint (2009) introduce a limited-memory quasi-Newton method that takes advantage of the multilevel hierarchy, in that they construct new (approximate) secant pairs (Si sk , Si yk ) for 0 ≤ i < r, (2.6) def

where Si = Pr · · · Pi+1 Ri+1 · · · Rr . These are filtered versions of the secant pair (sk , yk ) whose oscillatory components were mainly removed, bringing hence to the algorithm a wider range of curvature information on the objective function, and helping it to reduce faster the smooth modes of the error. The numerical results show a significant decrease of the number of iterations and function evaluations when these smoothed pairs are used.


3

Nonlinear conjugate gradient methods constitute another set of iterative methods that can tackle large-scale problems with small memory requirements, and were introduced by Fletcher and Reeves (1964). In these methods, the search direction dk+1 is updated from the previous one as dk+1 = −Mk gk+1 + βk dk

(2.7)

where we may use a symmetric positive definite preconditioner Mk , and where βk is some conjugacy factor (see Hager and Zhang, 2006b, for a survey). A linesearch is then performed in this direction. A standard strategy to choose the preconditioner Mk is to take an approximation to ∇xx f (x∗ )−1 coming from a quasi-Newton formula. In particular, we may set Mk to our last approximation Hk+1 of the inverse Hessian, and use the smoothed pairs given in (2.6) to define a limited-memory BFGS preconditioner for the nonlinear conjugate gradient method. However, the combination of all these potentially useful strategies leads to a substantial number of algorithmic variants, whose numerical efficiency and robustness have not yet been analyzed. It is the purpose of this paper to present systematic numerical experiments involving these variants, that is the various ways in which smoothed secant pairs may be used inside either a quasi-Newton or a nonlinear conjugate gradient method.

3

Algorithmic choices

In order to make our experiments as systematic as possible, we now detail several aspects of the considered algorithmic variants, namely the choice of the conjugacy factor used to define the search direction (Section 3.1), the choice of the linesearch algorithm (Section 3.2), and the computation of the approximate inverse Hessian using the smoothed pairs (Section 3.3).

3.1

Conjugacy factor

We consider three choices for the conjugacy factor βk . The first one is βkQN = 0, corresponding to the quasi-Newton methods, since the search direction given by (2.7) is then the same as in (2.2) with Mk = Hk+1 . The next two choices are advised by Hager and Zhang (2005) as the best choices actually known. Hence, our second choice is T gk+1 y T Mk y k βkHZ = Mk yk − 2dk k T , dk y k dTk yk which was proposed in the same paper, and our third choice is an hybrid of the formulae of Hestenes and Stiefel (1952) and Dai and Yuan (1999): T yk Mk gk+1 gkT Mk gk+1 DY HS βk = max 0, min , , dTk yk dTk yk which was introduced by Dai and Yuan (2001).

3.2

Linesearch

We selected two linesearch algorithms for our tests. The first one was described by Dennis and Schnabel (1983), and used by Gratton and Toint (2009) for their numerical experiments. It determines a step length αk > 0 that ensures the satisfaction of the Wolfe conditions: f (xk + αk dk ) ≤ δαk gkT dk , T gk+1 dk ≥ σgkT dk .

(3.8) (3.9)

for some parameters 0 < δ ≤ σ < 1. We translated the matlab code of Gratton and Toint (2009) to Fortran 95, and kept the default parameters δ = 10−4 and σ = 0.9. The second linesearch algorithm


4

comes from the CG DESCENT code (version 3.0) of Hager and Zhang (2006a), which ensures the satisfaction either of the Wolfe conditions (3.8)–(3.9) or of the approximate Wolfe conditions: T σgkT dk ≤ gk+1 dk ≤ (2δ − 1)gkT dk ,

where 0 < δ ≤ σ < 1. We chose the variant of their algorithm and switch permanently to the approximate ones as soon as small, that is when |f (xk+1 ) − f (xk )| ≤ ωCk , where Qk = 1 + Qk−1 ∆, Ck = Ck−1 + (|f (xk )| − Ck−1 )/Qk ,

that uses first the Wolfe conditions the function variation is relatively Q−1 = 0, C−1 = 0,

for some parameters ∆ and ω in the interval [0, 1]. The practical implementation is a Fortran 95 translation of the CG DESCENT code written in C, and uses the default parameters δ = 0.1, σ = 0.9, ∆ = 0.7 and ω = 10−3 .

3.3

L-BFGS update

We assume that at most m pairs may be stored. So we consider, at each iteration, a set Pk containing mk ≤ m pairs (sk,j , yk,j ), for j = 1, . . . , mk , that provide the available curvature information on the objective function. The approximate inverse Hessian Hk+1 is then defined using the L-BFGS formula, that is Hk+1 = Hk,mk where T Hk,j−1 I − ρk,j yk,j sTk,j + ρk,j sk,j sTk,j Hk,j = I − ρk,j sk,j yk,j

with ρk,j = (sTk,j yk,j )−1 , for j = 1, . . . , mk , and some chosen initial matrix Hk,0 . As mentioned by many authors, the choice of this initialization matrix Hk,0 is quite important to get a well scaled search direction. We thus use the classical initialization Hk,0 = γk I with γk =

sTk yk . ykT yk

By the way, since we are only interested in the products Hk+1 gk+1 , we need not to explicitly construct the matrix Hk+1 , but may use the (matrix-free) L-BFGS two-loop recursion instead (see Nocedal, 1980). The set Pk may contain exact pairs of the form (sk , yk ) and smoothed pairs of the form (Si sk , Si yk ). Different variants were proposed by Gratton and Toint (2009) to select the pairs kept in memory: • the L-BFGS strategy only uses the m last exact secant pairs; • the Full strategy uses the m last generated pairs, making no difference between smoothed and exact pairs; • the Local strategy uses only the min(r, m − 1) smoothed pairs from the current iteration, in addition to past and current exact pairs; • the Mless (memory less) strategy uses only (smoothed and exact) pairs from the current iteration. Each exact pair is always integrated in the update after its smoothed versions. However, we still may choice the order of the smoothed pairs in the L-BFGS formula: • the Coarse first strategy first integrates the pairs that have been smoothed on the coarser levels (that is integrating pairs (Si sk , Si yk ) with index i running from 0 to r − 1); • the Fine first strategy first integrates the pairs that have been smoothed on the finer levels (that is integrating pairs (Si sk , Si yk ) with index i running from r − 1 to 0).

Draft version (16 November 2009) — definitely not for circulation Name DNT P2D P3D DEPT DODC MINS-SB MINS-OB MINS-DMSA IGNISC DSSC BRATU NCCS NCCO MOREBV

Level 8 8 5 8 8 8 8 8 8 8 8 7 7 8

Size 511 261121 250047 261121 261121 261121 261121 261121 261121 261121 261121 130050 130050 261121

Type 1-D, quadratic 2-D, quadratic 3-D, quadratic 2-D, quadratic 2-D, convex 2-D, convex 2-D, convex 2-D, convex 2-D, convex 2-D, convex 2-D, convex 2-D, nonconvex 2-D, nonconvex 2-D, nonconvex

5

Description Dirichlet to Neumann transfer Poisson model problem Poisson model problem Elastic-plastic torsion problem Optimal design with composite materials Minimal surface problem Minimal surface problem Minimal surface problem Combustion problem Combustion problem Combustion problem Optimal control problem Optimal control problem Boundary value problem

Table 1: Test problems set Gratton and Toint (2009) moreover propose to control the collinearity of the smoothed pairs with respect to the original exact pair; the pairs that do not satisfy the condition | hSi sk , si | ≤ τ kSi sk k2 ksk k2 ,

(3.10)

for some τ ∈ (0, 1], are thus discarded. Additionally, we enforce the positivity constraint (2.5) by ignoring pairs that do not satisfy | hSi sk , Si yk i | ≤ µ hsk , yk i , for parameter µ ∈ (0, 1). We test the values 0.999 and 1.0 for the threshold τ , and set µ to 10−6 .

4

Numerical experiments

We now present numerical experiments whose objective is to clarify which of the above algorithmic options provides the most reliable and efficient method for solving grid-structured unconstrained optimization problems. All codes used in these experiments are written in Fortran 95, and the runs were performed on a bi-processor Intel Xeon X5482 (4 cores, 3.20 GHz) with 64 GB of RAM.

4.1

Test problems

In our tests, we consider the unconstrained optimization problems from the set provided by Gratton, Mouffe, Sartenaer, Toint and Tomanos (2009). A small description of each problem is given in Table 1 with the level and size at which it was considered.

4.2

Starting point and stopping criterion

We choose the starting point as [x0 ]j = 0.5 and consider that the algorithm has converged as soon as kgk k∞ ≤ 10−5 .

4.3

Results

We ran a large number (78) of possible combinations of the variants described in Section 3 on our set of 14 test problems and report all results of the 1092 runs on comet-shape graphs representing a measure of the effort spent in function/gradient evaluations vs. iterations number. More precisely, we have first scaled, separately for each test problem, on the one hand, the number of function evaluations plus five times the number of gradient evaluations, and on the other hand, the iterations


6

number, by dividing them by the best obtained for this problem by all algorithmic variants. We then plotted the averages of these scaled measures on all test problems for each algorithmic variant separately, after removing the variants which fail on at least one problem (the number of such variants is given in legend). All variants use m = 9, but experiments with other choices (not reported here) give similar results. In the first of these plots (Figure 1), we have used squares for the variants where the quasiNewton search direction (2.2) is chosen, stars for the variants where the conjugate-gradient search direction (2.7) with the conjugacy factor βkDY HS is chosen, and triangles for the variants where the conjugate-gradient search direction (2.7) with the conjugacy factor βkHZ is chosen. 18

scaled number of function/gradient evaluations

16

14

12

10

8

6

4

2

0

Quasi−Newton (7/26 fail) NCG (Dai−Yuan) (7/26 fail) NCG (Hager−Zhang) (8/26 fail) 0

5

10

15

scaled number of iterations

Figure 1: Comet-shape graph comparing the choice of conjugacy factor We note a substantial spread of the results, with some options being up to 15 times worse than others. The worst cases (in the top right corner) correspond to combination of the HagerZhang linesearch, the Mless strategy and the collinearity threshold set to 0.999. Among the other variants, we may also observe a second set of variants which requires (in average) twice the number of function/gradient evaluations. These correspond to the use of the Hager-Zhang linesearch inside the quasi-Newton method. The third set of variants gathered results from the quasi-Newton method (with the Dennis-Schnabel linesearch) and nonlinear conjugate gradient method, with in average, better results for the former methods than for the latter. We next compare the effect of the linesearch choice in Figure 2. In that picture, squares have been used for the variants using the Hager-Zhang linesearch, and stars for the variants using the Dennis-Schnabel linesearch.


7

18


16

14

12

10

8

6

4

2 Hager−Zhang (11/39 fail) Dennis−Schnabel (11/39 fail) 0

0

5

10

15


Figure 2: Comet-shape graph comparing the choice of linesearch We observe that the Hager-Zhang linesearch do not improve the performance, especially inside the quasi-Newton method, where the number of function and gradient evaluations per iterations is doubled in average. We would therefore advise the Dennis-Schnabel linesearch since it is also simpler. Figure 3 compares the variants performance on the basis of the pairs selection strategy. This time, squares have been used for the variants using the L-BFGS strategy, stars for the Full strategy, triangles for the Local strategy, and crosses for the Mless strategy. 18


16

14

12

10

8

6

4 L−BFGS (4/6 fail) Full (8/24 fail) Local (8/24 fail) Mless (2/24 fail)

2

0

0

5

10

15


Figure 3: Comet-shape graph comparing the choice of pairs selection


8

We observe that the L-BFGS strategies are in a factor 4, in average, from the best ones. The Local and Mless strategies give best results. We recommend the Local strategy since the number of pairs used by the Mless strategy is limited by the number of levels, and thus prevent the full use of the available memory capacity. We then compare the effect of the integration order of the smoothed pairs inside the L-BFGS update. In Figure 4, we have thus used squares for the variants that integrate first the pairs smoothed at the coarser levels (Coarse First strategy), and stars for the variants that integrate first the pairs smoothed at the finer levels (Fine First strategy). 18


16

14

12

10

8

6

4

2 Coarse first (6/36 fail) Fine first (12/36 fail) 0

0

5

10

15


Figure 4: Comet-shape graph comparing the choice of pairs order It is unclear which strategy is the more efficient. Nevertheless, the variants using the Coarse First strategy encounter less failures. Finally, we consider the interest to control the collinearity of the smoothed pairs with respect to the exact secant pair from which they were generated. In Figure 5, we have thus used squares to represent the variants that do not control this collinearity (setting the threshold τ to 1.0), and stars to represent the variants that control this collinearity, with a threshold τ set to 0.999.


9

18


16

14

12

10

8

6

4

2

0

τ=1.000 (13/36 fail) τ=0.999 (5/36 fail) 0

5

10

15


Figure 5: Comet-shape graph comparing the choice of pairs collinearity control The more efficient variant is again unclear, but the collinearity control seems to improve the robustness. To summarize our conclusions so far, we may first attempt to distinguish the best variant not using smoothed secant pairs, and then compare the strategies using the smoothed pairs with this selected contender, and again look for the best of the set. These two steps are illustrated by Figures 6 and 7, where we have restricted ourselves to considering performance in terms of function and gradient evaluations (the figures for the number of iterations are similar). Both figures are performance profiles (see Dolan and Mor´e, 2002), a now standard technique to present such results. In these profiles, the proportion of test problems solved by each variant using a number of functions/gradients evaluation within a factor σ of the best performance is plotted against σ for σ ≥ 1. Thus the algorithmic variant whose curve is on top is considered best.


10

1

0.9

0.8

0.7

0.6

0.5

0.4

0.3 Quasi−Newton, Hager−Zhang linesearch Quasi−Newton, Dennis−Schnabel linesearch NCG (Dai−Yuan), Hager−Zhang linesearch NCG (Dai−Yuan), Dennis−Schnabel linesearch NCG (Hager−Zhang), Hager−Zhang linesearch NCG (Hager−Zhang), Dennis−Schnabel linesearch

0.2

0.1

0

1

2

3

4

5

6

7

Figure 6: Performance profile for the variants not using smoothed secant pairs (proportion of test problems as a function of σ) The first of these figures illustrates our findings well: the best method in our tests appears to be that using either the Dai-Yuan or the Hager-Zhang formula for deriving the search direction, coupled with the Hager-Zhang linesearch. Interestingly, if one restricts one’s attention to quasiNewton methods (βk = 0), then the Dennis-Schnabel linesearch seems to dominate Hager-Zhang’s by a substantial margin on our examples. 1

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2 Local (Coarse First, colin. ctrl.), Quasi−Newton, Dennis−Schnabel linesearch Local (Coarse First, colin. ctrl.), NCG (Dai−Yuan), Dennis−Schnabel linesearch Local (Coarse First, colin. ctrl.), NCG (Hager−Zhang), Dennis−Schnabel linesearch L−BFGS, NCG (Hager−Zhang), Hager−Zhang linesearch

0.1

0

1

2

3

4

5

6

7

Figure 7: Performance profile for the variants using local smoothed secant pairs against the best one not using them


11

The second figure also stresses our earlier conclusions: the use of smoothed secant pairs (in their “local” flavour, which we already selected as best above) is clearly beneficial when possible. Indeed, all variants using them substantially dominate the best variant which does not. Amongst them the best variant is the quasi-Newton method using local pairs, starting from the coarsest, together with collinearity control and Dennis-Schnabel linesearch. The gap between this variant and the second best (the CG variant using the Dai-Yuan formula) is significant, although not as wide as that separating the variants using smoothed secant pairs from their best contender.

5

Conclusions

We have considered several variants of the multilevel limited-memory quasi-Newton and conjugategradients algorithms for unconstrained optimization and have conducted numerical tests on a battery of examples arising from optimization in the context of partial differential optimzation and involving a hierarchy of grid-based discretizations. Based on these tests, we have singled out a specific limited-memory variant (using the Dennis-Schnabel linesearch technique, local secant pairs applied starting from the coarsest grid level and collinearity control) as the most efficient and reliable algorithm amongst those tested. Our examples also show further systematic interplay between linesearch techniques and search direction formulae whose detailed interpretation remains unclear at this stage. While we believe the presented conclusions supporting the use of smoothed secant pairs are valuable, we also realize that the full potential of multilevel limited-memory optimization method can only be asserted by continuous use in a wider range of applications than that on which the current tests were performed. Further work in this direction is therefore very desirable. The range of concerned scientific and engineering fields is wide, ranging from optimal control of systems governed by partial differential equations to data assimilation in weather prediction and oceanography, and interesting applications in these areas is expected in the future.

Acknowledgements The research of Philippe Toint for this paper was conducted with partial support from the European Science Fundation OPTPDE program and from the “Assimilation de Donn´ ees pour la Terre, l’Atmosph` ere et l’Oc´ ean (ADTAO)” project, funded by the Fondation “Sciences et Technologies pour l’A´ eoronautique et l’Espace (STAE)”, Toulouse, France, within the “R´ eseau Th´ ematique de Recherche Avanc´ ee (RTRA)”.

References C. G. Broyden. The convergence of a class of double-rank minimization algorithms. Journal of the Institute of Mathematics and its Applications, 6, 76–90, 1970. Y. Dai and Y. Yuan. A nonlinear conjugate gradient method with a strong global convergence property. SIAM Journal on Optimization, 10(1), 177–182, 1999. Y. Dai and Y. Yuan. An efficient hybrid conjugate gradient method for unconstrained optimization. Annals of Operations Research, 103, 33–47, 2001. J. E. Dennis and R. B. Schnabel. Numerical Methods for Unconstrained Optimization and Nonlinear Equations. Prentice-Hall, Englewood Cliffs, NJ, USA, 1983. Reprinted as Classics in Applied Mathematics 16, SIAM, Philadelphia, USA, 1996. E. D. Dolan and J. J. Mor´e. Benchmarking optimization software with performance profiles. Mathematical Programming, 91(2), 201–213, 2002. M. Fisher. Minimization algorithms for variational data assimilation. in ‘Recent Developments in Numerical Methods for Atmospheric Modelling’, pp. 364–385, Reading, UK, 1998. European Center for Medium-Range Weather Forecasts.


12

R. Fletcher. A new approach to variable metric algorithms. Computer Journal, 13, 317–322, 1970. R. Fletcher and C. M. Reeves. Function minimization by conjugate gradients. Computer Journal, 7, 149–154, 1964. D. Goldfarb. A family of variable metric methods derived by variational means. Mathematics of Computation, 24, 23–26, 1970. S. Gratton and Ph. L. Toint. Approximate invariant subspaces and quasi-Newton optimization methods. Optimization Methods and Software, (to appear), 2009. S. Gratton, M. Mouffe, A. Sartenaer, Ph. L. Toint, and D. Tomanos. Numerical experience with a recursive trust-region method for multilevel nonlinear optimization. Optimization Methods and Software, (to appear), 2009. S. Gratton, M. Mouffe, Ph. L. Toint, and M. Weber-Mendon¸ca. A recursive trust-region method in infinity norm for bound-constrained nonlinear optimization. IMA Journal of Numerical Analysis, 28(4), 827–861, 2008a. S. Gratton, A. Sartenaer, and Ph. L. Toint. Recursive trust-region methods for multiscale nonlinear optimization. SIAM Journal on Optimization, 19(1), 414–444, 2008b. W. W. Hager and H. Zhang. A new conjugate-gadient method with guaranteed descent and an efficient linesearch. SIAM Journal on Optimization, 16(1), 170–192, 2005. W. W. Hager and H. Zhang. Algorithm 851: CG DESCENT, a conjugate gradient method with guratnteed descent. ACM Transactions on Mathematical Software, 32(1), 112–137, 2006a. W. W. Hager and H. Zhang. A survey of nonlinear conjugate-gradient methods. Pacific J. Optim., 2, 35–58, 2006b. M. R. Hestenes and E. Stiefel. Methods of conjugate gradients for solving linear systems. Journal of the National Bureau of Standards, 49, 409–436, 1952. D. C. Liu and J. Nocedal. On the limited memory BFGS method for large scale optimization. Mathematical Programming, Series B, 45(1), 503–528, 1989. S. G. Nash. A multigrid approach to discretized optimization problems. Optimization Methods and Software, 14, 99–116, 2000. J. Nocedal. Updating quasi-Newton matrices with limited storage. Mathematics of Computation, 35, 773–782, 1980. S. Oh, A. Milstein, Ch. Bouman, and K. Webb. Multigrid algorithms for optimization and inverse problems. in C. Bouman and R. Stevenson, eds, ‘Computational Imaging’, Vol. 5016 of Proceedings of the SPIE, pp. 59–70. DDM, 2003. D. F. Shanno. Conditioning of quasi-Newton methods for function minimization. Mathematics of Computation, 24, 647–657, 1970.

A

Detailed numerical results

In Tables 2 to 15, we display the details of our numerical results (each problem having its own table of results). The first column indicates the chosen conjugacy factor β (see Section 3.1): either the quasi-Newton (QN) choice, or the Dai-Yuan-Hestenes-Stiefel choice (DYHS), or the Hager-Zhang choice (HZ). The second column indicates the linesearch algorithm that was used (see Section 3.2): either the Hager-Zhang one (HZ), or the Dennis-Schnabel one (DS). The third and fourth columns indicate the strategies used to select and order, respectively, the secant pairs (see Section 3.3), while the fifth column gives the value of parameter τ in equation (3.10). The sixth column indicates the execution return status:


13

• 0: the algorithm ran successfully; • 4: the Hager-Zhang linesearch failed because of too many secant steps; • 8: the Hager-Zhang linesearch failed; • 12: the Dennis-Schnabel linesearch failed; • -7: the function value became -Inf. The last three columns indicates the number of iterations (nit), function evaluations (nf), and gradient evaluations (ng), respectively.

LS HZ HZ HZ HZ HZ HZ HZ HZ HZ HZ HZ HZ HZ DS DS DS DS DS DS DS DS DS DS DS DS DS HZ HZ HZ HZ HZ HZ HZ HZ HZ HZ HZ HZ HZ

select LBFGS Full Full Full Full Local Local Local Local Mless Mless Mless Mless LBFGS Full Full Full Full Local Local Local Local Mless Mless Mless Mless LBFGS Full Full Full Full Local Local Local Local Mless Mless Mless Mless

order — Coarse Coarse Fine Fine Coarse Coarse Fine Fine Coarse Coarse Fine Fine — Coarse Coarse Fine Fine Coarse Coarse Fine Fine Coarse Coarse Fine Fine — Coarse Coarse Fine Fine Coarse Coarse Fine Fine Coarse Coarse Fine Fine

τ 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999

status 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

nit 512 259 179 206 168 257 273 225 224 388 344 167 225 635 239 215 188 148 258 188 218 222 364 357 211 194 184 158 162 162 125 243 192 203 171 487 200 183 150

nf 1107 355 257 282 225 368 403 332 351 551 474 220 307 647 297 261 232 177 313 225 272 265 460 440 276 248 369 317 325 325 251 487 385 407 343 975 401 367 301

ng 1619 614 436 488 393 625 676 557 575 939 818 387 532 636 240 216 189 149 259 189 219 223 366 359 212 195 185 160 165 164 126 249 198 210 179 500 214 203 169

β DYHS DYHS DYHS DYHS DYHS DYHS DYHS DYHS DYHS DYHS DYHS DYHS DYHS HZ HZ HZ HZ HZ HZ HZ HZ HZ HZ HZ HZ HZ HZ HZ HZ HZ HZ HZ HZ HZ HZ HZ HZ HZ HZ

LS DS DS DS DS DS DS DS DS DS DS DS DS DS HZ HZ HZ HZ HZ HZ HZ HZ HZ HZ HZ HZ HZ DS DS DS DS DS DS DS DS DS DS DS DS DS



τ 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999

status 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

nit 662 236 198 199 135 231 165 215 159 420 358 172 200 184 194 145 180 125 181 216 206 163 371 271 152 177 1329 237 194 191 489 229 221 224 190 248 290 253 369

nf 1328 527 445 446 298 507 372 470 352 952 798 398 455 369 389 291 361 251 363 433 413 327 743 543 305 355 2694 534 432 426 1010 518 492 493 413 546 653 571 814

ng 663 237 199 200 136 232 166 216 160 423 363 173 203 185 198 146 184 126 185 221 214 167 381 284 174 201 1335 238 195 192 490 230 222 225 191 249 291 255 372


β QN QN QN QN QN QN QN QN QN QN QN QN QN QN QN QN QN QN QN QN QN QN QN QN QN QN DYHS DYHS DYHS DYHS DYHS DYHS DYHS DYHS DYHS DYHS DYHS DYHS DYHS

Table 2: Results for problem DN (level 8, 261121 variables) 14




τ 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999

status 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

nit 992 275 249 249 228 318 252 240 257 282 189 154 171 729 234 197 221 254 175 221 235 258 202 195 179 211 676 330 207 187 190 294 246 207 245 226 250 137 213

nf 1983 427 381 407 344 496 401 383 421 423 276 232 258 748 287 233 263 306 222 269 283 316 264 257 231 281 1353 661 415 375 381 589 493 415 491 453 501 275 427

ng 2975 702 630 656 572 814 653 623 678 705 465 386 429 730 235 198 222 255 176 222 236 259 204 198 180 216 677 341 211 190 193 301 254 213 252 236 267 151 246





τ 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999

status 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

nit 944 280 318 245 242 294 252 208 190 283 255 202 180 676 247 222 201 208 209 245 227 213 211 210 163 171 875 239 263 231 228 277 551 222 524 247 235 321 851

nf 1890 611 705 537 533 657 554 455 413 648 589 460 412 1353 495 445 403 417 419 491 455 427 423 421 327 343 1796 528 584 523 496 632 1142 494 1103 554 532 703 1754

ng 945 281 320 246 243 295 253 209 191 284 257 204 181 677 253 232 203 211 212 255 234 218 224 229 188 203 877 240 264 232 229 278 553 223 529 248 237 322 857



Table 3: Results for problem P2D (level 8, 261121 variables) 15




τ 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999

status 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

nit 250 125 119 114 99 97 93 92 100 135 101 82 104 207 114 100 91 102 96 87 98 94 99 93 107 96 153 107 109 92 87 95 98 100 99 131 91 107 98

nf 501 185 174 155 144 151 146 150 163 191 147 120 155 212 149 126 116 135 113 109 111 105 127 126 135 120 307 215 219 185 175 191 197 201 199 263 183 215 197

ng 751 310 293 269 243 248 239 242 263 326 248 202 259 209 116 102 93 104 98 89 100 96 101 95 109 98 154 109 113 95 90 98 100 103 101 135 95 114 103





τ 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999

status 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

nit 210 108 124 105 103 104 100 90 82 123 149 110 112 153 113 103 112 92 91 96 80 79 133 110 102 106 268 98 96 117 91 98 93 138 156 117 160 203 174

nf 423 244 273 248 235 216 211 196 178 276 337 252 260 307 227 207 225 185 183 193 161 159 267 221 205 213 557 232 214 265 205 214 206 290 320 258 365 441 386

ng 212 110 126 107 105 106 102 92 84 125 152 112 114 154 118 105 115 94 93 98 83 82 139 113 109 111 270 100 98 119 93 100 95 140 158 119 162 205 176



Table 4: Results for problem P3D (level 5, 250047 variables) 16




τ 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999

status 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

nit 1131 297 302 214 208 296 304 272 317 243 135 137 144 921 219 274 255 255 255 266 241 272 216 171 169 181 677 277 241 251 241 306 230 246 270 207 135 149 202

nf 2244 460 467 338 321 448 468 433 509 360 189 200 212 945 280 333 309 309 317 324 292 336 277 216 222 236 1355 555 483 503 483 613 461 493 541 415 271 299 405

ng 3375 757 769 552 529 744 772 705 826 603 324 337 356 922 220 275 256 256 256 267 242 273 217 172 170 185 678 283 245 254 248 312 242 253 277 222 146 166 222





τ 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999

status 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

nit 840 298 253 246 226 241 331 301 301 235 191 193 167 677 253 236 217 215 241 266 260 226 190 261 113 265 1220 309 302 222 371 189 216 353 323 334 280 508 269

nf 1682 658 566 529 500 531 718 681 660 544 450 439 370 1355 507 473 435 431 483 533 521 453 381 523 227 531 2519 685 688 504 795 434 478 760 693 731 623 1077 597

ng 841 299 254 247 227 242 332 303 302 238 198 194 168 678 257 238 219 217 244 277 266 238 200 284 131 294 1222 310 303 223 372 191 217 354 326 335 281 510 274



Table 5: Results for problem DEPT (level 8, 261121 variables) 17




τ 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999

status 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

nit 1699 400 396 554 415 451 613 383 520 347 321 393 249 1840 423 694 400 612 602 457 669 510 295 510 475 409 1591 477 408 287 668 359 425 448 401 478 305 353 388

nf 3395 611 609 864 653 695 947 609 826 527 489 604 379 1842 506 809 475 713 724 548 765 598 378 607 596 492 3183 955 817 575 1337 719 855 897 803 963 613 710 779

ng 5094 1011 1005 1418 1068 1146 1560 992 1346 874 810 997 628 1841 425 695 401 613 603 458 670 511 296 511 476 410 1592 485 415 291 681 370 441 457 408 502 329 386 417





τ 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999

status 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

nit 1647 528 671 501 933 436 570 738 637 432 464 417 526 1689 377 376 396 672 458 275 513 303 269 222 310 245 2434 426 586 656 515 308 500 875 526 502 356 622 238

nf 3296 1185 1462 1105 2011 960 1258 1603 1387 994 1047 973 1149 3379 755 753 793 1345 917 551 1027 607 543 445 621 491 4934 943 1269 1368 1104 695 1091 1811 1126 1115 776 1335 525

ng 1648 529 672 502 934 437 571 739 638 436 465 418 527 1690 383 380 403 681 465 282 525 309 284 235 330 268 2435 427 587 660 517 309 501 878 528 503 358 629 240



Table 6: Results for problem DODC (level 8, 261121 variables) 18




τ 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999

status 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

nit 1208 823 997 800 933 771 704 806 753 789 752 665 690 1260 799 907 789 909 824 723 819 816 812 805 796 802 1241 744 957 809 953 966 800 501 594 749 731 654 748

nf 2391 1304 1594 1312 1557 1219 1099 1344 1252 1231 1209 1089 1144 1275 1032 1068 926 1064 988 883 946 966 1037 1036 1016 989 2490 1512 1926 1630 1910 1949 1630 1007 1195 1534 1509 1326 1529

ng 3599 2127 2591 2112 2490 1990 1803 2150 2005 2020 1961 1754 1834 1261 800 908 790 910 825 724 820 817 813 806 797 808 1249 788 987 836 963 1005 854 516 618 819 826 692 839





τ 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999

status 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

nit 1251 854 814 992 1101 775 793 745 831 776 859 821 818 1081 826 951 658 791 652 552 620 642 782 831 638 850 1292 836 949 888 1033 710 764 896 866 1087 1225 1060 1214

nf 2504 1929 1773 2154 2378 1690 1745 1627 1817 1828 1979 1886 1840 2163 1668 1910 1321 1588 1315 1122 1248 1293 1579 1697 1289 1746 2653 1893 2078 1939 2290 1613 1684 1950 1900 2454 2683 2341 2659

ng 1252 855 815 993 1102 776 794 746 832 779 861 823 824 1082 859 975 675 811 674 590 636 662 818 907 673 965 1295 837 950 889 1035 711 765 903 869 1088 1232 1064 1236



Table 7: Results for problem MINS-SB (level 8, 261121 variables) 19




τ 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999

status 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

nit 1991 3932 3773 4383 4565 4758 4324 3273 3334 4179 4200 3683 3758 2107 3392 3917 4151 4628 4254 4342 3834 3874 4476 4771 4337 4352 1960 3724 4028 4872 4691 4860 4472 3386 3135 4252 4334 3581 3443

nf 3976 6494 6225 7804 8063 7981 7199 5880 5929 6903 6909 6352 6391 2143 4005 4596 4688 5253 4911 4987 4382 4405 5527 5880 5156 5204 3921 7461 8069 9756 9394 9725 8957 6779 6279 8532 8698 7188 6965

ng 5967 10426 9998 12187 12628 12739 11523 9153 9263 11082 11109 10035 10149 2108 3393 3918 4152 4629 4255 4344 3835 3875 4482 4794 4339 4368 1961 3779 4085 4919 4740 4923 4535 3415 3168 4348 4481 3671 3740





τ 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999

status 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

nit 2083 3510 4176 4881 4883 5107 4678 4097 4341 4931 4547 4229 4151 1950 3917 4086 4913 4367 5092 4475 3287 3366 4217 4414 3392 3461 2712 4143 4268 4438 4465 4094 4920 4969 5113 5973 7114 6880 7820

nf 4168 7680 9012 10276 10388 10875 10086 8698 9223 11171 10137 9306 9180 3901 7852 8182 9841 8741 10201 8966 6580 6739 8459 8875 6823 6972 5563 8995 9141 9504 9554 8813 10604 10625 10890 13180 15568 14889 16836

ng 2084 3511 4177 4882 4884 5108 4681 4100 4342 4951 4565 4232 4162 1951 3973 4131 4961 4409 5165 4536 3307 3398 4313 4627 3503 3660 2715 4144 4272 4440 4471 4095 4923 4989 5131 5979 7162 6908 7905



Table 8: Results for problem MINS-OB (level 8, 261121 variables) 20




τ 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999

status 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

nit 1499 1720 1702 1624 1781 1700 1609 1210 1307 1706 1580 1401 1633 1329 1313 1660 1601 1985 1669 1616 1614 1671 1553 1716 1629 1563 1227 1503 1832 1435 1825 1811 1496 1203 1132 1701 1662 1310 1518

nf 2990 2747 2730 2686 3020 2727 2581 2046 2224 2728 2543 2310 2760 1350 1613 1950 1933 2308 1976 1955 1938 1938 1983 2155 2020 1904 2455 3032 3679 2873 3655 3637 3008 2415 2269 3443 3393 2656 3121

ng 4489 4467 4432 4310 4801 4427 4190 3256 3531 4434 4123 3711 4393 1330 1314 1661 1602 1986 1670 1617 1615 1672 1555 1719 1630 1566 1228 1556 1880 1453 1845 1857 1544 1237 1156 1797 1825 1394 1721





τ 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999

status 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

nit 1591 1737 1991 1932 1958 1774 1737 1722 1470 1733 1793 1769 1716 1337 1620 1702 1680 1518 1357 1510 1224 1219 1638 1601 1361 1443 1528 1687 1896 1827 2088 1741 1581 1783 1587 2062 2406 2117 2630

nf 3184 3877 4362 4142 4248 3830 3790 3707 3182 4067 4135 4002 3897 2676 3269 3413 3372 3045 2736 3035 2462 2452 3307 3249 2742 2949 3123 3737 4163 3977 4536 3755 3480 3838 3419 4641 5345 4700 5719

ng 1592 1738 1992 1933 1959 1775 1738 1723 1471 1739 1805 1774 1729 1339 1682 1732 1714 1544 1415 1557 1260 1256 1716 1713 1419 1601 1531 1688 1897 1829 2091 1742 1584 1792 1595 2063 2425 2130 2658



Table 9: Results for problem MINS-DMSA (level 8, 261121 variables) 21




τ 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999

status 0 0 0 8 0 0 0 8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 8 0 8 0 8 0 0 0 8 0 0 0

nit 6515 4759 6171 8314 10751 3926 1380 2663 2740 1507 7496 3440 6746 7142 2648 4756 7494 10773 3295 1487 4653 3349 1757 2566 2453 2275 6471 976 5707 3550 8763 714 1868 5552 2931 1360 11206 2514 6592

nf 13022 5933 8649 10640 15170 5190 2189 3566 4498 2093 14260 4790 12845 7309 4200 6436 11150 14305 4663 1835 6235 3990 2592 3012 3345 2614 12943 2162 11612 7326 17715 1628 3807 11206 5897 2860 22426 5069 13218

ng 19537 10692 14820 18954 25921 9116 3569 6229 7238 3600 21756 8230 19591 7143 2649 4757 7496 10774 3296 1488 4654 3350 1761 2580 2454 2282 6472 1315 6358 4260 9627 998 2072 6077 3099 1617 11545 2740 6856





τ 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999

status 0 0 0 0 0 0 0 0 0 0 0 0 0 0 8 8 8 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

nit 7822 3115 4797 8206 9501 2836 1781 5126 1737 2537 2970 3472 3166 6462 499 5015 1561 7390 3290 1918 4922 2461 76 6355 3092 6561 6639 3807 5234 7318 12040 3266 1963 3865 2126 2028 6571 4065 2931

nf 15664 8073 11355 20663 22249 6831 3988 11995 3773 6241 6344 8311 6814 12925 1156 10234 3375 14953 6724 3859 9944 4965 213 12745 6257 13127 13635 9737 11883 17302 27087 7798 4269 8672 4574 4796 13624 8900 6297

ng 7823 3116 4798 8207 9502 2837 1782 5127 1738 2542 2982 3474 3174 6463 724 5599 2003 8126 3775 2066 5393 2617 143 6601 3370 6765 6651 3808 5239 7320 12054 3267 1971 3871 2129 2033 6644 4079 2991



Table 10: Results for problem IGNISC (level 8, 261121 variables) 22




τ 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999

status 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 -7 -7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

nit 1051 268 247 198 191 282 268 241 214 279 430 185 143 871 210 301 209 206 240 192 6 6 141 156 157 197 697 304 224 244 280 243 234 259 225 228 256 158 182

nf 2101 420 376 314 296 442 408 378 351 421 741 280 213 894 276 365 250 245 299 245 30 31 185 202 205 245 1395 609 449 490 561 487 469 519 451 457 513 317 365

ng 3152 688 623 512 487 724 676 619 565 700 1171 465 356 872 211 302 210 207 241 193 25 26 142 157 158 199 698 315 228 249 288 247 242 266 230 237 283 168 208





τ 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999

status 0 0 0 0 0 0 0 -7 -7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 -7 0 0 0 -7 -7 0 0 0 0

nit 1021 285 256 228 208 308 269 6 6 206 233 190 340 697 249 208 268 232 311 218 239 197 221 262 163 134 1376 195 325 66 220 281 213 6 6 232 242 272 526

nf 2044 630 562 512 452 695 609 36 37 471 526 429 756 1395 499 417 537 466 623 437 479 395 443 525 327 269 2839 434 723 169 476 615 480 36 36 509 531 604 1101

ng 1022 286 257 229 209 309 270 25 26 207 234 191 344 698 253 212 271 238 317 225 248 201 227 281 174 157 1384 196 326 85 221 282 214 25 25 233 246 274 533



Table 11: Results for problem DSSC (level 8, 261121 variables) 23




τ 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999

status 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

nit 1361 325 293 282 289 366 281 290 307 239 184 181 285 1049 221 259 254 302 232 243 292 252 259 223 167 176 896 279 214 226 259 330 304 313 280 235 191 192 193

nf 2696 494 448 439 451 570 432 464 485 350 266 267 466 1071 281 322 306 354 291 295 354 308 336 282 215 225 1793 559 429 453 519 661 609 627 561 471 383 385 387

ng 4057 819 741 721 740 936 713 754 792 589 450 448 751 1050 222 260 255 303 233 244 293 253 260 224 168 179 897 281 219 231 261 336 312 319 292 246 206 202 228





τ 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999

status 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

nit 1007 282 263 210 279 324 317 285 289 261 246 153 191 896 300 272 240 272 318 304 244 320 248 221 227 214 1050 291 434 281 264 249 263 214 501 264 4257 326 338

nf 2016 630 584 460 607 715 692 621 617 592 571 359 434 1793 624 545 481 545 637 609 489 641 497 443 455 429 2165 655 944 615 585 552 596 473 1046 599 8560 716 729

ng 1008 283 264 211 280 325 318 286 291 263 250 154 194 897 326 274 245 277 326 313 253 328 253 236 274 247 1052 292 435 282 265 250 264 218 504 265 4263 327 341



Table 12: Results for problem BRATU (level 8, 261121 variables) 24




τ 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999

status 0 0 0 0 0 0 0 0 0 0 0 0 0 12 0 0 0 12 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

nit 28933 30063 20870 22759 21467 11594 40262 11681 65863 14422 460429 15060 423963 130051 42006 24142 35948 21063 9776 5697 10629 5718 13101 47200 16718 19743 20449 33209 20271 29485 25171 12777 123395 11639 69692 13272 444361 13754 439886

nf 57848 34683 26531 25785 28736 13580 79306 14198 130588 18488 919338 19934 846496 133142 67802 34132 57713 29400 14318 6790 14662 6707 19192 52292 22928 22219 40677 67698 40715 60738 50532 26110 246671 23420 139206 26827 885886 27654 878698

ng 86781 64746 47401 48544 50203 25174 119568 25879 196451 32910 344941 34994 317614 130051 42007 24143 35949 21063 9777 5698 10630 5719 13109 47363 16726 19776 20674 40262 23393 37379 29772 15336 123782 13274 70166 15141 447771 15212 441514





τ 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999

status 12 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 0 0 0

nit 130051 47969 36051 27314 56895 10559 6592 13686 9438 14025 102389 17422 66648 19982 37048 19398 26898 134273 15004 50374 12148 113034 13218 434477 13439 361991 130051 41454 32023 26062 22723 10422 10634 13238 7976 15578 40658 22808 30613

nf 260335 125882 75684 71534 119970 25966 14056 32650 19416 34687 212689 42036 138358 39846 75623 38903 55242 267601 30626 100251 24542 225892 26669 868612 27029 722935 267255 105927 74727 64157 52630 25056 22358 30213 16631 35038 84202 51188 63786

ng 130051 47971 36052 27315 56896 10560 6594 13687 9442 14029 102668 17425 66811 20104 44422 22485 33572 139130 18036 51092 14002 113464 15068 435375 14757 363519 130307 41457 32047 26064 22752 10433 10688 13255 8012 15602 41232 22859 31002



Table 13: Results for problem NCCS (level 7, 130050 variables) 25




τ 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999

status 0 0 0 0 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

nit 29619 31022 22076 29235 20398 13121 61901 19041 73108 13547 439575 14228 421398 130051 40156 24111 29683 24529 14849 5329 13445 6113 14804 56213 16572 25522 21286 38748 22177 28494 26542 14398 44878 14097 70959 13707 410980 14738 431683

nf 59220 35849 28678 33036 25958 15415 122800 22983 145307 17301 877685 18900 841370 133157 64776 34121 47535 35038 21873 6275 18705 7252 21648 61958 23005 28347 42451 79016 44288 58710 53582 29487 89276 28562 141846 27649 820258 29659 862480

ng 88839 66871 50754 62271 46356 28536 184701 42024 218415 30848 329315 33128 315692 130051 40157 24112 29684 24530 14850 5330 13447 6114 14814 56409 16578 25609 21409 46265 25831 36273 31424 17440 45557 16634 71245 15720 413276 16387 433157





τ 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999

status 12 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 0 0 0

nit 130051 41178 25856 32108 29569 13284 5982 13994 6372 16287 112012 18368 51987 21374 32607 19242 31597 20454 13917 90512 13131 102106 12469 424130 14374 413701 130051 42900 27659 32117 26712 15361 8225 12422 9867 14768 53127 36372 31030

nf 260339 108083 62258 83554 65520 32614 12913 33510 13603 40373 232746 44421 108306 42518 66663 38429 65133 41148 28439 180394 26560 204120 25240 847645 28896 826422 267428 109942 64424 79428 61945 36424 17247 28276 20735 34142 110173 80519 64622

ng 130051 41179 25857 32109 29570 13285 5983 13995 6373 16290 112346 18372 52128 21606 39382 22873 39892 24340 16908 91390 15145 102495 14287 425300 16061 415240 130337 42901 27674 32121 26743 15378 8279 12429 9901 14820 53910 36474 31528



Table 14: Results for problem NCCO (level 7, 130050 variables) 26




τ 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999

status 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

nit 25283 28261 12786 12713 16839 9692 13773 9108 22490 8398 30066 7081 15482 14348 10952 15984 16559 15474 7994 6386 7279 5110 4316 2446 6979 4017 22442 16034 16755 17505 16080 7613 26900 10105 16662 4759 33220 7851 21431

nf 50538 34072 23779 15123 22622 11962 27479 11273 44837 10854 60050 9457 30910 14706 17632 22030 25562 21580 12079 6898 10228 5590 6350 2688 9632 4435 44885 33456 33810 36005 32601 16032 53828 20365 33328 9886 66453 15779 42884

ng 75821 62333 36565 27836 39461 21654 41252 20381 67327 19252 90116 16538 46392 14349 10953 15985 16560 15475 7995 6390 7280 5113 4319 2456 6981 4028 22443 19829 17430 21415 18037 9539 26966 11257 16678 5700 33258 8549 21471





τ 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999 1.000 0.999

status 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

nit 12367 17200 14249 17764 19856 4608 3198 9055 3300 6605 7204 9260 7265 22442 9776 12906 15266 12196 5098 9410 5760 15309 4467 18425 7271 22601 12417 16401 13789 14625 12178 4657 3357 8977 4149 3276 4795 10837 5260

nf 24756 45479 33704 45739 47571 11636 6680 21615 6968 16550 14924 22333 15135 44885 20243 25936 31419 24488 10736 18821 11708 30622 9296 36851 14666 45203 25534 42098 31642 35445 27812 11261 7093 20563 8706 7639 9994 23856 11116

ng 12368 17201 14250 17765 19857 4609 3199 9056 3302 6606 7234 9264 7277 22443 11937 13266 18701 12621 6411 9430 6537 15332 5377 18439 7965 22619 12441 16402 13800 14627 12200 4662 3384 8988 4172 3280 4871 10853 5353



Table 15: Results for problem MOREBV (level 8, 261121 variables) 27