A NEW FAMILY OF DAI-LIAO CONJUGATE GRADIENT METHODS WITH MODIFIED SECANT EQUATION FOR UNCONSTRAINED OPTIMIZATION

In this paper, a new family of Dai-Liao–type conjugate gradient methods are proposed for unconstrained optimization problem. In the new methods, the modified secant equation used in [H. Yabe and M. Takano, Comput. Optim. Appl. 28 (2004) 203–225] is considered in Dai and Liao’s conjugacy condition. Under some certain assumptions, we show that our methods are globally convergent for general functions with strong Wolfe line search. Numerical results illustrate that our proposed methods can outperform some existing ones. Mathematics Subject Classification. 65K05, 90C26, 90C30. Received May 19, 2021. Accepted October 16, 2021.


Introduction
Consider the following unconstrained optimization problem min ( ), ∈ R , (1.1) where the objective function : R → R is continuously differentiable and its gradient ( ) is available. The problem (1.1) has a wide range of applications in areas of scientific computing and engineering. Therefore, its efficient and effective numerical solution methods have been intensively studied in the literature, including the spectral gradient methods [5,15], conjugate gradient methods [4,13] and memoryless BFGS methods [16]. Among them, conjugate gradient methods are popular and efficient for solving (1.1), especially for large scale problems. Let be the th iterate point, the gradient of ( ) at , i.e. = ( ). The (nonlinear) conjugate gradient method is given by where is the step length computed by carrying out an one-dimension line search and is the search direction defined by where is a scalar. Since exact line search for searching is usually expensive and impractical, the strong Wolfe inexact line search is often considered in the convergence analysis and implementation of nonlinear conjugate gradient methods. It aims to find a step size satisfying the following two strong Wolfe conditions where 0 < < < 1.
Nonlinear conjugate gradient method for unconstrained optimization problem is generated from the linear conjugate gradient method for a special quadratic minimization problem min 1 2 + + or its equivalent line system = , where is a real symmetric positive definite matrix. Linear conjugate gradient methods generate a search direction such that the conjugacy condition holds, namely, = 0, ∀ ̸ = . (1.6) For general nonlinear functions, it follows from the mean value theorem that there exists some ∈ (0, 1) such that Therefore, it is reasonable to replace (1.6) by the following conjugacy condition: In 2001, Dai and Liao [6] suggested an extended one which leads to the following conjugate gradient parameter where > 0 is a scalar, −1 = − −1 . Note that the first item has been restricted to be nonnegative like [10]. The DL+ method (1.2)-(1.3) with in (1.9) is globally convergent for general functions under the sufficient descent condition ≤ − ‖ ‖ 2 , > 0. (1.10) and some other suitable conditions, where and hereafter ‖ · ‖ denotes the Euclidean norm of vectors. As a special case of Dai-Liao-type conjugate gradient method, the efficient CG descent method [12] utilizes a particular . The conjugacy parameter of CG descent method is Two further developments of the Dai and Liao's method were made by Yabe and Takano [20] and Li et al. [17] based on different modified secant equations. Some more efficient Dai-Liao-type methods were designed and studied in [2,9,22,23] by using different techniques. In this paper, we further give a new family of Dai-Liao-type conjugate gradient methods for unconstrained optimization problems, including their convergence analysis. Numerical experiments show that our methods can outperform the existing ones.
The rest of this paper is organized as follows. In Section 2, we introduce a new Dai-Liao-type method by modifying the conjugate gradient parameter. Based on the strong Wolfe line search rules, the global convergence for uniformly convex and general objective functions is studied in Section 3 and numerical experiments are performed in Section 4. Finally, in Section 5, we give some conclusions to end this paper.

New Dai-Liao-Type methods
We start with the original Dai and Liao's method in which the quasi-Newton techniques are used. In the quasi-Newton method, an approximation Hessian is updated such that −1 = −1 and the search direction is calculated by Combining the above two equations, we have The above relation implies that (1.7) holds in case of −1 = 0, i.e. the line search is exact. However, in practical numerical algorithms, the inexact line search is adopted instead of exact line search. Dai and Liao suggested the following conjugacy condition: In 2004, Yabe and Takano [20] used the modified secant equation where ∈ [0, 3] and = 2( − +1 ) + ( + +1 ) , is chosen s.t. ̸ = 0, to derive a new conjugacy condition through replacing by = + , the modified conjugacy parameter is In this paper, we will derive a new conjugacy condition from another view of point.

Y. ZHENG
Using the Dai-Liao's conjugacy condition which yields a new conjugate gradient parameter if ̸ = 0, otherwise, Dai and Liao's conjugate gradient parameter DL+ will be used. According to the experience of the quasi-Newton methods with modified secant equations [21], we choose = . In the case of = , the conjugacy parameter new +1 can be written as and we correct it as We call the method (1.2) and (1.3) with given in (2.4) NEW+ method. The corresponding algorithm is given as below: Algorithm 2.1. Improved Dai-Liao conjugate gradient method Step 1: Given 0 ∈ , , > 0, set 0 = − 0 , := 0; if ‖ 0 ‖ ≤ , then stop; Step 2: Compute such that strong Wolfe line search (1.4) and (1.5) hold; Step 3: Step 4: Step 5: Set := + 1 and go to Step 2.
In the rest of the paper, we first analyze the convergence properties of the new algorithm, then give some numerical results which show the modified algorithms are robust and efficient.

Convergence Analysis
Throughout this section, we assume that ̸ = 0 for all ≥ 0, otherwise a stationary point is found. We first give some standard assumptions.
Firstly, we give some estimation on . We know by mean value theorem that On the other hand, since is appeared in the denominator, too small value must be avoided for the numerical stability, we ask | | to satisfy 0 < ≤ | | as shown in Algorithm 2.1. Otherwise, DL+ will be used.
Let be a uniformly convex function, then there exists some constant > 0 such that Then we have that

Thus locates in the interval
Therefore, we assume that the following relationship always holds.
The following theorem states the global convergence property of new method for uniformly convex functions. Proof. It follows from is uniformly convex function that By using Triangular and Cauchy-Schwartz inequalities, we have Therefore, from Lemma 3.2 [6] and the fact is a uniformly convex function, we have For the general function, we only need to show the modified Dai-Liao method with in (2.4) satisfies the Property(*) depicted by Gilbert and Nocedal [10]. The rest analysis is similar to the original Dai-Liao's method.
Note that can be defined such that > 1. If we set Therefore, the NEW+ method has Property (*). Thus, we have the following convergence theorem.

Numerical Experiments
In this section, some numerical results are reported on a a set of 76 unconstrained optimization problems selected from [1] and CUTEst library [11]. We tested the conjugate gradient algorithms with the conjugacy parameters given in Table 1.
For the algorithms DL+, YT+ and our new method, different scaled parameters and are used. In the case where an ascent direction is generated, we restart the algorithm by setting = − . All codes were written in Fortran and in double precision arithmetic. (Note that, for the sake of fairness, at the beginning of experiments we do not directly run Hager and Zhang's CG descent codes for the test problems, we just use their conjugate parameter under the same linear search in our test framework). The stopping rule is set as ‖ ‖ ∞ ≤ 10 −6 . The iteration is also terminated if the total number of iterations exceeds 10,000. Partial numerical results are summarized in Tables 2-6 and given in the form of (number of iterations/number of function-gradient evaluations), the detailed complete numerical results can be downloaded from the website https://github.com/piratetwo/mdl.  In the Tables 2-6, the boldface font is used to mark the first and second efficient method which performs better than the other two algorithms for each and . The number of the best performance for Algorithm NEW+, YT+, DL+ are 32, 15 and 3, respectively.
In most cases, our new method improves Yabe-Takano's method. For a special = 3, which was used in the modified quasi-Newton method, we compare the numerical performance of YT+ and NEW+. We run the codes with different = 0.1, 0.2, . . . , 1 and compute the medians for each problem. The performance profiles introduced by Dolan and Moré [7] are used to display the behaviours of these two methods. Figure 1 shows that the NEW+ method performs the best result regarding the number of iterations and function-gradient evaluations, which is located at the top curve in Figure 1.   Figure 2, for about 62% of all problems, CG descent needs the least iterations, it has the best performance. However, CG descent has the poorer performance than the YT+ and NEW+ regarding the number of function-gradient evaluations which mainly affects the efficiency of the methods.

Conclusions
In this paper, based on the Dai and Liao's conjugacy condition and the modified secant condition proposed by Zhang and Xu [21], we derived a new family of Dai-Liao-type conjugate gradient methods. Under some certain assumptions, we show that our methods are globally convergent for general functions. Numerical results show that our new methods can outperform some existing ones.