A DESCENT MODIFIED HS CONJUGATE GRADIENT METHOD WITH AN OPTIMAL PROPERTY

. In this paper, by minimizing the distance between the CG direction and the direction of the improved Perry conjugate gradient method [Yao et al., Numer. Algorithms 78 (2018) 1255–1269], we propose a descent modified HS conjugate gradient method. A remarkable property of the modified HS method is that it can produce sufficient descent property, which is independent of the line search used. Under suitable conditions, we prove that the modified HS method with the standard Armijo line search is globally convergent for uniformly convex functions and the modified HS+ method with standard Wolfe line search is globally convergent for general nonlinear functions. Extensive numerical experiments show that the proposed method is efficient.


Introduction
Consider the unconstrained optimization problem min  (),  ∈ ℛ, where  is smooth and its gradient  is available.Conjugate gradient methods are welcome methods for solving (1.1), especially for large scale problems due to their simplicity and low storage.A typical conjugate gradient method has the form: where   is the stepsize determined by some line search and   is the search direction determined by where  −1 is the conjugate gradient parameter.Different choices of   lead to different conjugate gradients.
Well known formulate for   are the Fletcher-Reeves (FR) [6], Hestenes-Stiefel [11], Polak-Ribiere-Polyak [13,14] and Dai-Yuan [4] which are given by where   =  +1 −   .It is well known that the HS method is generally believed to be one of the most efficient conjugate gradient methods.However, the HS method lacks the descent property.Much efforts has been made to find some modified HS methods which not only satisfy some conjugate condition but also produce sufficient descent direction.Dai and Liao [3] where  is some parameter and   =  +1 −   .Dai and Liao [3] proved that the conjugate gradient method with is globally convergent for general functions.Combining with self-scaling memoryless BFGS method, Hager and Zhang [9] proposed the formula which can be regarded as (1.4) with  = 2 ‖  ‖ 2      .A good property of their method is that the direction   with    satisfies sufficient descent property   +1  +1 ≤ − 7 8 ‖ +1 ‖ 2 which is independent of the line search used.To establish the global convergence of general functions, they proposed the truncated form: By seeking the conjugate gradient direction that is closest to the direction of the scaled memory BFGS method, Dai and Kou [2] proposed the following formula where   is a parameter.The parameter   (  ) can also be regarded as (1.4) with To establish the global convergence of general functions, they considered the following truncated form Numerical results [2] shows that their method with   =      ‖  ‖ 2 performs best among four different choices of   .We refer to a recent review papers [2,10,12,[16][17][18] for details about the progress of conjugate gradient methods.
Recently, Yao et al. [16], proposed an improved Perry conjugate gradient method in which the direction can be written as a quasi-Newton direction.They showed that the method is not only globally convergence but also produces sufficient descent direction independent of the line search.For a large collection of test problems from CUTE library [8], they showed that this method works well in practice.Motivated by theoretical and numerical features of the method [16], by the use of the idea of Dai and Kou [2] we proposed a descent modified HS method.Specifically, we obtain the new parameter   by minimizing the distance between the CG direction and the direction of the improved Perry conjugate gradient method [16].The proposed method satisfies Dai-Liao conjugate condition and the new   can be seen as a special case of  DL  ().A common property of the modified HS method is that it can produce sufficient descent property, which is independent of the line search used.Moreover, if the exact line search is used, the method reduces to the standard HS method.Under suitable conditions, we prove that the modified HS method with the standard Armijo line search is globally convergent for uniformly convex functions and the modified HS+ method with standard Wolfe line search is globally convergent for general nonlinear functions.Extensive numerical experiments show that the proposed method is efficient.
The remainder of the paper is organized as follows.In Section 2, a modified HS method is proposed which satisfied the sufficient descent property independent of line search.In Section 3, we show global convergence of the proposed method.Some numerical results are given in Section 4. Finally, some concluding remarks are listed in Section 5.

A descent modified HS conjugate gradient method
In this section, by minimizing the distance between the CG direction and the direction of the improved Perry conjugate gradient method [16], we propose a descent modified HS method.The descent modified HS method can produce sufficient descent property, which is independent of the line search used.
Yao et al. [16] proposed an improved Perry conjugate gradient method as follows where   = ∇ (  ) is the gradient of  at   and Note that the direction   +1 can be written a quasi-Newton direction They showed that  +1 is symmetric and positive definite.Correspondingly, the direction   +1 produces sufficient descent property independent of line search.Since the direction of a typical conjugate gradient method is a combination of − +1 and   , this and the good properties of this direction (2.1) motivate us to find a new two-term direction  +1 close to   +1 .Similar to the idea of Dai and Kou [2], we find the parameter   by solving the following optimization problem min After easily computation, we obtain a unique solution as follows If the exact line search is used, then the   reduces to the standard  HS  .Moreover, it is worth noticing that the new parameter   corresponds to the Dai-Liao formula (1.4)  ‖  ‖ 2 .The following lemma shows that the direction (1.2) with the parameter (2.2) produces sufficient descent method indepentdent of the line search and the convexity of the objection function.

Convergence analysis
In this section, under suitable conditions, we prove that the modified HS method with the standard Armijo line search is globally convergent for uniformly convex functions and the modified HS+ method with Wolfe line search is globally convergent for general nonlinear functions.The standard Armijo line search is to find steplength where  ∈ (0, 1) and  ∈ (0, 1) are constants.And the Wolfe line search is to find steplength   satisfying where 0 <  <  < 1 are constants.We make the following assumptions on the objective function.
(2) In some neighbourhood  of ,  is continuous differentiable and the gradient of  is Lipschitz continuous with constant  > 0, i.The following lemma plays an important role in establishing global convergence of the proposed method for uniformly convex function and general functions, respectively.Lemma 3.1.Suppose that Assumption 3.1 holds and  () is bounded below.Let {  } be generated by the modified HS method, where   is obtained by Armijo line search or Wolfe line search.Then we have Case 2. Suppose that   satisfies the Armijo condition.By the use of Cauchy-Schwarz inequality to (2.3), we get If   = 1, there exists a positive constant  1 such that The last two inequalities yield Summing these inequalities and using (3.7), we get the conclusion.

Global convergence for uniformly convex functions
In this subsection, we prove that the modified HS method with the standard Armijo line search is globally convergent for uniformly convex functions.By the uniform convexity of  (), there exists a constant  > 0 such that (() − ())  ( − ) ≥ ‖ − ‖ 2 , ∀,  ∈ ℛ  . (3.12) The following theorem established the global convergence of the modified HS method with the standard Armijo line search for uniformly convex functions.

Global convergence for general functions
If the exact line search is used, then the new   reduces to the standard  HS  .The example constructed by Powell [15] shows that the proposed method can not guarantee global convergence for general functions.Similar to the idea of Dai and Kou [2], we replace (2.2) by , by Gauchy-Schwarz inequality, we have This together with Lemma 2.1 implies that there exists a positive constant  such that Note that Lemma 3.6 still holds for the modified HS method with (3.13).
The following lemma is similar to Lemma 3.4 in [7].
Lemma 3.2.Suppose that Assumption 3.1 holds and  () is bounded below.Let {  } be generated by the modified HS method with (3.13) and   be obtained by Wolfe line search.If ‖  ‖ ≥  for all  ≥ 1, then   ̸ = 0 and where Proof.It is clear that   ̸ = 0. Otherwise the condition (3.14) would imply   = 0. We divided formula (3.13) into the following two parts: , and Since ‖  ‖ = ‖ −1 ‖ = 1 and  −1 > 0, we get Note that Since ‖  ‖ ≥ , by the last two inequalities and (3.6) we get The following theorem established the global convergence of the modified HS method with (3.13) for general functions.
where  = 2 The last two inequalities imply that   has Property (*) in [7].Proceeding the similar analysis of Theorem 4.4 [2], we get the conclusion.

Numerical experiments
In this section, we present some numerical results of the new algorithm.All tests are performed under the configuration of Windows10 operating system (64-bit), Intel(R) Core(TM) i3-8145U CPU @2.10 GHz 2.30 GHz, 4.00 GB.We compare the new algorithm (3.13) with the following algorithm: CG DESCENT method [9], Yao method [16] and PRP+ [7].We set  = 0.7 in (3.13).We use the CG DESCENT code (version 5.3) for testing, which can be obtained from Hager's homepage: https://people.clas.ufl.edu/hager/software-archive/.The PRP+ code was obtained from Jorge Nocedal's homepage page: http://www.ece.northwestern.edu/~nocedal/software.html.All parameters of CG DESCENT and PRP+ are default.The new algorithm (3.13) and Yao method [16] is performed with the same line search as that of CG DESCENT method.We tested 84 problems with dimensions 10 000 and 150 000 in [1].We stopped all algorithms when the maximum number of iterations exceeds 20 000 or ‖∇ (  )‖ ∞ ≤ 10 −6 .
We use the performance profile by Dolan and Moré [5] to evaluate the numerical results.The related data can be downloaded from the website: https://github.com/feizaine/data.Figure 1 shows the comparison of the CPU time, the number of iterations, the number of function evaluations and the number of gradient evaluations of the four algorithms, respectively.The proposed algorithm solves 89% of the problem and it is significantly better than the PRP+ method and is very competitive with the CG DESCENT method and Yao method, which shows that the proposed algorithm is processing robust to high-dimensional problems.

Conclusion
In this paper, we have proposed a modified HS method by minimizing the distance between the CG direction and the direction of the improved Perry conjugate gradient method [16].A remarkble property of the new method is that it can always produce the descent direction which is independent of the line search.Under suitable conditions, we prove that the modified HS method is globally convergent with the Armijo line search for uniformly convex functions, and for the general functions, the convergence can also be guaranteed with the standard Wolfe line search.Extensive numerical experiments show that the proposed method is efficient.

Figure 1 .
Figure 1.The performance profile on the CPU time (a), the number of iterations (b), the number of function evaluations (c) and the number of gradient evaluations (d).
Theorem 3.1.Suppose that Assumption 3.1 holds,  () is uniformly convex and bounded below.Let {  } be generated by the modified HS method, where   is determined by (2.2) and   is obtained by Armijo line search or Wolfe line search.Then we have lim Proof.It follows that (3.3) and (3.12) that ‖  ‖ ≤ ‖  ‖ and      ≥ ‖  ‖‖  ‖.