A NEW FAMILY OF HYBRID CONJUGATE GRADIENT METHOD FOR UNCONSTRAINED OPTIMIZATION AND ITS APPLICATION TO REGRESSION ANALYSIS

. We know many conjugate gradient algorithms (CG) for solving unconstrained optimization problems. In this paper, based on the three famous Liu–Storey (LS), Fletcher–Reeves (FR) and Polak– Ribi´ere–Polyak (PRP) conjugate gradient methods, a new hybrid CG method is proposed. Furthermore, the search direction satisfies the sufficient descent condition independent of the line search. Likewise, we prove, under the strong Wolfe line search, the global convergence of the new method. In this respect, numerical experiments are performed and reported, which show that the proposed method is efficient and promising. In virtue of this, the application of the proposed method for solving regression models of COVID-19 is provided.


Introduction
Unconstrained optimization is a branch of optimization in which we minimize an objective function that depends on real variables with the total absence of restrictions on their values of those variables.Thus, we consider the general unconstrained optimization problems, as follows: where  : R  → R is the continuously differentiable function and its first derivative is represented by () = ∇ ().Though several robust optimization algorithms with rapid convergence have shown to be available to solve the above nonlinear optimization model, many researchers still refer to the conjugate gradient algorithm (CG) as it uses low memory and good convergence properties.Besides, this method was first established by Hestenes and Stiefel [18] to solve unconstrained linear optimization problems.Then, in 1964, Fletcher and Reeves [14] extended the form of the conjugate gradient method to solve unconstrained nonlinear minimization problems.As a consequence, the results of the expansion inspired researchers to suggest a new conjugate gradient method with good computational performance and, at the same time, good convergence properties [6].Generally, the iterates of the CG methods are usually determined through the following recursive computational scheme: and where   is the current iteration,   is the gradient of  at the point   ,   is the search direction,   ∈ R is the conjugate parameter which characterizes different versions of the CG methods and   > 0 is the step size many line search techniques can obtain.In the midst of which, exact line search, weak Wolfe line search, or strong Wolfe line search, but we use, in our research, strong Wolfe line search, which is defined by the following conditions [27,28]: where scalars  and  satisfy 0 <  ≤  < 1.
One of the essential classes of CG methods is the hybrid conjugate gradient algorithms.Further, the hybrid schemes have better computational performances and more robust convergence properties than conventional CG methods as they take advantage of the two parameters used to build them.For this reason, many researchers cared about hybrid or mixed conjugate gradient methods.Djordjević [10], proposed the following hybrid method: [5], proposed the following hybrid method:    =    DY  + (1 −   ) HS  , Li and Sun [20], proposed the following hybrid method: Liu and Li [21], proposed the following hybrid method: In addition, Sabrina et al. [17] proposed a new hybrid CG method based on combination of FR, PRP and DY conjugate gradient algorithms in which , where, 0 <   < 1.
Inspired by this research, we propose in this study a new hybrid CG method based on combination of LS, FR and PRP conjugate gradient algorithms for solving unconstrained optimization problems.In addition, we alike apply, in this study, the new method for solving a model of COVID-19 outbreak around the globe in which the data is taken from January to September 2020.
In light of this, the paper is structured as follows.In Section 2, we will describe the proposed method with its corresponding algorithm and further establish the descent condition and convergence under inexact line search.In Section 3, we present the numerical experiments to show the efficiency of our new method and the application of regression models of COVID-19 using the new method is illustrated in Section 4. Finally, a brief conclusion is drawn in Section 5.

Proposed method, algorithm
In this paper, we propose another combination of LS, FR and PRP conjugate gradient algorithms.We use the following conjugate gradient parameter: As a consequence, the direction   is given by: The parameters   ,   in (6) satisfying 0 ≤   ,   ≤ 1 which will be determined in a particular way that will later be described.It should be noted that: -If   = 1 and   = 0, then  New
-If   = 0 and 0 and  PRP

𝐾
is a convex combination between  LS  and  FR  .See [11].Finally, if   ∈]0, 1[,   ∈]0, 1[ and 0 <   +   < 1, then we have a new hybrid CG method as a convex combination of three methods "LS, FR and PRP".From ( 6) and (7), it is clear that: To select the parameters   and   , we use the traditional conjugacy condition, i.e. (  +1   = 0).Thus, we have the following lemma.Lemma 1.If the conjugacy condition   +1   = 0 is satisfied at every iteration, we get Proof.Multiplying (8) by   from the left and using the conjugacy condition, we obtain Finally, after some algebra, we have: The parameter   given by ( 9) can be outside the interval [0, 1].However, in order to have a real convex combination in (6), the following rule is considered: if (6).Therefore, under this rule for   selection.

The sufficient descent condition
In this study, we will establish the sufficient descent of our new method, which plays a vital role in the global convergence analysis.Thus, we need the following assumptions: Assumption 2. In a neighborhood  of  the function  is continuously differentiable and its gradient ∇ () is Lipschitz continuous, i.e. there exists a constant 0 <  < ∞, such that: Under Assumptions 1 and 2 on  , there exists a constant Γ ≥ 0, such that: for all  ∈  [4].
The following theorem proves that the search direction obtained by the new method satisfies the sufficient descent condition.
Theorem 1.Let the sequences {  } and {  } be generated by SCH method.Then, the search direction   satisfies the sufficient descent condition: where  > 0.
Proof.The following proof is by induction.We show that search direction   shall satisfy the sufficient descent condition holds for  = 0, the proof is a trivial one, i.e.  0 =  0 so   0  0 = −‖ 0 ‖ 2 , and we conclude that sufficient descent condition holds for  = 0. Next, we assume that (13) holds for some  ≥ 1.

Now we have:
we can write It follows that Produces after some arrangements Multiplying ( 14) by   +1 from the left, we get We have proven seven cases.
Case 1.If   = 1 and   = 0, then we get We are going to prove that the sufficient descent condition holds for LS, we have In addition, we have where  is positive, so • By using (16) in the above inequality, we get Therefore Case 2. If   = 0 and   = 1, the relation (15) becomes , under the strong Wolfe line search, the FR method satisfies the sufficient descent condition [16] where  2 > 0.
Case 3. If   = 0 and   = 0, the relation (15) becomes We are going to prove that the sufficient descent condition holds for PRP when the strong Wolfe line search is used.Thus, we have: we know that      ≤      where  is positive, multiply both sides by (−1), we have Implies that so the equation ( 18) becomes where  3 > 0.Then, the proof is completed.Case 4. If   = 0 and 0 <   < 1, we get In [9], Djordjević proved that the sufficient descent condition holds where  4 > 0. Case 5.If   = 0 and 0 <   < 1.Then The sufficient descent condition is fulfilled and mentioned in [1], such that where  5 > 0. Case 6.If 1 −   −   = 0 and 0 <   ,   < 1, then   = 1 −   and the relation (15) becomes Djordjević proved in [11] that  LSFR +1 satisfies the sufficient descent condition for all , i.e. there exists a number  6 > 0, such that Case 7.
[9] Now, we are going to prove that the direction satisfies the sufficient descent condition when (15), we get So, it is proved that  +1 satisfied the sufficient descent condition.

Convergence analysis
The Zoutendijk condition [32] is often utilized to prove the global convergence of the CG method.Moreover, the following lemma shows that the Zoutendijk condition holds for the proposed method under the strong Wolfe conditions of formulas ( 4) and ( 5).
Lemma 2. Suppose that Assumptions 1 and 2 hold.Consider common iterate (2), where   is a descent direction and   is determined by the strong Wolfe line search (4) and (5).Then, the Zoutendijk condition According to the Assumptions 1 and 2, the strong Wolfe conditions and (13), we conclude that   , which is obtained in our new method is not equal to zero, i.e. there exists a constant  > 0 such that The following theorem gives the global convergence of SCH method.

Numerical analysis
This section is devoted to testing the implementation of the new method.Based on this, we compare the computational performance of the proposed method with some known algorithms, such as the LS, FR, PRP conjugate gradient algorithms, and the new hybrid methods: FRPRPCC (Fletcher-Reeves-Polak-Ribiere-Polyakconjugate-condition) from [9] which we call DJA here, and hFRPRPDY from [17] which we call HYB here.For this comparisons, we consider 400 unconstrained optimization test problems from the CUTE library [7] along with other large-scale optimization problems presented in [3].Above and beyond, we selected 30 largescale unconstrained optimization problems in extended or generalized form.More to the point, each problem is tested for several variables:  = 2, 4, . . ., 25000.Nonetheless, the analysis was based on the number of iterations and central processing unit CPU time.For the numerical tests, the iterations are terminated when ‖  ‖ ∞ < 10 −6 , at which ‖ • ‖ ∞ is the maximum absolute component of a vector, the parameters in the strong Wolfe line searches are chosen to be  = 10 −3 and  = 10 −4 and the hybridization parameter   = 0.5.On the other hand, all programs are written in Matlab and compiler settings on the PC machine with Intel(R) Core(TM) i3-4030U CPU @1.90 GHz processor and 4GB RAM and Windows seven professional system.
Comparisons of these methods are given on the following two sides.On the first side, for the th problem, let   1  and   2  be the optimal value found by  1 method and  2 method, respectively.We say that, for the particular problem th, the performance of the  1 method was better than the performance of the  2 method if and number of iterations, or CPU time of  1 method is less than those of  2 method, respectively.On the other side, to obtain complete comparisons in CPU time, we used the profile of Dolan and Moré [12] to evaluate and compare the performance of the set of methods  on a test set  .Assume that  consists of   methods,  consists of   problems.For each problem,  ∈  and method  ∈  denote  , be the computing time required to solve problem  by method .The comparison between different methods is based on the performance ratio defined by  , :=  , | min ∈  , .In consequence, the performance profile is given by where  : R → [0, 1] and 1 ≤  ≤   .The function   is the distribution function for the performance ratio.Moreover,   for a method is a non-decreasing, piecewise constant function, continuous from the right at each breakpoint.Note that   ( ) is the probability for method  ∈  that log 2  , is within a factor  ∈ R + of the best possible ratio.Obviously, when  takes a certain value, a method with a high value of   ( ) is preferable or represents the best method.
In the first set of numerical experiments, we compare the performance of our new algorithm to the LS, FR and PRP conjugate gradient algorithms.Figures 1 and 2 represent the performance profiles of the new method versus LS, FR and PRP based on the CPU time and number of iterations, respectively.The two figures show that the new method is superior to the other conjugate gradient methods on the testing problems.
In the second set of numerical experiments, we present a comparison with the new hybrid method FRPRPCC from [9], which we call DJA here, this comparison shall be under the same above conditions.Moreover, Figures 3 and 4 represent the performance profiles of SCH versus DJA based on the number of iterations and CPU time.
The two figures show that our method performs better than the DJA method for the number of iterations and CPU time.Simultaneously, according to the performance of DJA method in [9], we can conclude that our method is also better than CCOMB from [4], HYBRID from [2], the algorithm of Touatti-Ahmed and Storey (ToAhS) from [26], the algorithm of Hu and Storey (HuS) from [19] and the algorithm (GN) of Gilbert and Nocedal from [15].The third set of numerical experiments compares our new method to the hybrid conjugate gradient algorithm hFRPRPDY from [17], which we call HYB here.Besides, Figures 5 and 6 represent the performance profiles of the new method versus HYB based on the CPU time and number of iterations, respectively.
From these figures illustrated above, we notice that the new algorithm behaves similarly to or better than the HYB method inspired by it.

Application of the New CG method to regression analysis
Novel coronavirus-19 (COVID-19) is a new chain of corona group viruses that was not recognized in human history earlier than December 2019.It was first discovered in Wuhan, China [30] and has spread to various urban areas in China as well as approximately 196 different countries of the world.It has since been declared  an outbreak by the World Health Organization (WHO).It is difficult to take a single point of view on this virus's origin.It can be due to a seafood market exchange, the people's migration from one location to another, or the transmission from animals to humans.Most people infected by the virus will develop mild to moderate symptoms, such as mild fever, cold, and difficulty breathing, and recover without special treatment.According to data reported by the WHO, on the 20th of October 2020, the laboratory declared that the number of confirmed cases is over 40 million, with more than one million deaths recorded in 215 regions and countries around the world since the disease was first reported in Wuhan.
Mathematical modeling plays a vital role in describing the epidemic of infectious diseases and thus overcoming the same at an early stage.Recently, numerous studies modeled various aspects of the coronavirus outbreak, and the application of numerical methods on some COVID-19 models was also studied [25,31].Besides, this paper aims to investigate the performance of the proposed method on a parameterized COVID-19 regression model.For deriving the COVID-19 regression model, the study will consider the total confirmed cases of the infection from January 2020 until September 2020.Subsequently, the obtained data would be transformed into an unconstrained optimization problem, which would later be solved using the proposed method.
Regression analysis is one of the most effective statistical tools for modeling problems in the applied sciences, physical sciences, management and many others.Based on the previous description, we can describe regression analysis as a statistical technique used to estimate the relationship between a dependent variable and one or more independent variables.In virtue of this, the function of regression analysis is defined as follows: where   ,  = 1, 2, . . ., ,  > 0 is the predictor,  is the response variable, and  is the error.For any problem related to regression analysis, the linear regression function can be derived by computing  such that: with  0 , . . .,   representing the regression parameters, these parameters are estimated to minimize the error  value.This scheme is often used when the relationship between  and  is approximated by a straight line.However, these cases rarely occur because most problems are often nonlinear.Therefore, the nonlinear regression scheme is frequently used.In this paper, we considered the nonlinear regression one.
To derive the approximate function, we consider the data from the global confirmed cases of COVID-19 from January to September 2020.Table 1 illustrates the process description from the statistics obtained from the World Health Organization [29].We have data for nine months (Jan-Sept), the months of data collection would be denoted by -variable, and the -variable would denote the confirmed cases corresponding to these months.However, the data for eight months (Jan-Aug) would be considered for fitting the data, while the data for September 2020 would be reserved for error analysis.
From the above data, the approximate function for the nonlinear least square method is defined by: The above function (30) will be utilized when approximating the  data values based on  data values from January to August.Let   denote the number of months and   be the confirmed cases for that month.Based on this information, the above least squares method ( 30) is transformed into the following unconstrained minimization problems.
The data of the first eight months from Table 1 are utilized to formulate the nonlinear quadratic model for the least square method, which is further used to derive the unconstrained optimization model.Based on the above discussion, it is evident that there exist some parabolic relations between the data   and the value of   with the regression function defined by (30) and the regression parameters  0 ,  1 and  2 min Subsequently, using the data of Table 1, we transform (32) to obtain our nonlinear quadratic unconstrained minimization model as follows: The above nonlinear quadratic model was constructed using data from January to August.Meanwhile, the data for September is reserved for relative error analysis of the predicted data.At present, we can apply the SCH, LS, FR and PRP methods for solving the model (33) under the strong Wolfe line search conditions (4) and (5), we obtain the performance results based on iteration numbers and CPU time illustrated in Table 2.
To overcome the difficulty of computing the values of  0 ,  1 ,  2 using matrix inverse, we implement the previously mentioned methods using different initial points.We terminate the computation : -The defined stopping criteria are satisfied based on the value defined for each function.
-The method is unable to solve the model.

Trend line method
In this subsection, we aim to estimate the confirmed cases of COVID-19 for eight (8) months using the proposed SCH, some known CG, and least square methods.From the actual data obtained from Table 2, we use Microsoft Excel software to plot the trend line as demonstrated in Figure 7. Furthermore, to show the efficiency of the proposed method, we compare the approximation functions of the SCH method with the functions of the LS, FR, PRP and trend line methods.Based on the results illustrated in Table 2, it is evident that the suggested SCH method is faster and more efficient compared to the used methods.On the other hand, from the plot, it is clear that the trend line equation obtained is a nonlinear quadratic equation.In light of this, the ideal purpose of the regression analysis is to estimate  0 ,  1 , . . .,   where the error  is minimized.From the above discussion, we can conclude that the SCH method can be used as an alternative to the trend line method and the least squares method, which implies that the method is applicable to real-world situations.

Conclusion
In light of the facts above, the hybrid CG methods are usually obtained based on the classical CG methods by integrating their advantages.In this paper, we proposed a new hybrid conjugate gradient algorithm in which the famous parameter   is computed as a convex combination of  LS  ,  FR  and  PRP  algorithms.Based on some conditions, we show that the proposed algorithm enjoys sufficient descent condition and converges globally under strong Wolfe line search.Further, numerical experiments are considered to illustrate the performance of the proposed method.The results show that the new method is more effective and has a better convergence rate than LS, FR, PRP, FRPRPCC and hFRPRPDY methods.More to the point, our proposed method can solve the COVID-19 case model.

Figure 1 .
Figure 1.Performance Profile based on the iteration number.SCH versus LS, FR and PRP conjugate gradient algorithms.

Figure 2 .
Figure 2. Performance Profile based on the CPU time.SCH versus LS, FR and PRP conjugate gradient algorithms.

Figure 4 .
Figure 4. Performance Profile based on the CPU time.SCH versus hybrid FRPRPCC (DJA).

Figure 5 .
Figure 5. Performance Profile based on the iteration number.SCH versus hybrid hFRPRPDY (HYB).

Figure 6 .
Figure 6.Performance Profile based on the CPU time.SCH versus hybrid hFRPRPDY (HYB).

Table 2 .
Test results for optimization of quadratic model for SCH, LS, FR and PRP.