Recurrent Neural Networks For Solving Linear Inequalities And

1 downloads 0 Views 470KB Size Report
neural networks can solve a linear inequality and equality system, can solve a linear ... many solution processes, e.g., as a preliminary step for solv- ... of the computing units and faster convergence properties. In ... We want to find a vector.
452

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS—I: FUNDAMENTAL THEORY AND APPLICATIONS, VOL. 46, NO. 4, APRIL 1999

Recurrent Neural Networks for Solving Linear Inequalities and Equations Youshen Xia, Jun Wang, Senior Member, IEEE, and Donald L. Hung, Member, IEEE

Abstract— This paper presents two types of recurrent neural networks, continuous-time and discrete-time ones, for solving linear inequality and equality systems. In addition to the basic continuous-time and discrete-time neural-network models, two improved discrete-time neural networks with faster convergence rate are proposed by use of scaling techniques. The proposed neural networks can solve a linear inequality and equality system, can solve a linear program and its dual simultaneously, and thus extend and modify existing neural networks for solving linear equations or inequalities. Rigorous proofs on the global convergence of the proposed neural networks are given. Digital realization of the proposed recurrent neural networks are also discussed. Index Terms—Linear equalities and equations, recurrent neural networks.

I. INTRODUCTION

T

HE PROBLEM of solving systems of linear inequalities and equations arises in numerous fields in science, engineering, and business. It is usually an initial part of many solution processes, e.g., as a preliminary step for solving optimization problems subject to linear constraints using interior-point methods [1]. Furthermore, numerous applications, such as image restoration, computer tomography, system identification, and control system synthesis, lead to a very large system of linear equations and inequalities which needs to be solved within a reasonable time window. There are two classes of well-developed approaches for solving linear inequalities. One of the classes transforms this problem into a phase I linear-programming problem, which is then solved by using well-established methods such as the simplex method or the penalty method. These methods employ many of the matrix operations or have to deal with the difficulty in setting penalty parameters. The second class of approaches is based on iterative methods. Most of them do not need matrix manipulations and the basic computational step in iterative methods is extremely simple and easy to program. One type of iterative methods is derived from the relaxation method for linear inequalities [2]–[6]. These methods are called relaxation methods because they consider one constraint at a time, so that in each iteration, all but one constraint is identified and Manuscript received July 9, 1997; revised May 3, 1998. This work was supported in part by the Hong Kong Research Grants Council under Grant CUHK 381/96E. This paper was recommended by Associate Editor J. Zurada. Y. Xia and J. Wang are with the Department of Mechanical and Automation Engineering, Chinese University of Hong Kong, Shatin, NT, Hong Kong. D. L. Hung is with the School of Electrical Engineering and Computer Science, Washington State University, Richland, WA 99352 USA. Publisher Item Identifier S 1057-7122(99)02754-3.

orthogonal projection is made onto the hyperplane corresponding to it from the current point. As a result, they are also called the successive orthogonal projection methods. Making an orthogonal projection onto a single linear constraint is computationally inexpensive. However, when solving a huge system which may have thousands of constraints, considering only one constraint at a time leads to slow convergence. Therefore, effective and parallel solution methods which can process a group or all of constraints at a time are desirable. With the advances in new technologies [especially very large scale integration (VLSI) technology], the dynamicalsystems approach to solving optimization problems with artificial neural networks has been proposed [6]–[16]. The neuralnetwork approach enables us to solve many optimization problems in real time due to the massively parallel operations of the computing units and faster convergence properties. In particular, neural networks for solving linear equations and inequalities have been presented in recent literature in separate settings. For solving linear equations, Cichocki and Unbehauen [20], [21] first developed various recurrent neural networks. In parallel, Wang [18] and Wang and Li [19] presented similar continuous-time neural networks for solving linear equations. For solving linear inequalities, Cichocki and Bargiela [20] developed three continuous-time neural networks using the aforementioned first approach. These neural networks have penalty parameters which decreases to zero as time increases to infinity in order to get better accuracy of solution. Labonte [22] presented a class of discrete-time neural networks for solving linear inequalities which implement each of the different versions of the aforementioned relaxation-projection method. He showed that the neural network that implemented the simultaneous projection algorithm developed by Pierro and Iusem [4] had fewer neural processing units and better computational performance. However, as Pierro and Iusem pointed out, their method is only a special case of Censor and Elfving’s method [3]. Moreover, we feel that Censor and Elfving’s method can be more straightforward to be realized by a hardware implementation neural network than by the simultaneous projection method. In this paper, we generalize Censor and Elfving’s method and propose two recurrent neural networks, continuous time and discrete time, for solving linear inequality and equality systems. Furthermore, two modified discrete-time neural networks with good values for step-size parameters are given by use of scaling techniques. The proposed neural networks retain the same merit as the simultaneous projection network and are guaranteed to globally converge to a solution of the

1057–7122/99$10.00  1999 IEEE

XIA et al.: RECURRENT NEURAL NETWORKS FOR SOLVING LINEAR INEQUALITIES AND EQUATIONS

linear inequalities and equations. In addition, the proof given in this paper is different from the ones presented before. This paper is organized as follows. In Section II, the formulations of the linear inequalities and equations are introduced and some related properties are discussed. In Section III, basic network models and network architectures are proposed and their global convergence is proved. In Section IV, two modified discrete-time neural networks are given and their architectures and global convergence is shown. In Section V, digital realization of the proposed discrete-time recurrent neural networks are discussed. In Section VI, operating characteristics of the proposed neural networks are demonstrated via some illustrative examples. Finally, Section VII concludes this paper. II. PROBLEM FORMULATION This section summarizes some fundamental properties of linear inequalities and equations and their basic application. , be arbitrary real matrixes and let Let , be given vectors, respectively. No relation is assumed among , and and the matrix or can be rank deficient or even a zero matrix. We want to find a vector solving the systems (1) The problem of (1) has a solution which satisfies all the inequalities and equations if and only if the intersect set and the one of between the solution of is nonempty. It contains two special and important cases. One is the system of inequalities

Another is the system of equations with a nonnegative constraint

453

Thus, they can be formulated in the form of (1) where

and To study the problem of (1) we first define the following energy function:

where . Let solve (1) . From [14], we have the following proposition which shows being zero. the equivalence of (1) and is convex continuously Proposition 1: The function differentiable (but not necessarily twice differentiable) and if and only if piecewise quadratic and and if and only if . fails to exist at points where Although the Hessian of for any where is the row of and the gradient of is globally Lipschitz. Proposition 2: where and and is globally Lipschitz . with constant Proof: Note that then

So

thus, for all

,

As an important application we consider the following linear program (LP): Minimize subject to

(2) and

where the following Maximize subject to

. Its dual LP is Finally, the function property. Proposition 3: For any ,

has an important inequality

(3)

is an By Kuhn–Tucker conditions we know that optimal solution to (1) and (2), respectively, if and only if satisfies and

(4) Proof: Using the lemmas in the Appendix and the second-order Taylor formula, we can complete the proof of Proposition 3.

454

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS—I: FUNDAMENTAL THEORY AND APPLICATIONS, VOL. 46, NO. 4, APRIL 1999

Fig. 1. Architecture of the continuous-time recurrent neural networks.

III. BASIC

and the corresponding discrete-time neural-network model

MODELS

In this section, we propose two neural-network models, a continuous-time one and a discrete-time one, for solving linear inequalities and equations (1), and discuss their network architectures. Then we prove global convergence of the proposed networks. A. Model Descriptions Using the standard gradient descent method for the miniwe can derive the dynamic equamization of the function tion of the proposed continuous-time neural-network model as follows: (5)

(6) is a fixed-step parameter and is the where learning rate. On the basis of the set of differential equations (5) and difference equations (6), the design of the neural networks implementing these equations is very easy. Figs. 1 and 2 illustrate the architectures of the proposed continuoustime and discrete-time neural networks, respectively, where , , and . It shows that each one has two layers of processing units and consists of adders, simple limiters, and integrators or time delays only. Compared with existing neural networks [20], [21] for solving (1), the proposed neural network in (5) contains no

XIA et al.: RECURRENT NEURAL NETWORKS FOR SOLVING LINEAR INEQUALITIES AND EQUATIONS

455

Fig. 2. Architecture of the discrete-time recurrent neural networks.

time-varying design parameter. Compared with existing neural networks [22] for solving (1), the proposed neural network in (6) can solve linear inequality and/or equality systems, and thus can linear program and its dual simultaneously. Moreover, it is straightforward to realize in hardware implementation. B. Global Convergence We first give the result of global convergence for the continuous-time network in (5). . The neural network in (5) is Theorem 1: Let asymptotically stable in the large at a solution of (1). Proof: First, from Proposition 2 we obtain that for any there exists only solution of an fixed initial point

the initial value problem associated with (5). Let and then

Since the function is continuously differentiable and it follows [23] that convex on

Note that

and

then

456

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS—I: FUNDAMENTAL THEORY AND APPLICATIONS, VOL. 46, NO. 4, APRIL 1999

thus

so

Hence, the solution

for (5) is bounded. Furthermore, if then and . except at the equilibrium points. So Therefore, the neural network in (5) is globally Lyapunov stable. Finally, the proof of the global convergence for (5) is similar to the one of Theorem 2 in our paper [13]. It should be pointed out that there are different advantages between continuous-time and discrete-time neural networks. For example, the convergence properties of the continuoustime systems can be much better since certain controlling parameters (the learning rate) can be set arbitrarily large without affecting the stability of the system, in contrast to the discrete-time systems where the corresponding controlling parameters (the step parameter) must be bounded in a small range. Otherwise, the network will diverge. On the other hand, in many operations discrete-time networks are preferable to their continuous-time counterparts because of the availability of design tools and the compatibility with computers and other digital devices. Generally speaking, a discrete-time neural-network model can be obtained from a continuoustime one by converting differential equations into appropriate difference equations though the Euler method. However, the resulting discrete-time mode is usually not guaranteed to be globally convergent since the controlling parameters may not be bounded in a small range. Therefore, we need to prove global convergence of the discrete-time neural network. and Theorem 2: Let . Then the sequence generated by (6) is globally convergent to a solution of (1). Proof: First, from Proposition 3 we have

On the other hand, for any

we have

and

Note that

and

then

Thus, substituting (7) we get

Since , and thus is bounded. Then there exists a subsequence such that

Then

Substituting

since the matrix nite. Thus, Moreover,

we get

since

is symmetric positive semidefiis monotonically decreasing and bounded. (7)

where

hence

then

is continuous, thus,

then the sequence thus

. Finally, because

has only one accumulation point and

Corollary 1: Assume that

and is bounded. If , then the sequence generated by (6) is globally convergent to a solution of (1). is bounded and any level set of Proof: Note that is also so [18], then the sequence the function generated by (6) is bounded since is monotonically . Similar to the decreasing and proof of Theorem 2 we can complete the rest of the proof. Remark 1: The above analytical results for discrete-time neural network in (6) provide only sufficient conditions for global convergence. Because they are not necessary conditions the network in (6) could still converge when . This point will be shown in illustrative examples.

XIA et al.: RECURRENT NEURAL NETWORKS FOR SOLVING LINEAR INEQUALITIES AND EQUATIONS

Remark 2: From the Courant–Fischer minmax Theorem , [24], it follows that but this inequality is not strict. For example, let and , then

457

Proof: The case of Condition 1). Note that are symmetric positive definite. Let

where and to the proof of Proposition 3 we have

Thus, . This shows that the step-size parameter of the network in (6) does not decrease necessarily as the number of the constraint in (1) increases.

where

and

, then similar

, thus

Since

IV. SCALED MODELS From the preceding section we see that the convergence rate of the discrete-time neural network in (6) depends upon the step-size parameter and thus upon the size of the maximum . In the present section, eigenvalue of the matrix by scaling techniques, we give two improved models which have good values for step-size parameters which do not depend upon the size of the maximum eigenvalue of the matrix .

with

and

A. Model Descriptions Using scaling techniques, we introduce two modifications of (6). They are the following: (8) and (9) , , and are where symmetric semipositive definite matrixes. From the view of circuit implementation, the modified network models almost resemble the network model (6). There is only one difference among the three network models, that is, the difference lives in the connection weights of the second layer since the , , , and can be prescaled, matrix respectively. However, these modified models shall have better values for step-size parameters than the basic model (6) when and thus increase the convergence rate.

Thus

and the rest of proof is similar to the Proof of Theorem 2. The case of Condition 2). Note that

and

thus

B. Global Convergence and . If one of Theorem 3: Assume that the following conditions is satisfied then the sequence generated by (8) is globally convergent to a solution of (1). and 1) Let where is the th column vector of the maand let trix and and where is the th . column vector of the matrix and and rank and let 2) Let rank and .

So by the above mentioned proof for Condition 1) we can complete the rest of the proof. and . If one of Theorem 4: Let the following conditions is satisfied, then the sequence generated by (9) is globally convergent to a solution of (1). and 1) Let where is the th row vector of the matrix and . and 2) Let rank .

458

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS—I: FUNDAMENTAL THEORY AND APPLICATIONS, VOL. 46, NO. 4, APRIL 1999

Proof: The case Condition of 1). Since and positive definite matrix By Proposition 3 we have

where

is a symmetric is as well.

. Substituting we get

Because

and thus is monotonically decreasing and bounded. Furthermore, we have

then of Theorem 2 we can obtain

. Similar to the proof

The case of Condition 2). It is similar to the proof of Theorem 3 with the Condition 2). Remark 3: In order to solve (1), we see by the conditions and rank , of Theorems 3 and 4 that if rank , then we may then we may use (8) If rank use (9). On the other hand, for the computational simplicity , , , and , we should select (8) for of and case, and we should select (9) for the the and case. and rank Remark 4: In general, rank . But rank often occurs. For example, let

Then rank rank

, rank

Corollary 2: Let and let and . If one , then of the following conditions is satisfied and generated by (8) is globally convergent to the sequence a solution of (2). and 1) Let where is the th column vector of the and . matrix and . 2) Let rank Proof: Note that

then by Theorem 3 we have the results of Corollary 2. , and . If Corollary 3: Let one of the following conditions is satisfied, then the sequence generated by (9) is globally convergent to a solution of (3). and 1) Let where is the th row vector of the matrix and . . 2) Let Proof: Similar to the proof of Corollary 2. Remark 5: The result of Corollary 2 under Condition 1) has been given by Censor and Elfving [6]. But our proof differs from theirs. Moreover, their method can not prove the results of Theorems 2 through 4. Remark 6: Although all the above mentioned theorems assume that there exists a solution of (1), that is, the systems (1) are consistent, the proposed models can identify this case. When the system is not consistent, the proposed models can give a solution of (1) in a least squares sense (a least squares [6]). This solution to (1) is any vector that minimizes point will be illustrated via Example 2 in Section VI. Remark 7: When the coefficient matrix of the system (1) is ill conditioned, the system of equations leads to stiff differential equations and, thus, affect convergence rate. On the other is very hand, if the row vector norms of the matrix large, the step size in model (9) is smaller, thus its convergence rate will decrease. To alleviate the stiffness of differential equations and simultaneously to improve the convergence properties, one may use preconditioning techniques for the or design a linear transformation for the vector matrix [24]. V. DIGITAL REALIZATION The proposed discrete-time neural networks are suitable for digital realization. In this section, we discuss the implementation issues. The discrete-time neural networks represented by (6), (8), and (9) can be put in the generalized form

, and rank

From Theorems 3 and 4 we obtain easily the following two corollaries.

(10) For (6)

XIA et al.: RECURRENT NEURAL NETWORKS FOR SOLVING LINEAR INEQUALITIES AND EQUATIONS

459

Fig. 3. Block diagram of the one-dimensional systolic array.

For (8)

For (9)

Note that . Let we can augment (10) by adding appropriate zeros into the matrixes and vectors in (10) such that

is stored in the registers R1 and R2 of the th PE. The element is then passed into the next PE’s R1 and R2 through the multiplexer MUX3 while at the same time processed by the current PE’s multiplier-accumulator (MAC) units, MAC1 and MAC2. After MAC executions, the values the th element and are produced by MAC1 and MAC2, respectively. of is stored in register R1 through the The th element of is stored in multiplexer MUX1, and the th element of register R2 through the multiplexer MUX2. The systolic array in the ring. After another then circulates the elements of MAC executions the value of the th element of is generated by the MAC2 in PE and stored in the PE’s R2 register. Finally, the updated th element of is obtained by sum up (in R1) and (in R2), and stored back into R1 and R2 for the next iteration cycle. Since operations in all PE’s are strictly identical and concurrent, the systolic array requires 2 -MAC execution time to complete an iteration cycle based on (11). Compared with a single processor that has the same MAC-execution speed as the PE for the computation based on (6), (8), and (9), assuming all constant matrix or vector -MAC multiplications are precalculated, it will take execution time per iteration cycle.

Then (10) can be rewritten as

VI. ILLUSTRATIVE EXAMPLES (11)

, , , , and can be precalculated In (11), and will converge as the number of iteration increases. The to the problem (1). To converged includes the solution , ease our later discussion let us define , and . For digital realization (11) can be realized by a onedimensional systolic array consisting of processing elements (PE’s) as shown in Fig. 3. Where each PE is responsible for contains elements of updating one element of the vector rearranged with the following the th row of the matrix data structure:

In this section, we demonstrate the performance of the proposed neural networks for solving linear inequalities and equations using four numerical examples. Example 1: Consider the following linear equality and inequality system:

Then

for

where is an element of at the th row and th column. the The functionality of an individual PE is shown in Fig. 4. At the beginning of an iteration, the th element of the vector

It is easy to see that discrete-time neural network in (6) with the solution is

. Using the and .

460

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS—I: FUNDAMENTAL THEORY AND APPLICATIONS, VOL. 46, NO. 4, APRIL 1999

Fig. 4. Data flow diagram of a processing element.

Then

Fig. 5. Transient behavior of the energy function in Example 1.

Fig. 5 depicts the convergence characteristics of the discretetime recurrent neural network with three different values of the design parameter . It shows that the states of the neural network converge to the solution to the problem within ten iterations. Example 2: Consider an inconsistent linear inequality system [19]

First, we use the continuous-time neural network in (5) to solve the inequality systems. In this case, for any an , the neural network always converges initial point globally. Next, we use the discrete-time neural network in , then (6) to solve the above inequalities. Let the neural-network solution in a least squares sense is with the residue vector of . Fig. 6 shows the transient behavior of the recurrent neural network in this example. Example 3: Consider the following linear equations with nonnegative constraint:

Then

, and is the identity matrix. Taking , and we use the discretetime neural network in (8) to solve the above inequalities and the neuralequations. When the initial point is . network solution is Example 4: Consider the following linear program [16]: where

Minimize subject to

XIA et al.: RECURRENT NEURAL NETWORKS FOR SOLVING LINEAR INEQUALITIES AND EQUATIONS

Fig. 6. Transient behavior of the energy function in Example 2.

whose optimal solution the following:

461

Fig. 7. Transient behavior of the energy function and duality gap in Example 4.

. Its dual program is

Maximize subject to

The linear program and its dual can be formulated the form of (1) where

and we use the discrete-time neural network in (6) to solve the above linear program and its dual. Let and let , then the neural-network the initial point be . solution is Since the actual value of the duality gap enables us to estimate directly the quality of the solution. Fig. 7 illustrates the values of energy function and squared duality gap over iterations along the trajectory of the recurrent neural network in this example. It shows that the squared duality gap decreases zig–zag while the energy function decreases monotonically. VII. CONCLUDING REMARKS Systems of linear inequalities and equations are very important in engineering design, planning, and optimization. In this paper we have proposed two types of globally convergent recurrent neural networks, a continuous-time and a discretetime one, for solving linear inequality and equality systems in

real-time. In addition to the basic models, we have discussed two scaled discrete-time neural networks in order to improve the convergence rate and ease the design task in selecting step-size parameters. For the proposed networks (continuous time and discrete time) we have given detailed architectures of implementation which are composed of simple elements only, such as adders, limiters, and integrators or time delays. Furthermore, each of the networks has the number of neurons increasing only linearly with the problem size. Compared with the existing neural networks for solving linear inequalities and equations, the proposed ones have no need for setting a time-varying design parameter. The present neural networks can solve linear inequalities and/or equations and a linear program and its dual simultaneously and, thus, extend a class of discrete-time simultaneous projection networks described in [20] in computational capability. Moreover, our proof on the global convergence differs from any other published. The proposed neural networks are more straightforward to realize in hardware than the simultaneous projection networks. Further investigation has been aimed at the digital implementation and verification of the proposed discrete-time neural networks on field-programmable gate arrays (FPGA). APPENDIX Lemma 1: For any

then (12)

Proof: Consider four cases as follows. 1) For both . So (36) holds. 2) For both So (36) holds equally. and 3) For both

and .

(13) So (36) holds.

462

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS—I: FUNDAMENTAL THEORY AND APPLICATIONS, VOL. 46, NO. 4, APRIL 1999

4) For both and . So (36) holds. From Lemma 1 we easily get the following similar result in [25]. . Then Lemma 2: Let and for any (14) Proof: Let and

. Then . By

Lemma 1 we have

Furthermore, we can generalize the following result. where Lemma 3: Let and . For any (15) Proof: From Lemmas 1 and 2, the conclusion follows. REFERENCES [1] D. Hertog, Interior Point Approach to Linear, Quadratic and Convex Programming: Algorithms and Complexity. Boston, MA: Kluwer, 1994. [2] S. Agmon, “The relaxation method for linear inequalities,” Canadian J. Math., vol. 6, pp. 382–392, 1954. [3] Y. Censor and T. Elfving, “New method for linear inequalities,” Linear Alg. Appl., vol. 42, pp. 199–211, 1982. [4] A. R. De Pierro and A. N. Iusem, “A simultaneous projections method for linear inequalities,” Linear Alg. Appl., vol. 64, pp. 243–253, 1985. [5] Y. Censor, “Row action techniques for huge and sparse systems and their applications,” SIAM Rev., vol. 23, pp. 444–466, 1980. [6] R. Bramley and B. Winnicka, “Solving linear inequalities in a least squares sense,” SIAM J. Sci. Comput., vol. 17, no. 1, pp. 287–303, 1996. [7] D. W. Tank and J. J. Hopfield, “Simple ‘neural optimization networks: An A/D converter, signal decision circuit, and a linear programming circuit,” IEEE Trans. Circuits Syst., vol. 33, pp. 533–541, 1986. [8] M. P. Kennedy and L. O. Chua, “Neural networks for nonlinear programming,” IEEE Trans. Circuits Syst., vol. 35, pp. 554–562, 1988. [9] A. Rodr´ıguez-V´azquez, R. Dom´ınguez-Castro, A. Rueda, J. L. Huertas, and E. S´anchez-Sinencio, “Nonlinear switched-capacitor ‘neural networks’ for optimization problems,” IEEE Trans. Circuits Syst., vol. 37, pp. 384–397, 1990. [10] C. Y. Maa and M. A. Shanblatt, “Linear and quadratic programming neural network analysis,” IEEE Trans. Neural Networks, vol. 3, pp. 580–594, 1992. [11] J. Wang, “Analysis and design of a recurrent neural network for linear programming,” IEEE Trans. Circuits Cyst. I, vol. 40, pp. 613–618, 1993. [12] J. Wang, “A deterministic annealing neural network for convex programming,” Neural Networks, vol. 7, no. 4, pp. 629–641, 1994. [13] Y. Xia and J. Wang, “Neural network for solving linear programming problems with bounded variables,” IEEE Trans. Neural Networks, vol. 6, pp. 515–519, 1995. [14] Y. Xia, “A new neural network for solving linear programming problems and its applications,” IEEE Trans. Neural Networks, vol. 7, pp. 525–529, 1996. [15] , “A new neural network for solving linear and quadratic programming problems,” IEEE Trans. Neural Networks, vol. 7, pp. 1544–1547, 1996. [16] A. Cichocki and R. Unbehauen, Neural Networks for Optimization and Signal Processing. London, U.K.: Wiley, 1993. [17] , “Neural networks for solving systems of linear equations and related problems,” IEEE Trans. Circuits Syst., vol. 39, pp. 124–138, 1992. [18] , “Neural networks for solving systems of linear equations; Part II: Minimax and least absolute value problems,” IEEE Trans. Circuits Syst., vol. 39, pp. 619–633, 1992. [19] J. Wang, “Electronic realization of recurrent neural network for solving simultaneous linear equations,” Electron. Lett., vol. 28, no. 5, pp. 493–495, 1992.

[20] J. Wang and H. Li, “Solving simultaneous linear equations using recurrent neural networks,” Inform. Sci., vol. 76, pp. 255–277, 1994. [21] A. Cichocki and A. Bargiela, “Neural networks for solving linear inequalities,” Parallel Comput., vol. 22, pp. 1455–1475, 1997. [22] G. Labonte, “On solving systems of linear inequalities with artificial neural networks,” IEEE Trans. Neural Networks, vol. 8, pp. 590–600, 1997. [23] J. M. Ortega and W. G. Rheinboldt, Iterative Solution of Nonlinear Equations in Several Variables. New York: Academic, 1970. [24] G. H. Golub and C. F. Loan, Matrix Computations, 3rd ed. Baltimore, MD: The Johns Hopkins Press, 1996. [25] X. Lu, “An approximate Newton method for linear programming,” J. Numer. Comput. Appl., vol. 15, no. 1, 1994. [26] T. H. Corman, C. E. Leiserson, and R. L. Rivest, Introduction to Algorithms. New York: McGraw–Hill, 1990, ch. 2.

Youshen Xia received the B.S. and M.S. degrees in computational mathematics from Nanjing University, China, in 1982 and 1989, respectively. Since 1995, he has been an Associate Professor with Department of Mathematics, Nanjing University of Posts and Telecommunication in China. He is now working towards the Ph.D. degree in the Department of Mechanical and Automation Engineering, Chinese University of Hong Kong, Shatin, NT, Hong Kong. His research interests include computational mathematics, neural networks, signal processing, and control theory.

Jun Wang (S’89–M’90–SM’93) received the B.S. degree in electrical engineering and the M.S. degree in systems engineering from Dalian Institute of Technology, Dalian, China, and the Ph.D. degree in systems engineering from Case Western Reserve University, Cleveland, OH. He is now an Associate Professor of Mechanical and Automation Engineering at the Chinese University of Hong Kong, Shatin, NT, Hong Kong. He was an Associate Professor at the University of North Dakota, Grand Forks. He has also held various positions at Dalian University of Technology, Case Western Reserve University, and Zagar, Incorporated. His current research interests include theory and methodology of neural networks, and their applications to decision systems, control systems, and manufacturing systems. He is the author or coauthor of more than 40 journal papers, several book chapters, two edited books, and numerous conference papers. Dr. Wang is an Associate Editor of the IEEE TRANSACTIONS ON NEURAL NETWORKS.

Donald L. Hung (M’90) received the B.S.E.E. degree from Tongji University, Shanghai, China, and the M.S. degree in systems engineering and the Ph.D. degree in electrical engineering from Case Western Reserve University, Cleveland, OH. From August 1990 to July 1995 he was an Assistant Professor and later an Associate Professor in the Department of Electrical Engineering, Gannon University, Erie, PA. Since August 1995 he has been on the faculty of the School of Electrical Engineering and Computer Science, Washington State University, Richland, WA. He is currently visiting the Department of Computer Science and Engineering, the Chinese University of Hong Kong, Shatin, NT, Hong Kong. His primary research interests are in applicationdriven algorithms and architectures, reconfigurable computing, and design of high-performance digital/computing systems for applications in areas such as image/signal processing, pattern classification, real-time control, optimization, and computational intelligence. Dr. Hung is a member of Eta Kappa Nu.