Complete Memory Structures for Approximating Nonlinear ... - CiteSeerX

3 downloads 0 Views 291KB Size Report
Bryan W. Stiles, Irwin W. Sandberg, and Joydeep Ghosh. Department of Electrical and Computer Engineering. The University of Texas at Austin. Abstract.
Complete Memory Structures for Approximating Nonlinear Discrete-Time Mappings

1

Bryan W. Stiles, Irwin W. Sandberg, and Joydeep Ghosh Department of Electrical and Computer Engineering The University of Texas at Austin Abstract This paper introduces a general structure that is capable of approximating input-output maps of nonlinear discrete-time systems. The structure is comprised of two stages, a dynamical stage followed by a memoryless nonlinear stage. A theorem is presented which gives a simple necessary and sucient condition for a large set of structures of this form to be capable of modeling a wide class of nonlinear discrete-time systems. In particular, we introduce the concept of a \complete memory." A structure with a complete memory dynamical stage and a suciently powerful memoryless stage is shown to be capable of approximating arbitrarily well a wide class of continuous, causal, time-invariant, approximately- nite-memory mappings between discrete-time signal spaces. Furthermore we show that any bounded-input bounded-output, time-invariant, causal memory structure has such an approximation capability if and only if it is a complete memory. Several examples of linear and nonlinear complete memories are presented. The proposed complete memory structure provides a template for designing a wide variety of arti cial neural networks for nonlinear spatio-temporal processing.

This work was supported in part by an NSF grant ECS 9307632 and ONR contract N00014-92C-0232. Bryan Stiles was also supported by the Du Pont Graduate Fellowship in Electrical Engineering.

2

Contents

I Introduction

3

II Two-Stage Dynamic Networks

4

III Complete Memory Structure Theorems

6

IV Examples of Complete Memory Structures

14

V Discussion

16

VI Appendix

19

IV-ALinear Examples : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 14 IV-B Nonlinear examples : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 15

VI-AProof of Theorem 1 : VI-B Proof of Corollary 3 VI-CProof of Corollary 5 VI-DProof of Theorem 2 :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

: : : :

19 21 22 23

3

I. Introduction

A large volume of theoretical work has been performed regarding the properties and capabilities of memoryless approximators. Many feedforward networks have been shown to be universal approximators of static maps in the sense of being able to approximate arbitrarily well any continuous real-valued function on a bounded subset of < [u ; u ;    ; uqt+q ] if t < tf (Kq (u))(t) = > qt+1 qt+2 (32) : [0; 0;    ; 0] otherwise.

20

Further we also de ne a set of mappings fHi g from 0 such that jx(i) ? y(i)j > . Third i < j  t implies x(j ) = y(j ). The number i represents the latest time prior to t at which x and y di er. Now we use i to de ne a value . If i < t, is de ned by

= 

t

Y

h=i+1

(1 ?  ? x(h)) = 

t

Y

h=i+1

(1 ?  ? y(h)):

(58)

If i = t, then =  . Observe > 0 because  +  < 1. Using Lemma 2 and some algebraic manipulation we derive the following: (b  x)(t) ? (b  y)(t) =

i

X

2 4

i

Y

j =1 h=j 2 Yi

(1 ?  ? x(h)) ?

+  4 (1 ?  ? x(j )) ? j =0

3

i

Y

h=j i Y j =0

(1 ?  ? y(h))5

(59)

3

(1 ?  ? y(j ))5 :

Since we have restricted  and to have positive values such that  +  < 1 we can make an important observation that 0 < (1 ?  ?  )  (1 ?  ? x(t))  (1 ?  ) < 1 (60) for all x 2 X and t 2 Z+ . Consider the special case in which i = 0. In this case (b  x)(t) ? (b  y)(t) = (y(0) ? x(0)):

(61)

Since i = 0, x(0) 6= y(0), and therefore (b  x)(t) 6= (b  y)(t). Now consider the case in which i  1. From (59) and (60) we derive a lower bound on j(b  x)(t) ? (b  y)(t)j in terms of i, , and :

j(b  x)(t) ? (b  y)(t)j   ?

i

X

j =2

(1 ?  )j ? (1 ?  ?  )j

(62)

(1 ?  )i+1 ? (1 ?  ?  )i+1 : ?  



Because (1 ?  ) > (1 ?  ?  ) and i > 0 the following set of inequalities hold. (1?  )2 1 Let  = (  )(  + ) ? 1. Let =  2 . i+1

Xh

j =2

i

(1 ?  )j ? (1 ?  ?  )j 

1

Xh

j =2

(1 ?  )i+1 ? (1 ?  ?  )i+1  (1 ?  )i+1  (1 ?  )2  1  (1 ?  )i+1 ? (1 ?  ?  )i+1   



i

(1 ?  )j ? (1 ?  ?  )j = 

(63) (64) (65)

25

jbx(t) ? by(t)j   ( ?  ? )

(66)

Since ,  , and  are positive values, it is sucient to show that the quantity  + can be made arbitrarily close to zero by selecting appropriate values for and  . The upper bound on the range of is given by the inequality  +  < 1. For any c such that (1 ?  ) > c > 0 the following is an acceptable value for .

= 1 ? ( + c)

(67)

Since  can take values arbitrarily close to zero, we can complete the proof of the second property by demonstrating that we can choose appropriate values  and c so that  + < . Let  > 0. For any  < 0:5 we can choose  =  and c = .

 )2 (1 + ) ? 1 = 1 + ( + c)2 (1 + 1?( +c) ) ? 1  + = 1 + (1 ?(  + ) (1 ? ( + c))(1 ? c)

(68)

If we plug in our assigned values for  and c we get:

 + = (1 ? 21)(1 ? ) 1 + 42 + 4 ? 82 ? 1 



(69)

Taking the limit as  approaches zero we get:

 + = lim (1)(1 + 0 + 0 ? 0) ? 1 = 0 !0

(70)

Since choosing any arbitrarily small  yields values of and  in the proper range, there must be acceptable values of and  for which  + < . Thus B satis es the second property of a complete memory. Next we show that the third property holds. If b 2 B then (b  T  x)(t) = (b  x)(t ? ) for all t 2 Z+ , all x 2 X and any such that 0   t. By using mathematical induction it can be readily shown that (b  T  x)(t) = 1 for all t < . Using this fact, it is then easy to show that by the recursive de nition of habituation given in Theorem 2 the third property is satis ed. Once again we use mathematical induction: Because b?1 = (b  T  x)( ? 1) = 1 and x(0) = (T  x)( ), (b  T  x)( ) = (b  x)(0); and since (T  x)(t) = x(t ? ) for all t > , it follows directly that the assumption, (b  T  x)( + k) = (b  x)(k) implies b  T  x( + k + 1) = (b  x)(k + 1) for any k 2 Z+ . Therefore B satis es the third property of a complete memory. The fourth requirement for B to be a complete memory is that the elements of B are causal. Causality is readily apparent from the de nition of B given in Theorem 2. Thus, B is a complete memory and the proof of Theorem 2 is complete.