Oscillatory Associative Memories 1 Associative ... - Semantic Scholar

1

Oscillatory Associative Memories Joachim M. Buhmann

Rheinische Friedrich{Wilhelms{Universitat Institut fur Informatik II, Romerstrae 164 D-5300 Bonn 1, Fed. Rep. Germany

1 Associative Storage in Dynamical Systems Associative recall and completion of information is one of the astonishing abilities of intelligent living beings. The search for mechanisms which produce this ability of associative memory yielded a class of computational systems composed of many neuron-like, non-linear units. The neural units are connected to arti cial neural networks. The basic principle of associative information recall is a dissipative network dynamics which maps initial network states to a subset of nal states. According to dynamical systems theory, the asymptotic dynamics of such an assembly of neurons can be a xed point, a limit cycle or a chaotic attractor [8]. The best understood arti cial neural networks with associative abilities are networks with a xed point dynamics. The prototypical model of this class, suggested by Hop eld [11], comprises a network of n binary units which are connected in a symmetric fashion and typically show a distributed activity pattern. The dissipative dynamics with synchronous or asynchronous (stochastic) update of neuron states drives the network into a stationary state, a xed point of the neural dynamics. These xed points are supposed to be identical or at least very similar to a set of p random patterns f~ j = 1; : : : ; pg; i 2 f1; 0g; i = 1; : : : ; n, the information content of the associative memory. A connectivity which guarantees the closeness of pattern states and xed points for a moderate number of patterns (p=n < 0:14)

First Draft:

Handbook of BT&NN, (April 22, 1994)

is de ned by Hebb's rule

2

p 1X (i ? 1 )(k ? 1 ); (1) N =1 2 2 i.e., correlated activity of presynaptic and postsynaptic units (i = k ) contributes to an excitatory synapse and uncorrelated activity (i 6= k ) favors an inhibitory synapse. The standard Hop eld model and variants of it were analysed by statistical physicists in great detail [10, 1]. What are the computational limits of xed point associative memories? Any stationary activity pattern of the network is interpreted as a stored memory trace. Suppose now that a xed point network is initialized in a state which corresponds to a superposition of several stored patterns, none of them being dominant. A classi cation of the input as one of the components is inappropriate since all the information of the admixed and equally important patterns has to be suppressed and, therefore, is lost for further processing. Most likely, the associative memory relaxes into a xed point state which corresponds to a superposition of patterns. The mixture is treated as a new, composite pattern. Unfortunately, the information is lost as to which part of the network response belongs to which component of the mixture. The labeling of (1)-bits or (0)-bits in the stationary network state as parts of a particular component pattern in the input mixture has been erased. This information loss, which has been called the superposition catastrophe [13], limits associative memories with xed point dynamics to a sequential associative recall of one pattern at a time. The observation of nonstationary neural activity in visual cortex [7] and olfactory cortex [4, 6] has stimulated interest in associative memory models with an oscillatory or chaotic dynamics. Neural modelling of associative memory has either aimed at mimicking biological behavior as faithfully as possible or at exploring new associative recall principles with segmentation capability which promises to make technical storage devices more versatile. We will discuss three dierent models of oscillatory associative memories which represent both research philosophies: (i) an arti cial neural network with adaptive threshold control,

Wik =

First Draft:


3

(ii) a minimal model of olfactory cortex and (iii) a model for pattern segmentation in the time domain. Related models of oscillatory neural networks without associative behavior are described in the sections on \ Rhythm generating circuits" and on \Oscillations in visual cortex". These models, however, do not exhibit associative capabilities for information storage and normally work with localized rather than distributed activity patterns.

2 A Neural Assembly with Dynamic Threshold Starting with the standard Hop eld model of a conventional associative memory, Horn and Usher [12] introduced a dynamic threshold to model fatigue eects in neuronal response. The neuron states are described by the binary variables Si 2 f?1; 1g. Neurons with a high activity level raise their threshold and, thereby, lower their sensitivity. The neurons are sequentially updated in random order. A probabilistic update rule, parameterized by a computational temperature T , mimics the stochastic in uences which are abundant in biological systems (see section on \Simulating annealing" or on \Computing with attractors"). The equation of motion of the neuron i is given by

1 0 X Si (t + 1) = FT @ Wik Sk (t) ? i (t)A ; 6

k=i

(2)

being FT (x) = 1 with probability (1 + exp(2x=T ))?1. T = 0 reduces Eq. (2) to a deterministic threshold rule. The qualitative behavior of the network at nite T can also be described by n deterministic equations for the expectation value of Si [12]. The dynamic threshold i (t) integrates the past activity of neuron i with an exponential decay, i.e., i (t + 1) =

i (t) + bSi (t + 1): c

(3)

First Draft:


4

Permanently active or inactive neurons (Si (t) = Si (t?1) = : : : = 1) saturate their threshold at the values i (t) = bc=(c?1), respectively. Obviously, appropriate values of b; c destabilize a stationary pattern state and force the network out of a xed point into another one. A Hop eld model with adaptive threshold control exhibits oscillatory pattern activity for a small number of patterns stored. Information recall takes place during intermittent periods of minimal uctuations in network activity. A readout network which is sensitive to changes in neuron states (Si (t +1) 6= Si (t)) has to integrate the neural activities of Horn's oscillatory associative memory and retrieves a stored pattern during periods of quasi-stationarity. The network has settled in a quasi-stationary state when only few neurons change their neural state. The length of these metastable recall periods is controlled by the parameter c, i.e., an increase in c prolongs the metastability of recall periods.

3 A Model for Olfactory Cortex A brain region which supposedly has an anatomical structure and a dynamics similar to associative memory models is the olfactory cortex. Tentative evidence for this hypothesis comes from the sparse, but distributed activity responses of olfactory neurons to similar odors [9], [5]. The olfactory cortex, whose dynamics is strongly dominated by an inhibitory feedback loop between pyramidal cells and inhibitory interneurons, is a brain region with pronounced activity oscillations. How might oscillatory activity enhance the associative abilities of the network compared to associative memories with xed point attractors? Oscillations in associative recall (according to Freeman's view) provide an unbiased access to the memory traces and facilitate transitions from one pattern to another one. Freeman's very detailed olfactory model [5] even shows deterministic chaotic behavior, stimulating speculation on the possible role of chaos in the brain [6]. Baird [2] has proposed an oscillatory associative memory which models the architecture of olfactory cortex in a simpli ed fashion. The model comprises excitatory neurons with

First Draft:


5

long-range connections and local inhibitory neurons which are only connected to nearby excitatory cells. The excitatory layer of neurons implements the associative recall dynamics with its mutual cooperation among cells belonging to the same assembly and its competition between cells which represent dierent patterns. Inhibitory cells are connected to their excitatory counterparts and serve as the inhibitory feedback element to stabilize a limit cycle in the neural activity. A schematic drawing of the model is shown in Fig. 1. The speci c neural dynamics suggested by Baird is de ned by n pairs of leaky integrator equations for the membrane potentials xi (t); yi (t), i = 1; : : : ; n of excitatory, inhibitory cells, respectively, i.e.,

dxi = dt

?xi ? hyi +

dyi = dt

?yi + gxi:

n=2 X j =1

Wij xj ?

n=2 X j;k;l=1

Wijkl xj xk xl + Ii (t);

(4)

(5)

The rate constant allows us to gauge the simulation time scale to the physiological time. Ii(t) abbreviates a time{dependent input to unit i. The pair-wise connections Wij between the excitatory units are designed to establish a set of desired patterns ; = 1; : : : ; n2 as limit cycles of the dynamics. The forth-order synapses Wijkl which are omitted in Fig. 1 for reasons of clarity stabilize these limit cycles and suppress spurious attractor states by competition. The competition mechanism resembles winner|take|all networks in its structure and functional role. The connections are chosen according to the Hebb rule

Wij =

n=2 X =1

i j ;

(6)

First Draft:


6

and the multiple outer product rule for the forth-order weights

Wijkl = cij kl ? d

n=2 X =1

i j k l :

(7)

The requirement c > d has to be satis ed for stable limit cycles [2]. With such a choice of weights the network exhibits oscillatory activity for asymptotically long times (t ! 1) of the form

0 1 0 1 j ~ j exp( i + i! t ) x ( t ) x B@ i C CA : A = B@ q yi (t) g=h j~ j exp(iy + i! t)

(8)

Equation (8) describes the recall of a single activity pattern ~ . Baird's network is not designed to recall multiple patterns as Horn's associative memory is, but to provide a mathematically tractable sketch of olfactory cortex. The phases x ; y are identical for all excitatory, inhibitory neurons, respectively, which is a consequence of the simplicity of the model. The phase lag x ? y between the two populations is nearly 900 degrees for a broad parameter range which ts nicely the experimentally measured dynamics of olfactory activity. Baird has shown that the network (4,5) is capable of storing Fourier components and suggested that complex motor patterns might be memorized in neural structures with oscillatory response. This hypothesis still has to be substantiated by experimental results from motor cortex. Recently an oscillatory network has successfully been used to recognize handwritten characters by the temporal signature of pen movement [3].

First Draft:


7

4 Pattern Segmentation in Oscillatory Associative Memory Obviously, the time domain endows the network with an additional degree of freedom which is not available to xed point attractor networks. Time can be used to label and segment patterns in neural networks which would avoid the superposition catastrophe of xed point memories. Wang et al. [14] have demonstrated that such an oscillatory neural network is able to separate two simultaneously presented patterns on a microscopic time scale of milliseconds but to preserve the information about both patterns on a psychological time scale of a fraction of a second. The network relaxes to a limit cycle representing one pattern and switches to another limit cycle after a brief period of time. Such a switching mechanism based on oscillatory activity might explain the ability of rats to discriminate dierent odors in a mixture without suppressing all fragrances except one. Binding and segmentation of pieces of information is not only a problem in associative recall in olfactory cortex but the question of feature binding appears also in early vision where no association takes place. Neurophysiological ndings of oscillatory activity in primary visual cortex [7] supports speculations that temporal correlations form the basic mechanism for information binding in cortex [13]. Wang et al. used population equations of the leaky integrator type for pools of excitatory and inhibitory neurons. The variables xi and yi are restricted to the interval [0; 1] and describe the average ring rate of excitatory or inhibitory neuron pools, respectively, i.e.,

dx x i = dt

0 1 X ?xi + Gx @Txx xi ? Txy F (yi) + Wik xk (t) + Ii ? HiA 6

k=i

(9)

First Draft:

Handbook of BT&NN, (April 22, 1994) y

dyi = dt

8

?yi + Gy (?Tyy yi + Tyxxi) ;

Hi =

Zt 0

(10)

xi ( ) exp(? (t ? ))d;

(11)

where x and y are the time constants of the excitatory and inhibitory activity of an oscillator. The architecture of the network parallels Baird's model apart from the fourth-order weights and is shown in Fig. 1. Gx and Gy are sigmoid gain functions of the form

Gr (v) = [1 + exp(?(v ? r )=r )]?1 ;

r 2 fx; yg;

(12)

with thresholds x or y and gain parameters 1=x and 1=y . Inhibitory neuron pools in uence excitatory neuron pools in a nonlinear way described by the quadratic function F (yi) = (1 ? )(yi =y) + (yi=y)2, (0 1). The tuning parameter y is used to adjust the average inhibitory activity. The delayed self-inhibition term Hi is analogous to a dynamic threshold. In addition to the interaction between excitatory units xi and inhibitory units yi which is parameterized by the weights Trs ; r; s 2 fx; yg, an excitatory unit xi receives timedependent external input Ii(t) from a sensory area or from other networks, and internal input Pk6=i Wik xk (t) from other oscillators. Let us now study how the oscillatory network performs the segmentation task. p sparsely coded, random N bit words f~ j = 1; : : : ; pg; i 2 f0; 1g; i = 1; : : : ; n are stored in the network. The probability that a bit equals 1 is a with typically a < 0:2. Similar to Horn's and Baird's model the synapses are chosen according to a Hebbian correlation rule Wik =

1 X( ? a)( ? a): i k aN

(13)

In a simulation study, 50 oscillators were used and 8 patterns (each with 8 active units) were stored in the memory. A superposition of the rst three patterns is oered to the network for associative restoration and segmentation. In Figure 2 the activity of ve representative

First Draft:


9

oscillators is monitored and the time slots in which one input component is separated from the other admixtures are marked with dashed lines. All three components are recognized as stored patterns. The exact number of patterns that can be represented simultaneously depends on details of implementation, but a reasonable estimate seems to be the 7 2 which is often cited as the storage capacity of short term memory. Oscillatory associative memories are motivated by biological ndings in olfactory and visual cortex. The merit of these networks as parallel computing devices is determined by their ability to exploit the time varying neural activity for information processing beyond associative pattern recall and pattern completion. The capability of oscillatory associative memories to segment superimposed patterns in time clearly enhances the functionality of conventional associative memories. It also demonstrates a plausible mechanism how neural assemblies might dynamically bind pieces of information together by activity correlations.

References [1] D. Amit. Modelling Brain Function. Cambridge University Press, Cambridge, 1989. [2] B. Baird. Bifurcation and learning in models of oscillating cortex. Physica D, 42:365{ 384, 1990. [3] B. Baird, W. Freeman, F. Eeckman, and Y. Yao. Applications of chaotic neurodynamics in pattern recognition. In SPIE Proceedings Vol 1469, Orlando, pages 12{23, 1991. [4] W. J. Freeman. Mass Action in the Nervous System. Academic Press, New York, 1975. [5] W. J. Freeman. Simulation of chaotic EEG patterns with a dynamic model of the olfactory system. Biological Cybernetics, 56:139{150, 1987. [6] W. J. Freeman. The physiology of perception. Scienti c American, 264:78{87, 1991.

First Draft:


10

[7] C. M. Gray, P. Konig, A. K. Engel, and W. Singer. Oscillatory responses in cat visual cortex exhibit intercolumnar synchronization which re ects global stimulus properties. Nature, 338:334{337, 1989. [8] J. Guckenheimer and P. Holmes. Nonlinear Oscillations, Dynamical Systems, and Bifurcations of Vector Fields. Springer Verlag, New York, 1983. [9] L.B. Haberly and J.M. Bower. Olfactory cortex: model circuit for study of associative memory? Trends in Neural Sciences, 12:258{264, 1989. [10] J. Hertz, A. Krogh, and R. G. Palmer. Introduction to the Theory of Neural Computation. Addison Wesley, New York, 1991. [11] J.J. Hop eld. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, USA, 79, 1982. [12] D. Horn and M. Usher. Parallel activation of memories in an oscillatory neural network. Neural Computation, 3:31{43, 1991. [13] C. von der Malsburg. The correlation theory of brain function. Internal Report, MaxPlanck-Institut fur Biophysikalische Chemie, Postfach 2841, D{3400 Gottingen, FRG, 1981. [14] D.L. Wang, J. Buhmann, and C. von der Malsburg. Pattern segmentation in associative memory. Neural Computation, 2:94{106, 1990.

First Draft:


Input I1

I2

x1

y1

11

In W21

x2

W12

xn excitatory layer

y2

inhibitory layer

yn

Figure 1: Schematic drawing of Baird's olfactory cortex model.

First Draft:


12

X2 X8 X13 X14 X19

Figure 2: Activity of an oscillatory associative memory for pattern segmentation. The activity of ve representative oscillators is monitored during pattern reccall. The oscillators 2, 8, 14 are exclusively active in the patterns 1, 2, 3, respectively. Oscillator 13 is supposed to be active in the patterns 2 and 3; oscillator 19 is part of the rst three patterns.