For activity, second order statistics may provide a reasonable description. Recent evidence suggests pairwise correlations in neuronal activity may account for ...
Toward a Second Order Description of Neuronal Networks Duane Nykamp School of Mathematics University of Minnesota
A low-dimensional description? To describe either ● ●
network activity, or network connectivity
requires a huge number of dimensions.
A low-dimensional description? To describe either ● ●
network activity, or network connectivity
requires a huge number of dimensions.
Can we extract key features of the network activity and connectivity to obtain a useful low-dimensional description?
A low-dimensional description? For activity, second order statistics may provide a reasonable description. Recent evidence suggests pairwise correlations in neuronal activity may account for observed higher order statistics in the activity. Schneidman, et al., Nature, 2006, Shlens, et al., J. Neurosci, 2006 Tang, et al., J. Neurosci, 2008, Yu, et al., Cereb Cortex, 2008
A low-dimensional description? What about for connectivity? Hypothesis: Second order statistics of connectivity may be sufficient to explain much of network behavior.
A low-dimensional description? What about for connectivity? Hypothesis: Second order statistics of connectivity may be sufficient to explain much ———– of network behavior. some
A low-dimensional description? What about for connectivity? Hypothesis: Second order statistics of connectivity may be sufficient to explain much ———– of network behavior. some Goal: Develop method to distill connectivity to its second order statistics and investigate resulting network behavior.
Second order statistics of connectivity??? What could one mean by 2nd order statistics of connectivity?
Second order statistics of connectivity??? What could one mean by 2nd order statistics of connectivity? I don’t know either, so I’ll give two answers to hedge my bets.
Second order statistics of connectivity??? What could one mean by 2nd order statistics of connectivity? I don’t know either, so I’ll give two answers to hedge my bets. Identify neurons only by their class (e.g., response to stimulus). Attempt one to define connectivity stats 1. What classes of neurons project to neuron of class A First order statistic
? ? ?
A
Second order statistics of connectivity??? What could one mean by 2nd order statistics of connectivity? I don’t know either, so I’ll give two answers to hedge my bets. Identify neurons only by their class (e.g., response to stimulus). Attempt one to define connectivity stats 1. What classes of neurons project to neuron of class A First order statistic 2. What classes of neurons simultaneously project to neurons of classes A and B Second order statistic
? ?
A
? ?
A
? ?
B
Second order statistics of connectivity??? Attempt two to define connectivity stats 1. First order statistic: Probability of an edge from a class A neuron onto a class B neuron. A
B
Second order statistics of connectivity??? Attempt two to define connectivity stats 1. First order statistic: Probability of an edge from a class A neuron onto a class B neuron. A
B
2. Second order statistics: Probability of two edge combinations. A
A
B
Reciprocal connections
B
A
C
Convergent connections
B
B
C
Divergent connections
A
C
Causal connections
The ideal (second order) scenario Form effective network based on these statistics and explore its properties Full network unknown
The ideal (second order) scenario Form effective network based on these statistics and explore its properties Full network unknown Group into populations
A
B
A
B B
A
B B
C C
C C
The ideal (second order) scenario Form effective network based on these statistics and explore its properties Full network unknown Group into populations
A
B
Track distribution of states in population
C
The ideal (second order) scenario Form effective network based on these statistics and explore its properties Full network unknown Group into populations
A
B
Track distribution of states in population Incorporate first and second order statistics of connectivity
C
Outline 1. Overview
2. Kinetic theory modeling of feed forward networks (using attempt 1 statistics)
3. Recurrent network modeling (using attempt 2 statistics)
Outline 1. Overview
2. Kinetic theory modeling of feed forward networks (using attempt 1 statistics) A developed story, but just Cliffs Notes version today 3. Recurrent network modeling (using attempt 2 statistics) Just a preface
Outline 1. Overview
2. Kinetic theory modeling of feed forward networks (using attempt 1 statistics) A developed story, but just Cliffs Notes version today 3. Recurrent network modeling (using attempt 2 statistics) Just a preface
Capturing second order statistics Neglect inter-population correlations A
B
Goal: capture intra-population correlations Tool: probability density that captures state of any pair of neurons in a population (kinetic theory)
C
Population density: ρ(v1 , v2 , t) v: voltage of integrate-and-fire neuron model, t: time
A second order kinetic theory ρ(v1 , v2 , t) captures effect of 2D random walk. B
vth
V2 (t)
C
A D
Er vreset
E
Er
V1 (t)
vth
Input: νij (t): rate that neuron pair synchronously receives i and j inputs.
A second order kinetic theory ρ(v1 , v2 , t) captures effect of 2D random walk. B
vth
V2 (t)
rsyn
C vth
V2 (t)
A D
Er vreset
rave
Jreset,1
E Jreset,3
Er
V1 (t)
vth
Input: νij (t): rate that neuron pair synchronously receives i and j inputs.
vreset V1 (t)
Jreset,2
vth
Output: ● Average firing rate rave ● Synchronous firing rate rsyn ● Cross-correlation
Continuity equation Given input rates νij , have continuity equation for ρ. ∂ρ (v1 , v2 , t) = −∇J(v1 , v2 , t) + reset terms ∂t J = Jl + J1,0 + J1,1 + J2,0 + J2,1 + J2,2 + · · · 1 2 Jl = (Jl1 , Jl2 ), J1,0 = (J1,0 , J1,0 ), · · ·
Jl1 (v1 , v2 , t) = −
v1 − Er ρ(v1 , v2 , t) τZ v1
1 J1,0 (v1 , v2 , t)
FA (v1 − θ1 )ρ(θ1 , v2 , t)dθ1
= ν1,0 (t)
1 J1,1 (v1 , v2 , t) =
vreset „Z v1
1 ν1,1 (t) 2 Z v1 Z + vreset
···
FA (v1 − θ1 )ρ(θ1 , v2 , t)dθ1
vreset v2
vreset
« FA (v1 − θ1 )fA (v2 − θ2 )ρ(θ1 , θ2 , t)dθ2 dθ1 ,
Couple populations using connectivity statistics Couple populations using the second order statistcs (attempt 1) A
B
1. Expected number of connections from population A to population B First order statistic: W
A A A
B
Couple populations using connectivity statistics Couple populations using the second order statistcs (attempt 1) A
B
1. Expected number of connections from population A to population B First order statistic: W 2. Expected fraction of AB connections shared by two B neurons Second order statistic: β
A A
B
A A
B
A A
B
Coupling pair of populations
How to couple ρA (v1 , v2 , t) with ρB (v1 , v2 , t) given second order connectivity statistics W and β?
Coupling pair of populations
How to couple ρA (v1 , v2 , t) with ρB (v1 , v2 , t) given second order connectivity statistics W and β? For any pair of neurons in population A, their average firing A rates are rave (t) and their cross-correlation is given by Cpeak (t).
Coupling pair of populations
How to couple ρA (v1 , v2 , t) with ρB (v1 , v2 , t) given second order connectivity statistics W and β? For any pair of neurons in population A, their average firing A rates are rave (t) and their cross-correlation is given by Cpeak (t). Transform to input to any pair in population B.
Test on feed forward network Test method and second order connectivity hypothesis using 10 layer feed forward network. Layer1
Layer2
Layer3
···
Hypothesis: Second order statistics of connectivity may be sufficient to explain much of network behavior.
Test on feed forward network Test method and second order connectivity hypothesis using 10 layer feed forward network. Layer1
Layer2
Layer3
··· Fix average connectivity W ; vary fraction of shared input β. Correlations will emerge in deeper layers Hypothesis: Second order statistics of connectivity may be sufficient to explain much of network behavior.
The emergence of correlations Correlations build up in the deeper layers
Layer 2 0 50
Layer 6 0 100 50 0 0
Layer 10 1 2 3 4 Time (seconds)
5
Average cross correlation (spikes/second2)
Average firing rate (spikes/second)
50
150 100 50 0 −50 150 100 50 0 −50 800 600 400 200 0 −100
0 100 Delay (ms)
Cross−correlation peak area (spikes/second)
Feed forward result summary Kinetic Theory
5 0
β = 0.05
Monte Carlo
1 2 3 4 5 6 7 8 9 10 β = 0.1
5 0
1 2 3 4 5 6 7 8 9 10 β = 0.2
5 0
1 2 3 4 5 6 7 8 9 10 Layer
Test connectivity hypothesis Goal is method to distill connectivity into key features. Hypothesized that second order connectivity statistics may be sufficient. Test by building different networks with same second order connectivity statistics but different higher order statistics.
Test connectivity hypothesis
CC peak area (spikes/second)
C 10
Kinetic Theory Monte Carlo Random
8
Monte Carlo Power Law Monte Carlo Power Law 100
6
Monte Carlo Power Law 500 Monte Carlo Power Law 2000
4
Monte Carlo Power Law 5000 Monte Carlo Gaussian
2 0 0
C 0.05
0.1 β
0.15
0.2
Outline 1. Overview
2. Kinetic theory modeling of feed forward networks (using attempt 1 statistics) 3. Recurrent network modeling (using attempt 2 statistics) More questions than answers.
Outline 1. Overview
2. Kinetic theory modeling of feed forward networks (using attempt 1 statistics) 3. Recurrent network modeling (using attempt 2 statistics) More questions than answers.
Challenges for modeling neuronal networks ●
With few exceptions, we don’t have good measurements of the network structure.
●
With few exceptions, connections are directed.
●
With few exceptions, a given neuron is either excitatory or inhibitory.
Therefore, we don’t know much about the adjacency matrix, except that it is asymmetric, some columns are non-negative, and other columns are non-positive.
Is Erd¨ os-R´ enyi good enough? We do have data from recordings of three cells. Song et al., PLoS Biology 2005 For an Erd¨os-R´enyi random graph with probability of a single edge p, the probability of any pair of edges should be p2 .
Is Erd¨ os-R´ enyi good enough? We do have data from recordings of three cells. Song et al., PLoS Biology 2005 For an Erd¨os-R´enyi random graph with probability of a single edge p, the probability of any pair of edges should be p2 . Check second-order motifs from data set: A
A
B
Reciprocal connections
B
A
C
Convergent connections
B
B
C
Divergent connections
A
C
Causal connections
Is Erd¨ os-R´ enyi good enough? We do have data from recordings of three cells. Song et al., PLoS Biology 2005 For an Erd¨os-R´enyi random graph with probability of a single edge p, the probability of any pair of edges should be p2 . Check second-order motifs from data set: A
A
B
Reciprocal connections Prob ≈ 4p2
B
A
C
Convergent connections Prob ≈ 1.4p2
B
B
C
Divergent connections Prob ≈ 1.3p2
A
C
Causal connections Prob ≈ 1.2p2
Build network based on second order motifs Goal: understand the network structure that one can predict just from these motifs. From these second order statistics, need to infer a probability distribution that characterizes the full network. Many probability distributions are consistent with these second order statistics. We’ll choose the one with the minimal additional structure – the maximum entropy network.
Build network based on second order motifs Goal: understand the network structure that one can predict just from these motifs. From these second order statistics, need to infer a probability distribution that characterizes the full network. Many probability distributions are consistent with these second order statistics. We’ll choose the one with the minimal additional structure – the maximum entropy network. A.k.a., exponential-family random graph models (ERGM).
Build network based on second order motifs Goal: understand the network structure that one can predict just from these motifs. From these second order statistics, need to infer a probability distribution that characterizes the full network. Many probability distributions are consistent with these second order statistics. We’ll choose the one with the minimal additional structure – the maximum entropy network. A.k.a., exponential-family random graph models (ERGM). In some circles, it’s known as an Ising model.
Build network based on second order motifs Goal: understand the network structure that one can predict just from these motifs. From these second order statistics, need to infer a probability distribution that characterizes the full network. Many probability distributions are consistent with these second order statistics. We’ll choose the one with the minimal additional structure – the maximum entropy network. A.k.a., exponential-family random graph models (ERGM). In some circles, it’s known as an Ising model. We’ll christen it a second-order network. . . .
Do second order network match data? Compare with Song et al. three neuron data
Do second order network match data?
data counts/model counts
Compare with Song et al. three neuron data
6 5 4 3 2 1 0
** 4
1375 579
1
2
9
* 274
33
3
4
25
5
41
6
41
* 24
17
6
3
5 4
6
7 8 9 10 11 12 13 14 15 16 network number
Just ignore the directed three cycle (11) and say model fits well.
Extrapolate to larger networks? MaxEnt/ERGM/Ising models have problems (degeneracies). Use alternative, related method (but that’s a technical detail). Get a nice probability distribution on the space of networks, parametrized by first and second order statistics.
Extrapolate to larger networks? MaxEnt/ERGM/Ising models have problems (degeneracies). Use alternative, related method (but that’s a technical detail). Get a nice probability distribution on the space of networks, parametrized by first and second order statistics. Next important set of questions to answer: ● ●
How well does this work? Does it capture important network features?
Extrapolate to larger networks? MaxEnt/ERGM/Ising models have problems (degeneracies). Use alternative, related method (but that’s a technical detail). Get a nice probability distribution on the space of networks, parametrized by first and second order statistics. Next important set of questions to answer: ● ●
How well does this work? Does it capture important network features?
Left as an exercise for the listener. . . .
Explore properties of the networks Looking in network literature to see what properties would be relevant for neuronal networks. Open for suggestions. A simple idea: synchronization of second order network (depending on the dynamics of individual neuron models as well as second order connectivity statistics). Take kinetic theory model of first part and apply it to our probability distribution on the space of networks.
Explore properties of the networks Looking in network literature to see what properties would be relevant for neuronal networks. Open for suggestions. A simple idea: synchronization of second order network (depending on the dynamics of individual neuron models as well as second order connectivity statistics). Take kinetic theory model of first part and apply it to our probability distribution on the space of networks. Left as an exercise for the network working group.
Explore properties of the networks Looking in network literature to see what properties would be relevant for neuronal networks. Open for suggestions. A simple idea: synchronization of second order network (depending on the dynamics of individual neuron models as well as second order connectivity statistics). Take kinetic theory model of first part and apply it to our probability distribution on the space of networks. Left as an exercise for the network working group. Or the neuron working group.
Explore properties of the networks Looking in network literature to see what properties would be relevant for neuronal networks. Open for suggestions. A simple idea: synchronization of second order network (depending on the dynamics of individual neuron models as well as second order connectivity statistics). Take kinetic theory model of first part and apply it to our probability distribution on the space of networks. Left as an exercise for the network working group. Or the neuron working group. (Sorry to neglect the molecular motors working group.)
Trying to find something that even I can do Cool result from Restrepo, Ott, and Hunt, 2005-2006. Given a network of SDEs dy (j) (t) = F (Y (j) (t), µj )dt + dW (j) (t) + K(y (j) (t))
N X
Ajm [Q(y (m) (t)) − hQ(y)i]dt
m=1
If scale up connectivity strength K, the point where synchrony emerges depends on the product λl M . λl : largest eigenvalue of adjacency matrix A. M : quantity depending only on individual node model (not on network)
Eigenvalues of balanced second-order network Simple starting point, a network with:
I
E
Excitatory and inhibitory input to each neuron is balanced.
E E
I
E
I I
E I
Im(λ)
N = 400 excitatory neurons and N = 400 inhibitory neurons ● E to E connections form a second-order network ● Erd¨ os-R´enyi for connections involving an I neuron
●
I
Realistic and simplifies spectrum Rajan and Abbott 2006 Re(λ)
Eigenvalues of balanced second-order network
Look at spectrum as add reciprocal connections,
Eigenvalues of balanced second-order network
Look at spectrum as add reciprocal connections, convergent and divergent connections,
Eigenvalues of balanced second-order network
Look at spectrum as add reciprocal connections, convergent and divergent connections, and causal connections. Causal connections give largest eigenvalue. Do they help networks to synchronize?
Synchronization results 0.5
Synchrony
0.4
Tests with heterogeneous networks of Kuramoto oscillators confirm results.
Independent Reciprocal only No causal Full model
0.3
0.2
Increasing likelihood of causal connections enhances synchrony.
0.1
0 0
0.1
0.2 0.3 Connectivity strength
0.4
0.5
Conclusions ●
Developed a low-dimensional description of connectivity.
●
Made two different attempts at characterizing second-order statistics of connectivity.
●
For feed forward networks, the second-order connectivity statistics determine the second-order activity statistics.
●
For recurrent networks, causal connections seem to play a role in synchrony.
●
Estimate connectivity statistics from data?
Thanks Graduate students Chin-Yueh Liu Liqiong Zhao A
B
A
B B
A
B
C C
C
A
B
B
Collaborator Michael Buice C
C
Funding Source National Science Foundation
