Processing of Recursive Sentence Structure: Testing Predictions from a Connectionist Model Morten H. Christiansen and Maryellen C. MacDonald Program in Neural, Informational and Behavioral Sciences University of Southern California Los Angeles, CA 90089-2520
[email protected] [email protected]
Abstract We present results from three psycholinguistic experiments which tested predictions from a connectionist model of recursive sentence processing. The model was originally developed to capture generalization using non-local information (Christiansen, 1994; Christiansen & Chater, 1994). From this model it was possible to derive novel empirical predictions concerning the processing of di erent kinds of recursive structure. We present behavioral results con rming network predictions about the acceptability of sentences involving multiple right-branching PPs (Experiment 1), multiple left-branching prenominal genitives (Experiment 2), and doubly center-embedded object relative clauses (Experiment 3). Importantly, these predictions derive from the intrinsic architectural constraints of the model (Christiansen & Chater, in submission), rather than arbitrary, externally speci ed memory limitations. We conclude that the SRN is well-suited for the modeling of human performance on recursive sentence structure.
1 Introduction
One way to evaluate computational models of psycholinguistic phenomena is to assess how well they match behavioral data and whether they make predictions beyond existing data. Many models only match data at a fairly gross level of performance, and few make predictions that inspire new experiments. This is true of both connectionist and symbolic computational models of language{especially within the area of sentence processing. We introduce a performance measure, Grammatical Prediction Error (GPE), which allows for the modeling of grammaticality ratings. We use this measure to derive novel empirical predictions from an existing connectionist model of the processing of recursive sentence structure (Christiansen, 1994; Christiansen & Chater, 1994). The predictions suggest that increasing depths of recursion not only decrease the acceptability for center-embedded constructions, but also for the simpler left- and rightbranching constructions. These predictions are at odds with many symbolic models of sentence processing. Results from three behavioral experiments are presented, con rming the model's predictions.
2 Connectionist Simulations
The predictions were derived from the Simple Recurrent Network (SRN, Elman, 1990) model of recursive sentence processing developed by Christiansen (1994; Christiansen & Chater, 1994). 1
Output (42 units)
copy-back Hidden (150 units)
Input (42 units)
Context (150 units)
Figure 1: The basic architecture of the SRN used in Christiansen (1994). Arrows with solid lines denote trainable weights, whereas the arrow with the dashed line denotes the copy-back connections. The SRN, as illustrated in Figure 1, is essentially a standard feedforward network equipped with an extra layer of so-called context units. The hidden unit activations from the previous time step are copied back to these context units and paired with the current input. This means that the current state of the hidden units can in uence the processing of subsequent inputs, providing the SRN with an ability to deal with integrated sequences of input presented successively. The SRNs were trained via a word-by-word prediction task on 50,000 sentences (mean length: 6 words; range: 3-15 words) generated by the context-free grammar in Figure 2 (using a 38 word vocabulary). This grammar involved left recursion in the form of prenominal genitives, right recursion in the form of subject relative clauses, sentential complements, prepositional modi cations of NPs, and NP conjunctions, as well as complex recursion in the form of object relative clauses. The grammar also incorporated subject noun/verb agreement and three additional verb argument structures (transitive, optionally transitive, and intransitive). The generation of sentences was further restricted by probabilistic constraints on the complexity and depth of recursion.
- NP VP \." - PropN j N j N rel j N PP j gen N j N and NP - V(i) j V(t) NP j V(o) (NP) j V(c) that S - who NP V(tjo) j who VP - prep locN (PP) - (gen) N +\s"
S NP VP rel PP gen
Figure 2: The small context-free grammar from used to generate the training corpus.
2.1 Deriving Predictions from the SRN Model
When evaluating how well the SRN has learned regularities of the grammar, it is important from a linguistic perspective not only to determine whether the words that were activated given prior context are grammatical, but also which items were not activated despite being sanctioned by the grammar. The GPE provides an activation-based measure of how well a network is obeying the training grammar in making its predictions, taking hits, false positives, correct rejections as well as false negatives into account. The GPE for predicting a particular word was calculated using: 2
GPE = 1 ?
hits hits + false positives + false negatives
Hits and false positives consisted of the accumulated activations of all units that were grammatical (G) and of all activated units that were ungrammatical (U ), respectively. hits =
X i2G
X
ui false positives =
i2U
ui false negatives =
X i2G
fni
False negatives were calculated as a sum over the (positive) discrepancy fni between the desired activation for a grammatical unit ti and the actual activation of that unit ui. (
positives)fi ti = (hits + false P f
t i ? ui 0 fni = 0ti ? ui ifotherwise
j 2G j
The desired activation, ti , was computed as a proportion of the total activation determined by the lexical frequency fi of the word that ui designate and weighted by the sum of the lexical frequencies fj of all the grammatical units. The GPE for an individual word re ects the diculty that the SRN experienced for that word given the previous sentential context, and can be mapped qualitatively onto word reading times, with low GPE values re ecting a prediction for short reading times and high values indicating long predicted reading times (MacDonald & Christiansen, in submission). The average GPE across a whole sentence expresses the diculty that the SRN experienced across the sentence as a whole, and have been found to map onto sentence grammaticality ratings (Christiansen & Chater, in submission), with low average GPE scores indicating a low acceptability rating and high scores re ecting high ratings. Using the GPE we were able to derive novel predictions concerning the acceptability of three types of sentences involving complex recursive constructions from the existing model by Christiansen (1994; Christiansen & Chater, 1994).
3 Testing SRN Predictions
A number of predictions regarding the processing of recursive sentence structure were derived from the model. Here we focus on the processing of sentences involving multiple instances of three di erent kinds of recursion: right-branching, left-branching, and center-embedding. These predictions from the SRN model were tested in on-line grammaticality judgment experiments using a self-paced reading task with word-by-word center presentation. Following the presentation of each sentence, subjects rated the sentence on a 7-point scale (7 = bad).
3.1 Experiment 1: Multiple PP Modi cations of Nouns
Increasing the number of recursions in right-branching constructions involving an NP modi ed by several PPs should make the sentences less acceptable. Subjects were presented with the sentences with 1 PP (1), 2 PPs (2), or 3 PPs (3):
SRN Prediction:
3
Mean Rating (7 = Bad)
7 6 5 4 3 2 1 1 PP
2 PPs
3 PPs
Sentence Type
Figure 3: The mean ratings for sentences incorporating 1, 2 or 3 PPs into an NP (Experiment 1). (1) The nurse with the vase says that the owers by the window resemble roses. (1 PP) (2) The nurse says that the owers in the vase by the window resemble roses. (2 PPs) (3) The blooming owers in the vase on the table by the window resemble roses. (3 PPs) The results in Figure 3 show that there was a signi cant e ect of depth of recursion in the direction predicted by the SRN model ( 1(2 70) = 10 87 0001; 2(2 16) = 12 43 001; N=36). F
;
:
;p < :
F
;
:
;p < :
3.2 Experiment 2: Multiple Prenominal Genitives
Having two levels of recursion in an NP involving left-branching prenominal genitives should be less acceptable in an object position than in a subject position. In this experiment, subjects were presented with sentences containing multiple prenominal genitives either in the subject position (4) or in the object position (5): SRN Prediction:
(4) Jane's dad's colleague's parrot followed the baby all afternoon. (subject) (5) The baby followed Jane's dad's colleague's parrot all afternoon. (object) As predicted by the SRN model, the results in Figure 4 show that multiple prenominal genitives were less acceptable in object position than in subject position ( 1(1 33) = 5 76 03; 2(1 9) = 3 48 1; N=34). F
F
;
:
;
:
;p
