Active Maintenance of Sentence Meaning in Working Memory

Running title & shortened title: active maintenance of sentence meaning

Active Maintenance of Sentence Meaning in Working Memory Henk J. Haarmannab, Katherine A. Cameronc, and Daniel S. Ruchkind a b

Department of Hearing and Speech Sciences

Neuroscience and Cognitive Science Program University of Maryland College Park, Maryland, USA c

Department of Psychology Washington College

Chestertown, Maryland, USA d

Department of Physiology School of Medicine University of Maryland

Baltimore, Maryland, USA (research site) July 16, 2002 (Submitted for publication) Corresponding author: Henk J. Haarmann, Ph.D. Department of Hearing and Speech Sciences College Park, Maryland 20742, USA Tel: 301-405-4229 fax: 301-314-2023 E-Mail: [email protected]

1

ABSTRACT The active maintenance of working memory for sentence meaning was investigated, using eventrelated brain potentials, event-related coherences, and source analysis. Participants read a sentence, retained it for 2.5 seconds, and verified a statement about its meaning. The sentences imposed a high processing demand on working memory and contained either three semantically related or unrelated nouns. Comprehension accuracy was lower in the unrelated than related condition. Both event-related brain potentials and coherences showed sensitivity to the semantic relatedness manipulation during retention, namely, decreased slow wave negativity across bilateral posterior-central scalp and increased coherence in the unrelated compared to related condition in the 10-14 Hz band. In addition, compared to a pre-sentence baseline and sentence presentation, coherences increased in the 10-14 Hz band during retention and decreased in the 46 Hz band. These findings suggest that the short-term retention of the meaning of a sentence incurs a demand on working memory, which is greater for sentences with semantically unrelated than related words. The coherence changes spanned pre-frontal and posterior brain regions, possibly reflecting increased synchronization in projection loops between attention control systems in pre-frontal cortex and activated meaning representations in semantic memory in posterior cortex. The source analysis revealed that the approximate same subset of brain regions was active during both sentence presentation and retention. These source results are consistent with activation-based network models of working memory for sentence meaning and with a proceduralist view of memory, according to which brain regions that subserve processing also support storage. Keywords: Semantic relatedness, Semantic binding, Sentence comprehension, Eventrelated brain potentials, Electroencephalogram coherences, Neural synchronization. 2

INTRODUCTION Comprehending a sentence involves a process of semantic binding [26], whereby the meanings of the words in a sentence are related to one another

and actively

maintained in working memory as part of an integrated overall representation. There are at least two major types of semantic relations that play a role in this process: pre-existing and dynamically computed semantic relations. Pre-existing semantic relations occur among semantically associated and related words (e.g., pilot and airport). They exist in semantic long-term memory prior to the processing of a sentence. Dynamically computed semantic relations occur between words that function as semantic role assigners and receivers and are computed on-line during sentence processing on the basis of structuralsyntactic information. A prototypical example of dynamically computed semantic relations involves the thematic role relations which a verb assigns to its surrounding nouns. For example, in the sentence, “The senator entered the airport”, the verb “entered” assigns the thematic roles of agent and location to senator and airport, respectively. A further example of a dynamically computed semantic relation is the attribute that an adjective assigns to a noun (e.g., “new airport”).

Much of the psycholinguistic research on sentence comprehension has focused on investigating the properties of the first phase of semantic binding, that is, the on-line integration of a word’s meaning into the prior sentence context [52]. When two semantically associated words follow one another in close proximity in a sentence, the second word is activated more quickly and integrated more easily into the preceding sentence context [11,43,44,53,54]. In addition, when a sentence is structurally less

3

complex, readers are faster at assigning thematic roles. The latter is indicated by on-line verb reading times which are faster in subject-relatives than in structurally more complex object-relatives, which have a less typical subject-object-verb word order [29].

However, there has been virtually no psycholinguistic research investigating the properties that govern the second phase of semantic binding, that is, the active maintenance of semantic relations following on-line meaning integration during a sentence and following the offset of the sentence. There is evidence for a postinterpretative process that maintains thematic role relations following their on-line syntactic computation [10].

This process is more error prone and/or engenders more

brain activation when a sentence expresses more propositions [8]. This finding could indicate that propositions and the thematic role relations they express are maintained by a capacity-limited semantic working memory process that operates upon and stores word meanings. To provide evidence for such a process, we used the continuous nature and temporal sensitivity of event-related brain potentials (ERPS) to demonstrate an effect of semantic relationships within a sentence in a brief unfilled retention interval following the presentation of the sentence and prior to a sentence-verification response. In particular, we investigated the effect of the presence versus absence of associative semantic relations among the words in a sentence on retention of sentence meaning.

Our use of a semantic relatedness manipulation was motivated in part by two activation-based, computational models which compute and maintain the thematic role bindings of a sentence in working memory [24,28]. One model has been used to simulate

4

complexity effects in the sentence comprehension of adults with aphasia [24], while the other model has been used to explain effects in tasks of verbal analogical reasoning [28]. We focus our discussion here on the properties that are common to both models. In both models, the meaning of a sentence is represented by a network of interlinked nodes, which represent noun meanings, thematic roles of verbs, and the thematic role bindings among them. For example, the sentence fragment “The pilot entered the airport” would be represented by nodes for the noun meanings of “actor” and “airport”, by nodes for the actor and location roles of the verb “entered”, and by higher-level nodes representing the semantic binding of “pilot” to the agent node and of “airport” to the location node. In both models, the network of interlinked nodes is gradually activated through a process of spreading activation in a capacity-limited working memory. This process includes the activation of thematic role bindings from noun and verb meanings. Consistent with results from the semantic priming literature [40], one of the models moreover assumes that spreading activation also occurs between words that are semantically associated [28]. As a result, words that are semantically associated support one another’s activation. Moreover, their enhanced activation makes it easier to activate the higher-level nodes representing thematic role bindings and to maintain their activation in working memory. We therefore reasoned that when words in a sentence are weak rather than strong semantic associates of one another it will be more difficult to actively maintain the meaning representation of a sentence in working memory, including its pre-existing and dynamically computed meaning relations. Accordingly, we expected an effect of degree of semantic association among the words in the sentence on brain activity during retention and on a subsequent behavioral test of retention efficacy. Demonstrating such

5

an effect upon brain activity during retention would be evidence of an active system for the short-term storage of semantic bindings. As one approach to this issue, an ERP amplitude measure was used to detect whether brain activity during retention is affected by semantic relationships encountered during sentence processing. ERPs can provide precise information about the timing of the processing of semantic relationships, and a rough indication of where in the brain the semantic effects occur.

However, ERPs cannot provide much information about how different brain regions interact during retention and how semantic associations among the words in the sentence affect the interactions. Cowan [13,14] has argued from behavioral data and theoretical considerations that short-term retention of information involves interactions between attentional control systems located in pre-frontal cortex and long-term memory representations in posterior cortex. Based upon neurophysiological recordings of single unit activity in primates, Fuster and co-workers [19] have similarly argued that the shortterm memory process depends upon the operation of projection loops between frontal and posterior cortex.

The loops projecting from frontal cortex mediate the focusing of

attention upon representations in posterior cortex to be maintained in short-term memory, while the projections from posterior cortex provide information about the state of posterior cortical systems to frontal neural networks. Presumably the resulting influence of frontal and posterior systems upon each other is actualized by an increase in synchronization between neural circuits in the two brain regions. Support for this view is provided by a study of verbal and visual-spatial short-term memory tasks in which the synchronization between electroencephalographic (EEG) recordings from different scalp

6

sites was analyzed with EEG coherence measures [50]. The patterns of coherence between EEG recordings from frontal and posterior sites were found to differ markedly between the stimulus presentation and subsequent retention intervals.

Given the

sensitivity of EEG coherence to short-term memory processes and its potential to reveal patterns of interactions among neural systems [21,37,50,56,57], EEG coherence was used to seek further evidence of an active system for the short-term storage of semantic bindings

A final aim of our study was motivated by Crowder’s (Crowder, 1993) proceduralist view of memory, namely that the brain systems that process particular items of information also subserve storage of those items. Thus we investigated whether some of the brain regions that contribute to the phrase-by-phrase processing of a sentence, also contribute to the active maintenance of the meaning of a sentence in working memory. For empirical as well as theoretical reasons, we hypothesized that a subset of the brain regions that contribute to the phasic, phrase-by-phrase processing of a sentence also support the active maintenance of the meaning of a sentence in short-term memory during a brief interval following presentation of the sentence. On theoretical grounds, the activation-based network models of sentence comprehension, discussed above, assume a tight integration between the processing and maintenance functions of working memory. Processing consists of activation of nodes in a network through spreading activation, while maintenance consists of sustained activation of the same nodes through continued spreading activation among the nodes. Thus, the parts of the brain that underlie the nodes in such a network would be active both during on-line sentence processing and sentence

7

retention in working memory. On empirical grounds, hemodynamic studies of retrieval of sensory and verbal information [42,58] have demonstrated that a subset of brain regions that were active during the initial processing of information were activated during subsequent remembering of the information. We addressed the issue of whether brain regions involved in sentence comprehension also contribute to subsequent retention of sentence meaning by spatial-temporal source modeling of the ERP data [51] (BESA). This approach provides approximate locations and time courses of activation of the brain areas contributing to the ERP activity recorded from the scalp during and after sentence presentation.

8

METHODS Participants Sixteen young, healthy adults participated in the experiment. The average age was 29 years (range, 23-46). All participants were college educated, right-handed (average laterality quotient on the Edinburgh handedness inventory [45] was .79), female, native speakers of English and had normal or corrected-to-normal vision. Informed consent was obtained in accord with the procedures of the U.S.P.H.S. Participants were paid $18.00/hour. Materials The sentences were Wh-object filler-gap sentences (henceforth filler sentences). A subject, object, and sentence final word in these sentences could be either semantically related or unrelated. There were 40 trials in each of these two conditions. Sentences (1) and (2) below illustrate a filler, related sentence and a filler unrelated sentence. (1) What box/ did the pilot/ that entered/ the airport/ leave/ in the plane? (Related) (2) What box/ did the actor/ that entered/ the airport/ leave/ in the shop? (Unrelated) The forward slashes were not shown to the subject but demarcate phrases, which were presented visually one at a time in immediate succession. The bold-faced nouns highlight the three related/unrelated nouns. All words were shown in regular font to the subjects. Filler sentences were question sentences, consisting of six phrases: A Wh-object phrase (e.g., What box), three phrases of a filler-gap interval (e.g., did the pilot / that entered / the airport), a main verb (e.g., leave), and a sentence-final, prepositional phrase (e.g., in the plane?). The sentence is called a Wh-object filler-gap sentence because the Wh-

9

object phrase fills a lexically empty object gap, which occurs immediately after the main verb, allowing the Wh phrase to be integrated with the main verb as its direct object. In the related condition, the subject noun of the main clause (e.g., pilot) was semantically related to the direct object noun in the relative clause (e.g., airport) and the sentence-final noun (e.g., plane). In the unrelated condition, this semantic relation was absent (e.g., What equipment did the gardener that mailed the rent use at the door?). Each main clause subject and each relative clause object occurred once in the related condition and once in the unrelated condition. Moreover, sentences in the unrelated condition were created by selecting each of their successive phrases from a different stimulus question in the related condition without replacement. As a result, stimulus questions in the related and unrelated condition were matched for lexical content. To check the semantic relatedness manipulation, a separate group of eight University of Maryland students were asked to generate four nouns that were semantically related to the subject nouns used in the sentences. On average, relative clause object nouns were generated by a larger proportion of subjects in the related than unrelated condition (42% versus 1.3%, respectively, t(78) = 8.333, p < .001) the same was true for sentence-final nouns (29% versus 1.3%, respectively, t(78) = 7.741, p < 0.001).

In addition, to the filler sentences, there were non-Wh-object filler-gap control sentences (henceforth non-filler sentences), 40 in the related condition and 40 in the unrelated condition. Non-filler sentences started with phrases and words making up the filler-gap interval and main verb in filler sentences (e.g., Did the pilot / that entered / the airport / leave) and continued with a main clause object (e.g., the box) and a prepositional

10

phrase (e.g., in the airport). For each filler sentence, there was a non-filler sentence with the same lexical content words (except the words what and the in the main clause object). The non-filler sentences served as control sentences in previously presented analyses of the ERP amplitudes and EEG coherences during on-line sentence processing. Those analyses were aimed at demonstrating that the meaning of the Wh-object filler is active and incurs a demand in working memory during the filler-gap interval and during the process of gap-filling [21,22].

We paired each stimulus question with a verification statement (e.g., What conflict did the lifeguard that wore the whistle observe on the beach? Someone witnessed a disagreement. True/False? Correct answer: True). Within each condition, half of the verification statements made a true statement about the question sentence and half a false statement (assuming the events in the stimulus question really happened). In addition, within each condition half of the verification statements probed into the verb-meaning relations of the main clause (including the meaning relationship between the verb and the object filler) and half into verb-meaning relations of the relative clause. Verification statements could contain synonyms, super-ordinate terms, paraphrases, or associates of words in the stimulus question, but never repeated them verbatim. These properties and also the length of the verification statements were matched across the different conditions and were meant to encourage participants to encode and retain the meaning of each noun and each noun-verb relation in the sentence. We also created sentence materials for practice purposes, using a different set of nouns.

11

Procedure Participants were seated in a dimly lit, electrically shielded sound-attenuating chamber. The sentence stimuli were presented on a computer monitor, located 140 cm in front of the participant, outside the room, and viewed through a window. Letters were white on a dark gray background and on average 0.5o high x 0.2o wide. The trial began when the subject moved their right index finger into the path of the center beam of a three-beam optical sensor switch. After a 1,000-ms delay, the six successive phrases of the stimulus question were presented one at time, each starting at the left-hand side of the screen. Content words were presented longer (i.e., 400 ms) than function words (250 ms) to approximate a more natural reading rate. Each phrase was presented for the total duration of its words (e.g., 900 ms for the phrase Did the pilot) followed by a 50-75 ms inter-phrase interval (The interval between the penultimate phrase of the filler-gap interval and the gap was 50 ms, all other inter-phrase intervals were 75 ms). The last phrase of the stimulus question was followed by 2,475-ms delay interval (henceforth retention interval), which in turn was followed by the presentation of the entire verification statement (See Fig, 1 for the timing of trial events). Participants silently read the stimulus question and verification statement and had to move their right index finger to the right to indicate that the verification statement was true and to the left to indicate that the statement was false. Participants were instructed to assume that the events in the stimulus question really happened and to respond with true only when the verification statement followed necessarily from the meaning of the stimulus question. They were furthermore told that performance accuracy was more important than response speed. To ensure that there were no overt movements, a closed circuit TV system was used to

12

observe the participant. Each participant performed 72 practice trials in a training session and 12 practice trials and 160 experimental trials in an experimental session. With the exception of the first 8 practice trials in the training session, participants received no correct/incorrect feedback. The order of the practice and experimental trials was randomized with the constraint that sentences belonging to a particular condition were separated by at least one trial. Semantic relatedness (unrelated, related) and sentence type (filler, non-filler) were manipulated within-subjects. The sequence of trials was the same for each participant. Participants received a short break (about 5 min) after every 32 trials. -------------------------------------------------Insert Fig. 1 approximately here --------------------------------------------------EEG recording Ag/AgCl electrodes were placed on occipital-cerebellar scalp (Cb1, O1, O2, Cb2)1, parietal scalp (T5, P3, Pz, P4, T6), central scalp (T3, C3, Cz, C4, T4), frontal scalp (F7, F3, Fz, F4, F8), prefrontal scalp (Fp1, Fp2), two cm below the outer canthi of the eyes (E1, E2), and on the temporal-central midline one cm below the tragus (A1, A2) (See Fig. 1 for a layout of the electrodes across the scalp). The A1 electrode was the reference for the recordings from the other 24 electrodes. The amplifiers were set to a gain of 10000, an upper cutoff frequency (-3dB) of 30 Hz and an AC coupling time constant of 5.3 sec (-3 dB attenuation frequency: .03 Hz). For each trial, the EEG was digitized over a 7,500 ms interval, beginning 500 ms prior to the onset of the stimulus sentence, continuing for 4525 ms during the entire sentence, and lasting until the end of the retention interval. The sampling rate was 50 samples/sec (sampling interval = 20 ms).

13

Participants were not required to make special efforts to suppress blinks and eye movements. Eye movement artifacts were removed via a spatial-temporal modeling procedure [5,32,33], implemented in Scherg’s brain source localization program BESA (for a detailed description see [49]. As a conservative measure, we excluded trials with very large and prolonged eye movement activity from the average ERPs as well as trials with artifactually high amplitudes (i.e., greater than 50 µV). The percentage of discarded trials was 23% of the total trials on average and about equal in each of the four experimental conditions (22 to 24%, F< 1). Analysis of amplitude measures A calibrated amplifier ground was used for the conversion of the ERPs obtained with the 5.3 sec time constant to wave forms approximating those that would have been obtained by DC recording [48]. The ERPs were digitally converted to linked A1 and A2 reference wave forms, and further smoothed with a zero phase shift low-pass digital filter (-3 dB frequency: 4.4 Hz). For each participant, average ERPs for correct response trials were computed at all electrode sites for filler sentences in the related and unrelated condition. We used the average amplitude over the 500 ms epoch prior to the onset of the stimulus question as a baseline for slow wave effects during a sentence-end and retention epoch. ERP amplitudes were measured by computing the average amplitude over each of two latency ranges comprising these two epochs. The sentence-end epoch spanned the final 1500 ms of the sentence (starting 3040 ms into the presentation of the sentence and ending 15 ms after its offset at 4540 ms), with the exception of a time interval in its middle (from 3900 to 4260ms into the presentation of the sentence) that included an N400 epoch (see results for amplitude measures). The retention epoch started 515 ms

14

after the onset of the retention interval, lasted 1900 ms, and ended 60 ms before the offset of the retention interval (see Fig. 1). By starting the retention epoch 515ms after the onset of the retention interval, we were able to avoid the contaminating influence of difficultto-remove eye blinking artefacts, occurring immediately after the offset of the sentence.

For statistical analysis, repeated measures analyses of variance (ANOVAs) were conducted on the ERP amplitudes separately for the retention and sentence-end epochs with Semantic relatedness (levels: related, unrelated) and Electrode site (levels: 23 channels) as factors. Corrected degrees of freedom for the ANOVA were obtained by multiplying the original degrees of freedom by the Greenhouse-Geisser epsilon when applicable and only corrected P values are reported. Significant interactions (p