6-month-old infants' processing of multisensory stimulus compounds. The infants .... the compound stimulus in terms of the intersensory relation- ship between ...
Copyright 1988 by the American Psychological Association, Inc. 0012-1649/88/$00.75
Developmental Psychology 1988, Vol. 24, No. 2, 155-171
Sensory Dominance in Infants: 1. Six-Month-Old Infants' Response to Auditory-Visual Compounds David J. Lewkowicz New York State Office of Mental Retardation and Developmental Disabilities Institute for Basic Research in Developmental Disabilities, Staten Island, New York To investigate sensory dominance in early development, a series of studies was conducted to examine 6-month-old infants' processing of multisensory stimulus compounds. The infants werefirsthabituated with a compound stimulus consisting of aflashingcheckerboard and a pulsing sound. To assess attention to aspects of the compound stimulus, the infants received separate test trials where compounds differed in the rate and/or duration at which the visual, the auditory, or both components were presented. One consistent finding was that the infants discriminated changes in the temporal characteristics of the auditory component but not in the visual component. Their responsiveness to the auditory information depended on the number of discriminative cues available during either the habituation or the test phases and on the temporal distinctiveness of the auditory and visual components during the habituation phase. This consistent failure to respond to changes in the visual component led to the conclusion that auditory dominance was operating. This conclusion was reinforced by thefindingthat the infants failed to discriminate a change in the rate of the visual component even when the intensity of the visual component was increased relative to that of the auditory component, and by the finding that the infants could discriminate temporal changes in the visual component following habituation with just the visual component.
infants' detection of auditory-visual equivalence, for example, investigators have used the paired-comparison method. Typically, in such studies infants are presented with two different visual stimuli, side by side, and their preference for the visual stimulus that corresponds to a concurrently presented sound is measured. With regard to infants' detection of temporally based equivalence, these studies have shown that infants as young as 4 months of age exhibited a visual preference for a moving object whose temporal properties corresponded to the temporal properties ofa concurrent sound (Spelke, 1979; 1981; Spelke, Born, & Chu, 1983; Walker-Andrews, 1986), whereas others have shown that infants of this age did not exhibit a visual preference for a spatially static (flashing) visual stimulus whose temporal properties corresponded to the temporal properties of a concurrent sound (Lewkowicz, 1985a). Detection of intersensory equivalence between a static visual stimulus and a corresponding sound has, however, been demonstrated in 6-monthold infants (Lewkowicz, 1986). Although the studies of infants' detection of intersensory equivalence have contributed important information about the development of intersensory mechanisms, no studies to date have explored the possible existence of sensory dominance and its relationship to intersensory functioning in early human development. The importance of considering sensory dominance hierarchies in relation to intersensory processes was noted a number of years ago by Birch and Lefford (1967) in their classic monograph on the relationship between visual differentiation, intersensory integration, and motor control. Birch and Lefford proposed the operation of three related and concurrently active processes responsible for the development of afferent control of directed action: (a) a developmental shift in sensory dominance hierarchies characterized by a shift from proximoreceptor control to teloreceptor control, (b) increasing tendency for intersen-
An infant's everyday world consists of multidimensional objects and events. The multiple dimensions characteristic of these objects and events can be represented either within a single sensory modality or across several modalities. For example, within a single sensory modality, such as vision, objects and events can be distinguished on the basis of shape, color, size, and orientation. Across several sensory modalities objects can be specified by any combination of modality-specific properties, such as color, pitch, or smell, and by amodal properties (properties that are common to more than one modality), such as shape, spatial extent, rate, duration, intensity, and synchrony. Despite the fact that objects and events that are specified by multisensory attributes are ubiquitous in the infants' everyday world, the vast majority of studies of perceptual development have focused on infants' responsiveness to single dimensions of stimulation within either the visual or the auditory modality (Aslin, Pisoni, & Jusczyk, 1983; Banks & Salapatek, 1983; Olson & Sherman, 1983). In the past several years, however, investigators have become increasingly interested in the question of infants' response to multisensory attributes, and a small but steadily growing body of work has emerged around this question. Most of this work has been concerned with infants' perception of intersensory equivalence. In studies examining This work was supported in part by a grant from the National Foundation/March of Dimes and was performed while David Lewkowicz was at Northwestern Medical School and the Evanston Hospital, Evanston, Illinois. I thank Ann Dolinko for her assistance in data collection. Correspondence concerning this article should be addressed to David J. Lewkowicz, New York State Institute for Basic Research in Developmental Disabilities, 1050 Forest Hill Road, Staten Island, New York 10314. 155
156
DAVID J. LEWKOWICZ
sory integration, and (c) intrasensory differentiation. Although we already have some information about the development of intersensory integration and a good deal of information on intrasensory differentiation, we have virtually no information about sensory hierarchies and possible shifts in these hierarchies during development. There is no question that sensory dominance is the rule in perception. Work with adults and with children has consistently shown that the visual modality dominates the auditory modality across a variety of tasks (Colavita, 1974; Colavita & Weisberg, 1979; Egeth & Sager, 1977; Hermelin & O'Connor, 1964; O'Connor & Hermelin, 1965). Information on whether visual dominance is also present in early human development is not available. It is quite likely, however, that sensory dominance is present in early development, because both the structural and functional development of the different sensory systems is asynchronous (Bronson, 1982; Gottlieb, 1971; Kasatkin, 1972; Volokhov, 1968). Contrary to adults, however, auditory dominance might be expected in early development because the auditory modality develops and differentiates earlier than does the visual modality. To determine whether sensory dominance is present in early development, 6-month-old infants' processing of multisensory compound stimuli was studied in a series of experiments. The general procedure used throughout the experiments consisted of two phases. The first was the habituation phase during which an auditory-visual compound stimulus was repeatedly presented. As soon as the last habituation trial was completed, the test phase began. Because the purpose of the test phase was to find out to which aspects of the compound stimulus the infants were attending, three types of test trials were administered. In all three test trials some temporal aspect of the compound stimulus was changed. In one test trial the rate of both components was changed, in another the rate of the auditory component was changed while the rate of the visual component remained the same, and in the third the rate of the visual component was changed while the rate of the auditory component remained the same. To determine what processes were specifically responsible for the infants' responses, the relationship between the auditory and visual components and the number of cues differentiating the components was varied across the different studies. The compound stimulus in all of the experiments consisted of a flashing checkerboard and a pulsing tone. These types of stimuli were used for two reasons. First, it was felt that such stimuli would permit the greatest experimental control over the properties of the components as well as over the possible confounding effects of preferences that are usually intrinsic to more familiar stimuli. Second, it was felt that they would be better suited for future studies with infants of different ages, because such stimuli are less likely to be affected by developmental experience than are more common stimuli. Experiment 1 The purpose of the first experiment was to determine how infants respond to a multisensory compound consisting of components that are united by virtue of being presented at the same rate during the habituation phase. Three different types of outcomes might have been expected. First, if the infants encoded the compound stimulus in terms of the intersensory relation-
ship between the two components, they would have dishabituated in both single-component test trials but not in the twocomponent test trial. This was because it was only in the two single-component test trials that the intersensory relationship changed from a temporally concordant one to a temporally discordant one. Second, ifthe infants encoded the compound stimulus in terms of its holistic properties, they would have exhibited dishabituation of response in all three test trials because the holistic nature of the compound stimulus changed in all three. Finally, if sensory dominance determined responsiveness, dishabituation of response would have been expected in the twocomponent test trial and in either the auditory test trial or the visual test trial, depending on whether the auditory or the visual modality dominated responsiveness. Method Subjects, The subjects in thefirstexperiment were 24 six-month-old infants (10 boys, 14 girls); they ranged in age from 26 to 30 weeks (M age = 28 weeks, 4 days). An additional 20 infants were tested but were excluded from data analysis; 18 were excluded because of fussing or crying and 2 because of distracting conditions. All of the infants in this experiment as well as in all subsequent experiments were full-term at birth (i.e., greater than 2,500 g, and greater than 37 weeks gestational age at birth), with uncomplicated perinatal histories, and in good health at the time of testing. Apparatus and stimuli. During testing, the infant sat in a high chair and was presented with a compound auditory-visual stimulus consisting of aflashingcheckerboard and a pulsing sound. Curtains on either side of the high chair blocked the infant's lateral view. A panel facing the infant was covered with black posterboard and had one opening in it that was covered with a light diffuser (milk-white Plexiglass), The diffuser was covered with a transparency of a black-and-white check pattern that was made up of a random arrangement of '/>in. checks. The opening measured 15X15 cm. At a viewing distance of 43 cm, the checkerboard subtended 19" 18' of visual angle and was placed 45" to the left of the infant. The check pattern was lightedfrombehind the diffuser by two 14-W white fluorescent bulbs that operated silently and were housed inside a 50 x 18 X 23 cm box. To permit an essentially instantaneous onset, these bulbs were kept "warm" by applying 9-V D.C. to them during the "off" periods. To light the bulbs, 300-V D.C. were applied. During stimulation, the luminance of the white squares in the patterns was 5.1 ftL and the luminance of the black squares was approximately zero footlamberts. An Apple lie computer was used to control stimulus presentation and to measure duration of visual fixations. A Quan 8-S2, 4-in. speaker was placed directly above the visual stimulus. The auditory stimulus was a 330-Hz square-wave tone that measured 66 dB at the infant's ear (.0002 dynes/cm2, A scale). The rise time of the tone was controlled by a Coulbourn Instruments rise/fall switch and was set to 10 ms. Procedure. Testing took place in a quiet, dimly illuminated room. The ambient sound pressure level in the room as measured at the infant's ear was 46 dB (re .0002 dynes/cm2, A scale). The procedure used in these studies was a modified version of the procedure used by Demany (Demany, 1982; Demany, McKenzie, & Vurpillot, 1977). In essence, the presentation of the compound auditory-visual stimulus was contingent on the infant's visual behavior. The infant initiated the presentation of the compound stimulus by looking at the checkerboard and terminated it by looking away from it for a minimum of I s. The experimenter observed the infants' visual fixations on a video monitor and depressed a switch on the computer for the duration of each visual fixation. The computer was programmed to initiate and to continue presenting the stimuli as long as the switch was depressed and, at the same time, to measure the length of the visual fixation. The computer also
157
INFANTS' PROCESSING OF MULTISENSORY COMPOUNDS HABITUATION 0 - 1 0 0 0 rns.
1000-2000 m i .
2 Hz Aud
2 Hz Vis
4 Hz Aud
4 Hz Vis
AUD/VIS TEST TRIAL
VISUAL TEST TRIAL 0-1000 ms.
0-1000 ms.
1000-2000 ms.
2 Hz Aud
4 Hz Aud
4 Hz Vis
2 Hz Vis
4 Hz Vis
4 Hz Aud
2 Hz Aud
2 Hz Aud
2 Hz Vi»
4 Hz Vis
2 Hz Vis
1000-2000 ms.
Figure 1. Schematic representation of the temporal distribution of stimulation in the four types of trials in Experiment 1. (Across the four diagrams, the top two time-lines show the changes in rate that occurred from one type of trial to the next for half of the infants, and the bottom two time-lines show the changes that occurred for the other half of the infants. Thefilledbars represent stimulus on-time and the unfilled bars represent stimulus off-times.)
determined whether the 1-s look-away criterion was met; once the criterion was met the computer was ready to initiate the next trial at the depression of the switch. Each session was recorded on videotape for later reliability checks. Interobserver reliability, computed by correlating the duration of visualfixationscored by two observers on each trial for a random subsample of infants, was .97. A total of 24 trials was administered to each infant. The first 12 trials constituted the habhuatian phase, and the remaining 12 trials constituted the test phase. During the habituation phase each infant was presented with a compound stimulus whose auditory and visual components were concordant in rate. The auditory and visual stimuli were pulsed or flashed, respectively, at one of two rates, 2.0 Hz or 4.0 Hz. The stimulus-on time was set to 150 ms for both rates. Figure 1 is a schematic representation of the temporal distribution of the stimuli and their phase relations. As can be seen in the top panel, two types of compound stimuli were presented during the habituation phase. The compound stimulus consisting of a 2.0-Hz auditory and a 2.0-Hz visual component was presented to half of the infants, and the compound stimulus consisting of a 4.0-Hz auditory and a 4.0-Hz visual component was presented to the other half of the infants. As can be seen in the bottom three panels, three types of test trials were administered: (a) the visual test trial, which
involved a change in the rate of the visual component; (b) the auditory test trial, which involved a change in the rate of the auditory component; and (c) the auditory/visual test trial, which involved a change in the rate of both the auditory and the visual components. In the visual and auditory test trials the rate was changed from 2.0 Hz to 4.0 Hz in the group habituated with the 2.0-Hz compound stimulus and from 4.0 Hz to 2.0 Hz in the group habituated with the 4.0-Hz stimulus. In the auditory/visual test trial the rate of both components was changed from 2.0 Hz to 4.0 Hz in one group and from 4.0 Hz to 2.0 Hz in the other group. The test phase began as soon as the infant met the look-away criterion after the last habituation trial. It consisted of the following sequence of trials: two test trials, three rehabituation trials (where the original habituation stimulus was presented), two test trials, three rehabituation trials, and two test trials. The order in which the three types of test trials were presented was counterbalanced across infants. Because the observer was located behind the panel containing the visual stimulus and because the testing room was dimly lit, reflections of the flashing checkerboard on the infant's face were not visible to the observer. To prevent her from knowing what auditory stimuli were being presented, the observer listened to music through a set of headphones during the experimental session.
158
DAVID J. LEWKOWICZ
Data analysis. Because the standard deviation increased proportionally with increases in the mean, the raw data were transformed to common logarithms. The data from the last two habituation trials were then averaged, and this single score served as the baseline comparison against which the results from the test trials were compared. For each type of test trial the data from the two trials were averaged to obtain a single score.
Results The first step in the analysis of the habituation data was an examination of whether presentation of the two different compound stimuli exerted a differential effect on habituation. A two-way analysis of variance (ANOVA), was carried out, with group (2.0 Hz and 4.0 Hz) as the between-subjects factor and trial as the within-subjects factor. The effect of group approached significance, F{\9 22) = 3.25, p < .10, because of generally longer looking in the 4.0-Hz group during the habituation phase. The only other significant effect was a trials effect, F{\\, 242) = 4.50? p < .001, indicating that the infants exhibited a reliable overall decline in visual fixation over trials (see Figure 2). Despite the marginal difference in overall looking, the infants in both groups reached comparable levels of habituation, which was indicated by the fact that the average looking time during the last two trials of the habituation phase did not differ across the two groups. A second analysis was carried out to determine whether the rehabituation trials were effective in maintaining the state of habituation. This was done by comparing the duration of the infants' visual fixation in the last rehabituation trial in each of the two rehabituation series of trials, respectively, with the mean duration of visual fixation in the last two trials of the habituation phase (see Figure 2). Either no difference or a further decrement in looking would have indicated that these trials were effective. In this experiment results indicated that there was a significant drop in looking by the end of thefirstblock of rehabituation trials, #23) = 2.40, p < .025, as well as by the end of the second block of rehabituation trials, /(23) = 2.06, p < .05. The results from the test trials can be seen in Figure 2. In order to determine whether there were any differences in responsiveness across the four types of trials (the last two habituation trials, the visual test trial, the auditory test trial, and the auditory/visual test trial), a repeated measures ANOVA was performed, with trial type as the repeated factor. Results of this analysis indicated that there was an overall trials effect, i^3,69) = 5.85, p < .01. To determine whether reliable dishabituation occurred in any of the three test trials, Newman-Keuls post hoc tests were used to compare the magnitude of response in each of the test trials to the magnitude of response in the last two habituation trials. These comparisons indicated that the infants exhibited significant response recovery in the auditory/visual test trial (p < .01) but not in the other two test trials. Additional Newman-Keuls tests were used to determine whether response magnitude differed among the three types of test trials by comparing their data with one another. The results of these comparisons indicated that response magnitude was greater in the auditory/visual test trial than in both the visual (p < .01) and the auditory (p < .05) test trials. To determine whether the pattern of results found in the analysis of the overall data was reflected in the response of the indi-
vidual infants, the number of infants dishabituating to changes in the stimulus components was examined. This was done by comparing each infant's mean logfixationdata for each of the three test trials with the mean logfixationdata for the last two habituation trials. Out of the 24 infants tested, there were 9 who looked longer in the visual test trial (ns, binomial probability test), 14 who looked longer in the auditory test trial (ns), and 17 who looked longer in the auditory/visual test trial (p < .05).
Discussion The results from this experiment indicated that the infants discriminated the simultaneous change in the rate of components. Although detection of a change in either component alone would have been sufficient to produce the significant dishabituation obtained in the auditory/visual test trial, the absence of a significant dishabituation in either the auditory or the visual test trial suggests that the infants encoded the temporal properties of both components together. This does not, however, mean that the infants processed the holistic properties of the compound stimulus. Holistic processing would have required that they exhibit significant dishabituation in all three test trials, because a change in both components or in either component alone has the effect of changing the holistic properties of the compound stimulus. Thesefindingsalso suggest that the infants did not process the intersensory relationship inherent in the compound stimulus because no dishabituation was found in the auditory and visual test trials. Finally, the lack of significant response recovery in either the auditory or the visual test trial indicates that sensory dominance did not operate in this experiment. Experiment 2 In order to process the compound stimulus holistically or to process the intersensory relationship between the components, the infants must be able to detect and encode the temporal properties of each of the components, respectively. Although the basis for the infants' discrimination in the auditory/visual test trial in Experiment 1 was unclear, their failure to discriminate the changes in either the auditory or the visual test trial suggested that the infants did not encode the temporal properties of either component independently. This may have been due, in part, to the fact that the temporal properties of the auditory and visual components were not distinct enough during the habituation phase to give the infants an opportunity to learn about the specific temporal properties of each component. In order to determine whether the temporal similarity of the auditory and visual components was, in fact, partly responsible for the results obtained in Experiment 1, in this experiment the infants were given an opportunity to learn about the specific temporal properties of each component. This was done by habituating them with a compound stimulus whose auditory and visual components were presented at different rates.
Method Subjects. There were 24 six-month-old infants (11 boys, 13 girls) in this experiment; they ranged in age from 26 weeks, 6 days to 29 weeks, 4 days (Af age = 28 weeks, 4 days). An additional 7 infants were tested
INFANTS' PROCESSING OF MULTISENSORY COMPOUNDS
Hab
ReHabi ReHab2 Vis TYPE OF TRIAL
Aud
159
Aud/Vis
Figure 2. The top-rightfigureshows the decline in looking time during the habi tuation phase in Experiment 1. The bottom-left figure shows the mean looking times for the habituation, rehabituation, and test trials in Experiment 1.
but were excluded from data analysis (6 because of fussing or crying and 1 because of sleepiness). Apparatus and stimuli. The apparatus and stimuli in this experiment were identical to those used in Experiment 1. Procedure. The procedure was identical to the procedure used in Experiment 1, with one important exception. As can be seen in Figure 3, in this experiment the infants were presented with a temporally discordant stimulus during the habituation phase. (The term discordant means that the rate at which the auditory and visual components were presented was different.) As Figure 3 shows, in the current experiment the rate of one component was twice the rate of the other component. It should also be noted that the onset and offset of every other occurrence of the faster component corresponded precisely to the onset and offset of each occurrence of the slower component. As can be seen in the top panel of Figure 3, half of the infants were presented with a compound stimulus consisting of a 2.0-Hz auditory stimulus and a 4.0-Hz visual stimulus and half were presented with a compound stimulus consisting of a 4.0-Hz auditory stimulus and a 2.0 Hz visual stimulus. The bottom three panels of thefigureshow the rate changes involved in the three types of test trials.
Results Analyses of the habituation data from the two groups indicated no significant effects attributable to group membership. As can be seen in Figure 4, the infants exhibited an overall decline in visual fixation over trials that was statistically signifi-
cant, Fi\ 1,242) = 6.06, p < .001. Analysis of the rehabituation trials indicated that the infants maintained their state of habituation. There was no significant difference between the mean duration of looking during the last two trials of the habituation phase and during the last trial of each of the two series of three rehabituation trials (see Figure 4). A one-way ANOVA performed on the means from the last two habituation trials and the three types of test trials indicated that there was a significant main effect of trial type, F(3,69) = 8.26, p < .001. As was true for the infants habituated with the temporally concordant compound stimulus in Experiment 1, the infants in this experiment dishabituated to the simultaneous change in both components {p < .01). In contrast to the infants in Experiment 1, however, the infants in Experiment 2 dishabituated to the change in the rate of the auditory component (p < .01) as well. They did not dishabituate to the change in the rate of the visual component. Comparisons of the test trial data showed that response magnitude in the auditory/visual test trial was greater than in the visual test trial (p < .01) and that the response magnitude in the auditory test trial was greater than in the visual test trial (p < .05). There was no difference in response between the auditory/visual and the auditory test trial. Analysis of the number of infants showing greater responsiveness in the test trials supported the results of analyses of re-
160
DAVID J. LEWKOWICZ HABITUATION 0 - 1 0 0 0 ms.
1000-2000 ms.
2 Hi Aud
4 Hz Via
4 Hz Aud
2 Hz Vis
VISUAL TEST TRIAL
0-1000 mi.
AUDITORY TEST TRIAL
1000-2000 m».
0 - 1 0 0 0 ms.
AUD/V1S TEST TRIAL
1000-2000 ms
0 - 1 0 0 0 ms.
2 Hz Aud
4 Hz Aud
4 Hz Aud
2 Hz Vb
4 Hz Vis
2 Hz Vis
4 Hz Aud
2 Hz Aud
2 Hz Aud
4 Hz Ms
2 Hz Vis
4 Hz Vis
1000-2000 ms.
Figure 3. Schematic representation of the temporal distribution of stimulation in the four types of trials in Experiment 2.
sponse magnitude. Out of 24 infants, 14 looked longer in the visual test trial (its), 17 looked longer in the auditory test trial (p < .05), and 20 looked longer in the auditory/visual test trial Qx.Ol). Discussion As in Experiment 1, the infants in Experiment 2 discriminated the simultaneous change in both components. In contrast to the infants in Experiment 1, however, the infants in this experiment also discriminated the change in the temporal properties of the auditory component. This shows that making the temporal difference between the components more distinct permitted the infants to encode the temporal properties of the auditory component independently of the visual component. In addition, because the infants discriminated the change in the auditory component alone, and because response magnitude in the auditory/visual test trial was not greater than in the auditory test trial, the basis of the discrimination in the auditory/visual test trial may be the detection of the change in the rate of the auditory component alone. One of the most interesting aspects of these findings is the fact that the temporal distinctiveness of the components in this
experiment was not effective in allowing the infants to learn about the temporal properties of the visual component. This suggests that the auditory stimulation dominated the encoding process and prevented the infants from learning about the temporal characteristics of the visual component. Nonetheless, even if the findings from this experiment are interpreted as evidence of auditory dominance, they do not rule out the possibility that visual information played an important role in the encoding process. The key difference between this experiment and the first one is that the relationship between the auditory and visual components changed from a concordant one to a discordant one during the habituation phase. The only way in which the positive efFect arising out of the greater temporal distinctiveness of the components could have been obtained was if the infants in the current experiment capitalized on this distinctiveness and actively compared the temporal characteristics of the visual and auditory components during the habituation phase. In other words, although the temporal characteristics of the visual component were not sufficiently encoded to perform a discrimination, the most reasonable interpretation of the obtained results is that the visual information served as a useful contrast against which the auditory information could be differentiated. There is, of course, the possibility that, rather than reflecting
161
INFANTS' PROCESSING OF MULTISENSORY COMPOUNDS
1 2 i
4 5 6 7 8 9 10 I I 12
TRIALS
Hab
ReHabi ReHob2 Vis TYPE OF TRIAL
Aud
Aud/Vis
Figure 4. The top-rightfigureshows the decline in looking time during the habituation phase in Experiment 2. The bottom-left figure shows the mean looking times for the four different types of trials in Experiment 2.
auditory dominance, the current results reflect a visual discrimination deficit. In other words, it may be that the infants were not able to discriminate between the two rates of visual stimulation used in the current experiment even in the absence of concurrent auditory stimulation. That this is not likely, however, is indicated by prior studies of 6-month-old infants' response to variations in the rate of visual stimulation. Lewkowicz (1985b) showed that 6-month-old infants responded differentially to checkerboards that flashed at rates that were the same as those used in the current experiments. Experiment 3 Thus far the data suggest that auditory dominance has its effect during the habituation phase. That is, it appears that it is during the habituation phase that the auditory information blocks the processing of visual information and prevents the infant from encoding it. It is possible, however, that auditory dominance operates during the test phase as well. The purpose of this experiment was, therefore, to investigate this possibility. As in Experiment 1, infants were habituated with a temporally concordant compound stimulus. In contrast, however, in this experiment the discriminability of the auditory and visual components was increased by providing the infants with an ad-
ditional discriminative cue during the test phase. This was achieved by the way in which the stimuli were presented. Instead of keeping the stimulus duration constant across rate as was done in Experiment 1, in this experiment the stimuli were presented at a 50% on/offratio. As a result, whenever two components differed in rate they also differed in duration. Because the components were temporally concordant during the habituation phase no change resulted from the use of a 50% on/off ratio. During the test trials, however, the use of the 50% on/ off stimulus ratio meant that a change in the rate of a given component was always accompanied by a change in stimulus duration. If auditory dominance operates during the test phase as well, then making a second discriminative cue available during the test phase may have an effect similar to the one obtained in Experiment 2. That is, the infants may not discriminate the change in the rate of the visual component but may discriminate the change in the rate of the auditory component as well as the change in both components.
Method Subjects. There were 24 six-month-old infants (13 boys, 11 girls) in this experiment, who ranged in age from 26 weeks, I day to 28 weeks, 1 day (M age = 27 weeks). An additional 10 infants were tested but were excluded from data analysis (8 because of fussing or crying, 1 because of equipment failure, and 1 because of experimenter error).
162
DAVID J. LEWKOWICZ HABITUATION O-1OOO ms.
1000-2000 ms.
.5 Hz Aud
.5 Hz Vis
•yn=
2 Hz Aud
2 Hz Vis
AUD/VIS TEST TRIAL
AUDITORY TEST TRIAL
VISUAL TEST TRIAL 0 - 1 0 0 0 ms.
0 - 1 0 0 0 ms.
1000-2000 ms.
0-1000 ms.
1000-2000 ms.
tPtd
2 Hz Aud
.5 Hz Aud
2 Hz Aud
2 Hz Vis
.5 Hz Vis
2 Hz Vis
2 Hz Aud
.5 Hz Aud
5 Hz Aud
.5 Hz Vis
2 Hz Vis
^
^
1000-2000 ms.
-tzFt
5 Hz Vis
Figure 5. Schematic representation of the temporal distribution of Stimulation in the four types of trials in Experiment 3. Apparatus and stimuli. The apparatus and stimuli were identical to the apparatus and stimuli used in the two previous experiments. Procedure. The procedure was identical to the procedure used in Experiment 1, with one important exception. As can be seen in the top panel of Figure 5, the rates of stimulation used in this experiment were 0.5 Hz and 2.0 Hz. Because the purpose of this experiment was to provide the infants with duration as well as rate variations, a 50% on/off stimulus ratio was used. As a result, at a rate of 0.5 Hz the stimulus was on for 1,000 ms and off for 1,000 ms, whereas at a rate of 2.0 Hz the stimulus was on for 250 ms and off for 250 ms. The reason that 0.5 Hz and 2.0 Hz were used rather than 2.0 Hz and 4.0 Hz was because the duration difference between a 0.5-Hz stimulus and a 2.0-Hz stimulus is much more likely to be discriminable than the duration difference between a 2.0-Hz and a 4.0-Hz stimulus (which is 250 ms vs. 125 ms). As can be seen in the top panel of Figure 5, half of the infants were habituated with a temporally concordant compound stimulus whose rate was 0.5 Hz, and half were presented with a temporally concordant compound stimulus whose rate was 2.0 Hz. As can be seen in the bottom three panels of the figure, for the group habituated with the 0.5-Hz compound stimulus the rate of the components was changed to 2.0 Hz during the test trials; for the group habituated with the 2.0-Hz compound stimulus the rate was changed to 0.5 Hz.
Results Analysis of the habituation data indicated that there were no effects attributable to group membership. For the group as a
whole, there was a reliable overall decrement in visual fixation during the habituation phase, F( 11, 242) = 4.82, p < .001 (see Figure 6). Analysis of the rehabituation trials indicated that there was a significant drop in visual fixation in both the first series, f(23) = 2.27, p < .05, and in the second series, r(23) = 2.82, p < .01, of rehabituation trials (see Figure 6). The results of the test trials are depicted in Figure 6. A oneway ANOVA on the four types of trials indicated that there was a significant trials effect, F(S, 69) = 4.71, p < .01. NewmanKeuls tests indicated that the response to the change in both components was significant (p < .05), that the response to the change in the rate of the auditory component was marginally significant (p < .10), and that the response to the change in the rate of the visual component was not significant. Comparisons of response magnitude in the different test trials indicated that the response in the auditory test trial was significantly greater than the response in the visual test trial (p < .05) and that the response in the auditory/visual test trial was also greater than the response in the visual test trial (p < .05). There was no difference in response magnitude between the auditory and the auditory/visual test trials. Analysis of the number of infants exhibiting greater responding in the test trials in general supported the analyses of response magnitude. Out of the 24 infants tested, there were 9
163
INFANTS' PROCESSING OF MULTISENSORY COMPOUNDS
1.0
.9 .8
ll
6 1 2 5 4 5 6 7 8 9 10 11 12
TRIALS
.5 .4
o
3
0)
.2 .1
Hab
ReHabi ReHob2
Vis
Aud
Aud/Vis
TYPE OF TRIAL
Figure 6. The top-rightfigureshows Ihe decline in looking time during the habituation phase in Experiment 3. The bottom-left figure shows the mean looking times for the four different types of trials in Experiment 3.
who looked longer in the visual test trial (ns), 17 who looked longer in the auditory test trial (p < .05), and 18 who looked longer in the auditory/visual test trial (p < .05). Discussion The data from the current experiment provided some limited evidence that making two discriminative cues (rather than one) available during the test phase was equivalent to making the two components temporally distinct during the habituation phase. That is, the infants in this experiment exhibited a tendency to differentiate the temporal properties of the auditory component independently of the temporal properties of the visual component when a second discriminative cue was made available during the test phase. The limited nature of this evidence stemmed from the marginal dishabituation that was obtained in the auditory test trial. Nonetheless, the analysis of the number of infants showing dishabituation in the auditory test trial indicated that significantly more infants dishabituated in the auditory test trial than did not, and comparison of response magnitude in the test trials indicated that the response in the auditory test trial was greater than in the visual test trial. As in the preceding experiments, the infants in this experiment did not respond to the change in the rate of the visual component, nor did they show a significantly greater response
in the auditory/visual test trial than in the auditory test trial. Thus, the fact that the availability of a second discriminative cue during the test phase had a marginal effect on the processing of the auditory information but that it had no effect on the infants' processing of the visual information suggests that auditory dominance may also operate during the test phase. Despite this, however, the marginal improvement in processing argues once again for the importance of the visual information in providing the infant with a useful contrast. In this case, the clearer contrast between the two components in the test phase—made possible by the addition of the duration difference—presumably aided the infants in detecting the auditory change. Experiment 4 The data from the previous experiments indicated that increasing the temporal distinctiveness of the components during the habituation phase and adding to the temporal distinctiveness of the components during the test phase permitted the infants to process the auditory information inherent in the compound stimulus and to do so independently of the visual information. Taken together, these twofindingsfurther suggest that an increase in the distinctiveness of the components during both phases of the experiment may permit the infants to differentiate the temporal characteristics of each of the components
164
DAVrD J. LEWKOWICZ
independently of the other and thus may permit the infants to process the visual as well as the auditory information. To test this possibility, in this experiment infants were habituated with a temporally discordant compound stimulus composed of components that were presented at a 50% on/off stimulus ratio. As a result, the two components differed in rate and duration during the habituation phase, and during the test phase a change in the rate of either component was accompanied by a change in its duration.
Method Subjects. There were 24 infants (14 boys, 10 girls) in this experiment, who ranged in age from 26 weeks, 3 days to 29 weeks, 2 days (M age = 28 weeks, 1 day). An additional 9 infants were tested but were excluded from data analysis (6 because of fussing or crying, 1 because of equipment failure, and 2 because of experimenter error). Apparatus and stimuli. The apparatus and stimuli were the same as in Experiment 3. Procedure. The procedure was the same as in Experiment 3 (i.e., a 50% on/off ratio was used for stimulus presentation). The one important difference in this experiment was that a temporally discordant compound stimulus was presented during the habituation phase. As can be seen in the top panel of Figure 7, half of the infants were presented with a compound stimulus consisting of an auditory stimulus presented at a rate of 0.5 Hz and a visual stimulus presented at a rate of 2.0 Hz, whereas the other half were presented with a compound stimulus consisting of an auditory stimulus presented at a rate of 2.0 Hz and a visual stimulus presented at a rate of 0.5 Hz, The bottom three panels of Figure 7 show the rate changes involved in the three types of test trials.
Results Analyses of the habituation data indicated that there was a significant Group X Trial interaction, F({ 1, 242) = 1.90, p < .05. Despite this interaction, however, a comparison of the mean amount of looking during the last two trials of the habituation phase indicated that there was no difference between the two groups. As can be seen in Figure 8, for the sample as a whole there was an overall decrement in visual fixation during the habituation phase, which the ANOVA indicated was statistically reliable, F{ 11,242) = 6.29, p < .001. The means for the rehabituation trials can also be seen in Figure 8. A comparison of the mean of the last two habituation trials and the duration of visualfixationin the last rehabituation trial of each series of three rehabituation trials, respectively, indicated that there were no significant differences. Responding in the test trials is depicted in Figure 8. A oneway ANOVA of the data from the habituation trials and the three types of test trials indicated a significant trials effect, F(3,69) = 5.24, p < .01. Comparisons of the data from the test trials with the data from the habituation trials indicated that the infants discriminated the change in both components (p < .01). They did not, however, exhibit discrimination of the change in either the auditory or the visual components. Direct comparisons of the data from the test trials showed that the response in the auditory/visual test trial was significantly greater than the response in both the visual (p < .05) and the auditory (p < .05) test trials. Analysis of the number of infants looking longer in the test trials supported the results of analyses of response magnitude. It showed that out of 24 infants, 10 looked longer in the visual test trial (ns), 14 looked longer in the auditory test trial
(ns), and 22 looked longer in the auditory/visual test trial
Discussion The outcome of this experiment was not consistent with the prediction. Although the infants discriminated the change in both components, they did not discriminate the change in either component alone. Thus, it appears that the presence of rate and duration differences during both phases of the experiment did not help the infants to differentiate the temporal properties of the two components independently. When the results from the current experiment are considered in conjunction with the data from the preceding three experiments, it becomes clear that the processing of multisensory information is affected by the complexity of the task. Although discriminability was improved by making the components temporally distinct in Experiment 2, this was not the case in the current experiment. It seems that the positive effect of making the components temporally distinct was offset by the availability of multiple discriminative cues. Specifically, a closer examination of the information that was available to the infants shows that in the habituation phase of the current experiment the infants were presented with more information than were the infants in all of the three preceding experiments. That is, in the current experiment, the auditory and visual components differed in rate and duration. In contrast, in Experiments 1 and 3, because of the presentation of temporally concordant information, the auditory and visual components did not differ at all, whereas in Experiment 2 the two components only differed with respect to rate. Thus, during the habituation phase in the current experiment the infants had to process rate and duration information simultaneously. As a result, they were faced with a much more complex task during the learning phase than were the infants in the preceding three experiments. Similarly, during the test phase the infants in this experiment were faced with a simultaneous change in rate and duration and, therefore, had to determine whether both of these attributes were the same or different from those encoded during habituation. In contrast, the infants in Experiment 2, where a temporally discordant compound stimulus was also presented during the habituation phase, only had to determine whether there was a difference in rate. Consequently, the fact that the infants in the current experiment had to attend to more information than the infants in Experiment 2 might have accounted for the failure of the infants in the current experiment to discriminate changes in the auditory and visual components. The same logic can be applied to the comparison of the outcome in the current experiment to the outcome in Experiment 3. Because the two components were temporally concordant in Experiment 3, the infants could either rely on rate or on duration for encoding the information; they did not have to process both attributes simultaneously. Thus, when presented with a change in one of the components, the fact that they had both rate and duration available as a cue for discrimination most likely made it easier for them to detect the difference because of the availability of redundant information. In contrast, because of the explicit difference in rate and duration during the habituation phase in this experiment, the infants were forced to attend to both stimu-
165
INFANTS' PROCESSING OF MULTISENSORY COMPOUNDS HABITUATION
0-1000 nu.
1000-2000 ra.
.5 Hz Aud
2 Hz V a
2 Hz Aud
.5 Hz V «
0-1000 ms.
AUD/V1S TEST TRIAL
AUDITORY TEST TRIAL
VISUAL TEST TRIAL
0-1000 ms.
1000-2000 ms.
0-1000 ms.
1000-2000 ms.
.5 Hz Aud
2 Hz Aud
2 Hz Aud
.5 Hz Va
2 Hz Va
.5 Hz Vis
2 Hz Aud
.5 Hz Aud
.8 Hz Aud
.5 Hz Va
2 Hz Va
2 Hz Va
-tPt=Ftflz
1000-2000 ms.
Figure 7. Schematic representation of the temporal distribution of stimulation in the four types of trials in Experiment 4.
lus attributes during habituation and during the test trials. Having to process and attend to both stimulus attributes simultaneously most likely placed processing demands on the infants that were at the limits of their capacity.
discrimination of the changes in the visual component. To test this possibility, infants' detection of temporal changes in the visual component was studied in the current experiment using the identical method used in the preceding experiments.
Experiment 5 The consistent failure of the infants to detect the changes in visual information across a range of temporal frequencies and across a range of experimental manipulations designed to enhance the distinctiveness of the visual information suggests, at a minimum, that the auditory modality dominates processing. Support for this conclusion was provided by thefindingthat 6month-old infants could make a visual discrimination between the two rates belonging to one of the two pairs of rates used in the previous experiments (2.0 and 4.0 Hz) when tested with a paired preference technique (Lewkowicz, 1985b). Nonetheless, one very important question that remained unanswered was whether the failure of the infants in the previous experiments to respond to the change in the rate of the visual component reflected auditory dominance, or whether the memory constraints imposed by the habituation/test technique placed too great a processing demand on the infants and thus prevented
Method Subjects. There were 24 infants (12 boys, 12 girls) in this experiment; they ranged in age from 26 weeks, 5 days to 29 weeks, 1 day (Mage = 27 weeks, 6 days). An additional 22 infants were tested but were excluded from data analysis (15 because of fussing or crying, 5 because of distracting conditions, and 2 because of inattentiveness). Apparatus and stimuli. The apparatus and stimuli were identical to those used in Experiment 3, except that only the visual component was presented. Procedure. The procedure was identical to the procedure used in Experiment 3. Half of the infants were habituated with the checkerboardflashingat 0.5 Hz and half with itflashingat 2.0 Hz. Following the habituation phase, two test trials were administered to each infant; for those habituated with the 0.5-Hz visual component the rate was changed to 2.0 Hz during the two test trials, and the opposite was done for the other half of the infants.
166
DAVID J. LEWKOWICZ
1 2 2 4 5 6 7 8 9 10 11 12
TRIAL5
Hob
ReHabi ReHab2 Vis TYPE OF TRIAL
Aud
Aud/Vis
FigureS, The top-right figure shows the decline in looking time during the habituation phase in Experiment 4. The bottom-left figure shows the mean looking times for the four different types of trials in Experiment 4.
Results and Discussion As can be seen in Figure 9, there was an overall decline in the infants' visualfixationduring the habituation trials. A one-way ANOVA indicated that this was a significant effect, F( 11,242) = 3.06, p < .001. Thefigurealso shows that there was significant recovery of response in the test trial, *(23) = 2.92, p < .005, indicating that the infants detected the change in the temporal characteristics of the visual component when it did not have to compete with the auditory component for the infants' attention. Thisfindingalso ruled out the possibility that the failure of the infants in the previous experiments to respond to the changes in the temporal characteristics of the visual component was a methodological artifact. Experiment 6 A second unanswered question generated by the results from the first four experiments was why the infants did not respond to the intersensory relationship. In general, studies of intersensory matching have reported that by 4 months of age infants can detect temporally based auditory-visual correspondences between familiar, moving, visual stimuli (Bahrick, 1983; Mendelson & Ferland, 1982;Spelke, 1979; 1981; Walker-Andrews, 1986) and various sounds. Studies of intersensory matching us-
ing visual stimuli that were unfamiliar and spatially static (similar to the stimuli used in the current studies) did notfindauditory-visual matching at 4 months of age (Lewkowicz, 1985a) but didfindit at 6 months of age (Lewkowicz, 1986). When the findings from the studies of intersensory matching are considered in conjunction with the failure of the infants in the current studies to respond to the intersensory relationships inherent in the compound stimuli, the possibility that procedural differences may be responsible for the different results emerges. That is, it may have been that the simultaneous forced-choice procedure used in all the studies showing successful intersensory matching may have made it easier for the infants to detect intersensory equivalence than did the serial-processing procedure used in the current experiments. To determine if this was the case, the infants in this experiment were also tested for their ability to make bisensory matches of rate.
Method Subjects. Eighteen infants (6 boys, and 12 girls) were tested; they ranged in age from 27 weeks, 1 day to 31 weeks, 2 days (M age = 28 weeks). An additional 5 infants were tested but were excluded from data analysis (4 because of fussing or crying and 1 because of drowsiness). Apparatus and stimuli. The apparatus used was the same as in all the preceding experiments except that a second checkerboard, identical to
INFANTS' PROCESSING OF MULTISENSORY COMPOUNDS
167
1 .OO T .94 .88
Figure 9. Visualfixationduring the habituation trials (represented by the line) and the average visual fixation in the last two habituation trials and in the test trial (represented by the bars) in Experiment 5.
the one used in all the other experiments, was added to the display on the infant's right side and that the infant sat halfway between the two checkerboards. The inner edges of the two checkerboards were 23.5 cm apart. A set of 5 colored light-emitting diodes arranged in a cross configuration was located halfway between the two visual stimuli and was used to attract the infants' attention during the intertrial intervals. The infants' visual fixations were videotaped with a camera whose lens was placed halfway between the stimuli and then scored by an observer blind with respect to the stimuli. The visual stimuli were presented at one of two rates, 0.5 Hz or 2.0 Hz. At each rate the stimulus was presented at a 50% on/oflF ratio. The auditory stimulus was a 1,000 cps sine-wave tone that was presented at 72 dB, measured at the infant's ear, and had a rise time of 10 ms. The sound was presented through a speaker placed halfway between the two checkerboards. Procedure. As soon as the infant fixated the colored lights, they were turned off and the stimuli were turned on. Each infant was administered a total of six 20-s trials. During the first two trials the visual stimuli were presented in silence. The lateral position of the stimuli was reversed on the second trial, and the left-right positioning of the stimuli was counterbalanced across infants. During the remaining four trials the visual stimuli were presented together with the sound. For each infant the lateral position of the two visual stimuli alternated during these four remaining trials. For half of the infants the slower sound was presented during the first two of these trials and the faster sound was presented during the last two trials. For the other half of the infants the order of sound presentation was reversed.
Results and Discussion During the silent trials, the infants spent 60% of their time looking at the faster visual stimulus, t( 17) = 2.53, p < .05 (twotailed). As in the previous experiment, the infants in this experiment exhibited differential responsiveness to the two rates of visual stimulation. The results from the bisensory matching trials did not provide any evidence of matching. Although the in-
fants looked longer at the faster visual stimulus in the presence of the faster sound (M proportion = .57), r(17) = 2.06, p < .05, they also looked longer at the faster visual stimulus in the presence of the slower sound (M proportion = .60), t{\l)= 3.72, p < .005. Given that the infants showed a preference for the faster stimulus in the absence of sound, the results from the bisensory trials suggest that the infants' response in these trials was primarily determined by the rate of the visual stimulus. Inspection of the mean proportions of looking at the faster visual stimulus in silence, in the presence of the faster sound, and in the presence of the slower sound indicates virtually identical results and suggests that the rate of the auditory stimulus had no effect on visual behavior. In sum, these data show that the infants' failure to respond to the intersensory relationships in the preceding experiments was not due to the use of the serial-processing procedure. Experiment 7 Although the results from Experiments 5 and 6 indicated that the two rates of visual stimulation were discriminable, the fact that the auditory and visual components were not matched for their subjective equality in any of the preceding experiments made it possible that the observed effects were due to the differential effectiveness of the two components. To determine if this was in fact the case, Experiment 3 was repeated, but this time the intensity of the visual component was increased. Thus, in the current experiment infants were presented either with a 0.5Hz or a 2.0-Hz temporally concordant compound stimulus during the habituation phase. If the auditory component was subjectively more intense than the visual one, then increasing the intensity of the visual component might make the two compo-
168
DAVID J. LEWKOWICZ
nents more equivalent. Either this should permit the infants to discriminate the change in the visual component or, at least, lead to a reduction in the differential responsiveness to the auditory and visual components observed in Experiment 3. Method Subjects. There were 24 subjects in this experiment (11 boys, 13 girls), ranging in age from 27 weeks, 1 day to 29 weeks, 3 days (Mage = 28 weeks, 1 day). An additional 24 infants were tested but were excluded from data analysis (20 because of crying or fussing and 4 because of the presence of distracting conditions). Apparatus and stimuli. The apparatus and stimuli were identical to the apparatus and stimuli used in Experiment 3, with one important exception. The intensity of the visual stimulus was increased so that the luminance of the white squares was now 9.4 ftJL Procedure. The procedure was identical to the procedure used in Experiment 3.
a comparison of the results from this experiment with those from Experiment 3 (in which the identical procedure was used, with the exception that the visual component was dimmer) shows that the dishabituation in the auditory test trial was more robust in this experiment. This difference obviously resulted from the increased intensity of the visual component and serves as further proof that the visual component did play an active role in the infants' encoding of the auditory component. The other effect of increasing the intensity of the visual component was that the response in the auditory/visual test trial in the current experiment was significantly greater than in the auditory test trial. That was not the case in Experiment 3. This enhancement of responding was intriguing because it showed that the discriminability of the visual component was increased but only when it was processed together with the auditory component; the increase in its intensity was still not sufficient to permit the infants to encode its temporal properties independently of the auditory component.
Results Analysis of the habituation data indicated that there was an overall decrement in visualfixationas a function of trials, F( 11, 242) = 8.92, p < .001 (see Figure 10). Comparison of the mean amount of looking during the last two habituation trials and the mean amount of looking during each of the rehabituation trials yielded no differences (see Figure 10). The results from the test trials are shown in Figure 10. A oneway ANOVA of the response in the habituation trial and the three types of test trials indicated a highly significant trials effect, F(3, 69) = 15.91, p < .001. Comparisons of the means indicated that the infants did not discriminate the change in the visual test trial. They did, however, discriminate the change in both the auditory (p < .01) and the auditory/visual (p < .01) test trials. Furthermore, the response in the auditory test trial was greater than the response in the visual test trial (p < .01), and the response in the auditory/visual test trial was greater than the response in the visual test trial (p < .01). Finally, in contrast to all the previous experiments, the infants in this experiment exhibited significantly greater response in the auditory/visual test trial than they did in the auditory test trial (p < .01). Analysis of the number of infants looking longer in the test trials was consistent with the results of analyses of response magnitude. Out of 24 infants, 11 looked longer in the visual test trial (its), 20 looked longer in the auditory test trial (p = .001), and 21 looked longer in the auditory/visual test trial, (p < .001). To determine whether the infants in the present experiment differed in their response during the habituation phase from the infants in Experiment 3, a two-way ANOVA was carried out, with group as the between-subjects factor and trials as the within-subjects factor. No significant differences were found. Discussion The results from this experiment provide additional evidence in support of the dominance interpretation offered earlier. Despite a marked increase in the intensity of the visual component, the auditory component still dominated the infants* responsiveness. As in the previous experiments, however, there was evidence that the visual component was involved in responsiveness, albeit indirectly. This was evident in two effects. First,
General Discussion The major finding from the current set of experiments is that the infants exhibited auditory dominance when processing a compound stimulus that was composed of temporally modulated auditory and visual information. As a result, they were not responsive to the different intersensory relationships that were present nor were they responsive to the holistic properties of the compound stimulus. In addition, the infants' response to the multisensory compound was determined jointly by the distinctiveness of the temporal cues, the number of those cues available, and whether those cues were available during the encoding phase, the test phase, or during both phases. The auditory dominance found in these experiments is not surprising when considered in the context of the ontogenetic development of the sensory systems. In mammalian development, including human development, the auditory system precedes the visual system structurally and functionally (Bronson, 1982; Gottlieb, 1971; Kasatkin, 1972; Volokhov, 1968). In humans the auditory system becomes responsive to sound somewhere between the 25th and 27th week of gestation (Grimwade, Walker, Bartlett, Gordon, & Wood, 1971; Starr, Amlie, Martin, & Sanders, 1977; Weitzman & Graziani, 1968), and, because sound is capable of penetrating the abdominal cavity during this time (Armitage, Baldwin, & Vince, 1980; Bench, 1968; Bench, Anderson, & Hoare, 1979; Grimwade et al., 1971), a considerable amount of auditory experience is available prior to birth. Studies of newborn infants have shown that this experience has an important effect, as evidenced by the finding that in the first days of life infants discriminate their own mothers* voices from strangers' voices (DeCasper & Fifer, 1980; DeCasper & Spence, 1986; Panneton & DeCasper, 1986). Furthermore, within the first 2 months of life infants can (a) discriminate a wide variety of phonetic contrasts in a categorical manner (Aslin, Pisoni, & Jusczyk, 1983), (b) make categorical discriminations on the basis of context and relational information (Jusczyk, Pisoni, Reed, Fernald, & Myers, 1983), and (c) make auditory discriminations based on rhythmic structure (Demany, 1982; Demany, McKenzie, & Vurpillot, 1977). The visual modality, on the other hand, is considerably more immature. Nearly all the basic functions of the visual system
169
INFANTS' PROCESSING OF MULTISENSORY COMPOUNDS
1 2 3 4 5 6 7 8 9 10 11 12
TRIALS
Hob
ReHabi ReHab2 Vis TYPE OF TRIAL
Aud
Aud/Vis
Figure 10. The top-right figure shows the decline in looking time during the habituation phase in Experiment 7. The bottom-left figure shows the mean looking times for the four different types of trials in Experiment 7.
are quite poor at birth and require several months to reach a level of functional maturity that is roughly comparable with that of the auditory system (Banks & Salapatek, 1983). Among the functions that undergo major changes during the first 3 to 6 months after birth are accommodation, smooth pursuit eye movements, acuity, and contrast sensitivity; the immature status of all these functions places limitations on the visual system's ability to respond to the environment. In sum, it is possible that the auditory dominance observed in the current experiments is a vestige of the asynchronous development of the auditory and visual modalities. The finding of auditory dominance is the opposite of what has consistently been found in adult subjects. It suggests a developmental shift in dominance, and, even though they did not specifically predict this type of dominance in early infancy, the apparent developmental shift is consistent with Birch and Lefford's (1967) general predictions. It is also consistent with Bronson's (1982) prediction of an Age X Modality interaction in the infant's sensitivity to the sensory qualities of the environment. Although the precise timing of the shift is uncertain, studies in progress in the author's laboratory have shown that, in addition to being able to discriminate changes in auditory information, 10-month-old infants are able to discriminate changes in visual information when tested in the same tasks. Thus, it appears that by 10 months of age the visual modality
is no longer dominated by the auditory modality. By adulthood it is the visual modality that is the dominant modality. The evidence for visual dominance in adulthood is very impressive indeed. When subjects are asked to respond to auditory and visual stimuli as fast as possible by pressing a separate key in the presence of one or the other stimulus, they respond by pressing the appropriate key and have a shorter reaction time to the auditory stimulus. When, however, on some trials they are presented with both stimuli, which are matched for subjective intensity, they overwhelmingly press the key for the visual stimulus and act as if the auditory stimulus does not exist (Colavita, 1974; Colavita & Weisberg, 1979; Egeth & Sager, 1977). This effect is so robust that the same results are obtained even when the subjective intensity of the auditory stimulus is made twice that of the visual stimulus and when subjects are actually instructed to respond to the auditory stimulus during the bisensory trials. Similar dominance has been demonstrated in normal and retarded children (Hermelin & O'Connor, 1964; O'Connor & Hermelin, 1965). In localization studies, there is the well-known "ventriloquism effect," where subjects tend to perceive an auditory stimulus as emanating from the visual stimulus despite the fact that two stimuli are spatially displaced with respect to one another (Hay, Pick, & Ikeda, 1965; Jack & Thurlow, 1973; Radeau & Bertelson, 1977). Studies where temporal factors are clearly involved have also found visual
170
DAVID J. LEWKOWICZ
dominance. For example, in bimodal speech perception lip movements play such a powerful role that when a subject views the lips making one syllable and at the same time listens to another syllable, he or she reports hearing a syllable that is different from the one presented aurally (McGurk & MacDonald, 1976). It has been argued that visual dominance in adults reflects an active process of compensation by the visual modality that is due to its poorer alerting capacities relative to the auditory modality (Posner, Nissen, & Klein, 1976). The data from the current experiments suggest that the visual modality has not overcome the greater alerting properties of the auditory modality by 6 months of age. Although these findings suggest that auditory information plays a dominant role in the processing of auditory-visual compounds at 6 months of age, this does not mean that visual information is completely ignored. There were three instances where visual information appeared to play a role in responding, albeit one that was subordinate to that played by the auditory information. Two of these have already been noted. To reiterate, the fact that increasing the distinctiveness of the components led to greater differentiation of the compound stimulus suggests that the visual component served as a background against which the auditory information was compared. Moreover, in Experiment 6 the greater intensity of the visual component enhanced the infant's ability to discriminate the change in the auditory component. This type of enhancement of responding to stimulation in one modality by stimulation in another modality is consistent with findings reported in studies with adults (Egeth & Sager, 1977; Halpern & Lantz, 1974; Loveless, Brebner, & Hamilton, 1970). Finally, the fact that in Experiments 1 and 4 the infants discriminated the change in both components but failed to discriminate the change in the auditory component suggests that the change in the visual information was as necessary as was the change in the auditory information in order for the infants to detect the change in the auditory/visual test trial. It is important to note that the current findings may be specific to the case where spatially static visual information is presented in conjunction with simple, socially neutral, and nonpatterned auditory stimulation. It is possible that the results might be different if spatially dynamic visual stimuli and patterned and socially meaningful auditory stimuli were used. At the same time, however, it should be noted that many of the studies demonstrating visual dominance in adults have used spatially static visual and simple auditory stimuli. Consequently, the question of whether auditory dominance would be observed with multisensory compounds composed of more complex and meaningful stimuli is an empirical one. In conclusion, thefindingsfrom this set of experiments indicate that there is a hierarchy in the functional priority of sensory systems in early development. This hierarchy has important implications for the processing of multisensory information in early development and for the way in which young infants learn about their world.
References Armitage, S. E., Baldwin, B. A., & Vince, M. A. (1980). The fetal environment of sheep. Science, 208, 1173-1174. Aslin, R. N., Pisoni, D. B., & Jusczyk, P. W. (1983). Auditory development and speech perception in infancy. In P. H. Mussen (Series Ed.)
and M. M. Haith & J. J. Campos (Vol. Eds.), Handbook ofchild psychology: Vol. 2. Infancy and developmentalpsychobiology (pp. 573687). New York: Wiley. Bahrick, L. E. (1983). Infants' perception of substance and temporal synchrony in multimodal events. Infant Behavior and Development, 6,429-451. Banks, M. S., & Salapatek, P. (1983). Infant visual perception. In P. H. Mussen (Series Ed.) and M. M. Haith & J. J. Campos (Vol. Eds.), Handbook of childpsychology: Vol. 2. Infancy and developmental psychobiology (pp. 435-57'1). New York: Wiley. Bench, J. (1968). Sound transmission to the human foetus through the maternal abdominal wall. Journal ofGenetic Psychology, 113, 85-87. Bench, J., Anderson, J., & Hoare, M. (1979). Measurement system for fetal audiometry. Journal of the Acoustical Society of America, 47, 1602-1606. Birch, H. G., & Lefford, A. (1967). Visual differentiation, intersensory integration, and voluntary control. Monographs ofthe Society for Research in Child Development, 32, (Serial No. 110). Bronson, G. W. (1982). Structure, status, and characteristics of the nervous system at birth. In P. Stratton (Ed.), Psychobiology ofthe human newborn (pp. 99-118). New York: Wiley. Colavita, F. B. (1974). Human sensory dominance. Perception and Psychophysics, 16, 409-412. Colavita, F. B., & Weisberg, D. (1979). A further investigation of visual dominance. Perception and Psychophysics, 25, 345-347. DeCasper, A. J., & Fifer, W. P. (1980). Of human bonding: Newborns prefer their mother's voices. Science, 208, 1174-1176. DeCasper, A. J., & Spence, M. J. (1986). Prenatal maternal speech influences newborns' perception of speech sounds. Infant Behavior and Development, 9, 133-150. Demany, L. (1982). Auditory stream segregation. Infant Behavior and Development, 5, 261-276. Demany, L., McKenzie, B., & Vurpillot, E. (1977). Rhythm perception in early infancy. Nature, 266,718-719. Egeth, H. E., & Sager, L. C, (1977). On the locus of visual dominance. Perception and Psychophysics, 22, 77-86. Gottlieb, G. (1971). Ontogenesis of sensory function in birds and mammals. In E. Tobach, L. R. Aronson, & E. Shaw (Eds.), Thebiopsychology of'development (pp. 67-126). New "Vbrk: Academic Press. Grimwade, J. C, Walker, D. W., Bartlett, M., Gordon, S., & Wood, C. (1971). Human fetal heart rate change and movement in response to sound and vibration. American Journal ofObstetrics and Gynecology 109, 86-90. Halpern, J., & Lantz, A. E. (1974). Learning to utilize information presented over two sensory channels. Perception and Psychophysics, 16, 321-328. Hay, J. C, Pick, H. L., & Ikeda, K. (1965). Visual capture produced by prism spectacles. Psychonomic Science, 2, 215-216. Hermelin, B., & O'Connor, N. (1964). Effects of sensory input and sensory dominance on severely disturbed, autistic children and on subnormal controls. British Journal ofPsychology, 55, 201 -206. Jack, C. E., & Thurlow, W. R. (1973). Effects of degree of visual association and angle of displacement on the "ventriloquism effect." Perceptual and Motor Skills, 37, 967-979. Jusczyk, P. W, Pisoni, D. B., Reed, M. A., Fernald, A., & Myers, M. (1983). Infants' discrimination of the duration of a rapid spectrum change in nonspeech signals. Science, 222, 175-176. Kasatkin, N. I. (1972). First conditioned reflexes and the beginning of the learning process in the human infant. In G. Newton & A. Riesen (Eds.), Advances in psychobiology (Vol. 1, pp. 213-257). New York: Wiley. Lewkowicz, D. J. (1985a). Bisensory response to temporal frequency in 4-month-old infants. Developmental Psychology, 21, 306-317. Lewkowicz, D. J. (1985b). Developmental changes in infants' visual response to temporal frequency. Developmental Psychology, 21, 858865.
INFANTS' PROCESSING OF MULTISENSORY COMPOUNDS Lewkowicz, D. J. (1986). Developmental changes in infants' bisensory response to synchronous durations. Infant Behavior and Development, 9, 335-353. Loveless, N. E., Brebner, J., & Hamilton, P. (1970). Bisensory presentation of information. Psychological Bulletin, 73, 161-199. McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746-748. Mendelson, M. J., & Ferland, M B. (1982). Auditory-visual transfer in four-month-old infants. Child Development, 53, 1022-1027. O'Connor, N., & Hermelin, B. (1965). Sensory dominance. Archives of General Psychiatry, 12,99-103. Olson, G. M , & Sherman, T. (1983). Auditory development and speech perception in infancy. In P. H. Mussen (Series Ed.) and M. M. Haith & J. J. Campos (Vol. Eds.), Handbook of child psychology: Vol. 2. Infancy and developmental psychobiology (pp. 1001-1080). New York: Wiley. Panneton, R. K., & DeCasper, A. J. (1986, March). Newborns'postnatal preference far a prenatally experienced melody. Paper presented at the International Conference on Infant Studies, Beverly Hills, CA. Posner, M. I., Nissen, M. J., & Klein, R. M. (1976). Visual dominance: An information-processing account of its origins and significance. Psychological Review, 83, 157-171. Radeau, M., & Bertelson, P. (1977). Adaptation to auditory-visual dis-
171
cordance and ventriloquism in semirealistic situations. Perception and Psychophysics, 22, 137-146. Spelke, E. S. (1979). Perceiving bimodally specified events in infancy. Developmental Psychology, 15,626-636. Spelke, E. S. (1981). The infant's acquisition of bimodally specified events. Journal of Experimental Child Psychology 31, 279-299. Spelke, E. S., Born, W. S., & Chu, F. (1983). Perception of moving, sounding objects by four-month-old infants. Perception, 12,719-732. Starr, A., Amlie, R. N., Martin, W. H., & Sanders, S. (1977). Development of auditory function in newborn infants revealed by auditory brainstem potentials. Pediatrics, 60, 831-839. Volokhov, A. A. (1968). Comparative studies of the functional development of analyzer systems in animals in the process of ontogenesis. Progress in Brain Research, 22, 527-554. Walker-Andrews, A. S. (1986). Intermodal perception of expressive behaviors: Relation of eye and voice? Developmental Psychology, 22, 373-377. Weitzman, E. D-, & Graziani, L. J. (1968). Maturation and topography of the auditory evoked response of the prematurely born infant. Developmental Psychobiology, 1, 79-89. Received February 16, 1987 Revision received July 14, 1987 Accepted July 23, 1987 •
Inconsistencies and Aggravations: Word Processing Manuscripts in APA Style If some of the requirements of APA editorial style and typing format, as described in the Publication Manual, conflict with the capabilities of your word-processing system, the APA Journals Office would like to hear from you. We are reexamining style and format requirements in the light of how easy or how difficult they are to implement by those preparing manuscripts on computers or word processors. For example, some systems cannot easily place the page number under the short title in the upper-right corner of each page. If you have had difficulties with this requirement or any others, we would greatly appreciate your taking the time to jot down which requirements have presented problems and what hardware and software systems you used to prepare the manuscript. Write to Leslie Cameron, Room 710, APA, 1400 North Uhle Street, Arlington, Virginia 22201.
