Brain Injury

This article was downloaded by:[University of Minnesota] On: 18 March 2008 Access Details: [subscription number 788779079] Publisher: Informa Healthcare Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Brain Injury Publication details, including instructions for authors and subscription information: http://www.informaworld.com/smpp/title~content=t713394000

Retrospective confidence judgements made by adults with traumatic brain injury: relative and absolute accuracy Mary R. T. Kennedy a a University of Minnesota, Minneapolis, MN, USA.

Online Publication Date: 01 June 2001 To cite this Article: Kennedy, Mary R. T. (2001) 'Retrospective confidence judgements made by adults with traumatic brain injury: relative and absolute accuracy', Brain Injury, 15:6, 469 - 487 To link to this article: DOI: 10.1080/02699050010007380 URL: http://dx.doi.org/10.1080/02699050010007380

PLEASE SCROLL DOWN FOR ARTICLE Full terms and conditions of use: http://www.informaworld.com/terms-and-conditions-of-access.pdf This article maybe used for research, teaching and private study purposes. Any substantial or systematic reproduction, re-distribution, re-selling, loan or sub-licensing, systematic supply or distribution in any form to anyone is expressly forbidden. The publisher does not give any warranty express or implied or make any representation that the contents will be complete or accurate or up to date. The accuracy of any instructions, formulae and drug doses should be independently verified with primary sources. The publisher shall not be liable for any loss, actions, claims, proceedings, demand or costs or damages whatsoever or howsoever caused arising directly or indirectly in connection with or arising out of the use of this material.

Downloaded By: [University of Minnesota] At: 16:44 18 March 2008

BRAIN INJURY,

2001,

VOL.

15,

NO.

6, 469± 487

Retrospective confidence judgements made by adults with traumatic brain injury: relative and absolute accuracy MARY R. T. KENNEDY University of Minnesota, Minneapolis, MN, USA (Received 22 July 2000; accepted 3 September 2000 ) Eighteen adult survivors of moderate or severe traumatic brain injury (TBI) and 18 controls studied and were tested on two lists of noun-pairs, and made item-by-item retrospective confidence judgements (RCJs) of their answers using a Likert scale. For both groups, correlations between RCJ ratings and recall were extremely high, indicating that survivors of TBI were as accurate as non-injured controls when comparing their confidence in one answer to another. However, as groups, survivors of TBI were over-confident and control participants were under-confident, but only when uncertain as to the accuracy of their answer. Recall errors of ìnterference’ were analysed post-hoc. Survivors of TBI displayed higher confidence in interference errors than in non-interference errors, while control participants did not. These results are discussed within the context of self-monitoring by clinical and non-clinical populations. The importance of both relative and absolute measures of metamemory accuracy is highlighted.

Introduction Various types of memory and learning impairment after traumatic brain injury (TBI) are well documented in the literature, ranging from attention and organizational deficits, forgetting newly learned information, to interference over trials and impaired transfer into long-term memory [1± 12]. In addition to memory impairment, deficits in metamemory processes in the TBI population are also receiving attention. Metamemory is cognition about one’s memory and can be viewed as a collection of various beliefs about one’s memory and learning, the ongoing awareness about one’s memory and learning, both from past experience and in the present during a learning activity, and memory control strategies, from which one makes selections in an effort to enhance remembering and learning [13, 14]. Clinicians, family members and researchers report that some survivors of TBI are ùnaware’ of the severity of their cognitive impairments and inflate their self-assessments [15± 25]. A survivor of TBI may have severe memory impairment and may or may not be àware’ of it. Presumably, if one is unaware of his or her limitations, then the executive decisions about the use of compensatory strategies may be faulty. Studies including adults with amnesia and adults with Korsakoff ’s disease have provided some evidence that self-monitoring of memory and memory performance Correspondence to: Mary R. T. Kennedy, PhD, Department of Communication Disorders and The Center for Cognitive Sciences, University of Minnesota, 115 Shevlin Hall, 164 Pillsbury Dr. S. E., Minneapolis, MN 55455, USA. e-mail: [email protected] Brain Injury ISSN 0269± 9052 print/ISSN 1362± 301X online # 2001 Taylor & Francis Ltd http://www.tandf.co.uk/journals DOI: 10.1080/02699050010007380


470

M. R. T. Kennedy

are separate processes. In a study by Shimamura and Squire [26], only adults with Korsakoff ’s disease were unable to predict their future memory recognition. Adults with amnesia demonstrated severely impaired memory, but maintained the ability to accurately self-monitor their poor memory. Findings from a study conducted by Janowsky et al. [27] expanded on this, by including a group with frontal lobe lesions without amnesia, along with those with Korsakoff ’s disease, those with amnesia and a control group. Participants learned sentences and took two delayed recall tests in which they provided the missing word from sentences they had studied. Although delayed recall was most impaired for adults with amnesia, adults with Korsakoff ’s disease demonstrated the poorest ability to predict recognition. Adults with frontal lobe lesions demonstrated poor predictive accuracy in the extended delayed recall test. Frontal lobe damage appears to interfere with self-monitoring, but the combination of memory impairment and frontal lobe damage as in Korsakoff ’s disease further reduces self-monitoring accuracy. This provides important information for clinicians rehabilitating survivors of diffuse brain injury, including those with frontal and temporal lobe damage. Clinicians have long held the view that survivors of TBI who do not accurately assess their own skill level, progress at a slower pace or do not reach similar eventual levels of success, when compared to those who make accurate self-assessments [28, 29]. In recent years, reports describing types of unawareness after TBI have emerged in the rehabilitation research literature [16, 22]. The majority of these studies used methodology in which self-reported symptoms by survivors of TBI were compared to the ratings of family members or clinicians, or were compared to the survivors’ actual performance in a given activity. For example, Sbordone et al. [23] reported that family members reported more serious symptoms within emotional, physical and cognitive/behavioural domains than did survivors of TBI. However, it appears that not all survivors are unrealistic about their cognitive capabilities. Some survivors overestimate, some underestimate, and some estimate their cognitive ability accurately [21]. Others found that the longer a survivor lived with a disability, the more likely they were to be realistic in their self-assessments. Allen and Ruff [25] found this to be the case with a group of survivors of TBI in judging their memory ability. Newman et al. [30] had a group of in-patient survivors of TBI make self-ratings shortly after admission to the rehabilitation unit. Compared to the ratings of clinicians, survivors’ ratings were inflated. Self-ratings more closely matched clinicians’ ratings at the time of discharge, but this was due to an improvement in performance rather than a change in self-ratings. While studies of general metacognitive beliefs have provided some evidence of overconfidence in survivors of TBI, only recently has self-monitoring during an activity been documented in the research literature. Comparing self-ratings to actual performance avoids the inherent methodological issues in studies that compare survivor’s self-ratings to the ratings of survivor’s abilities by a significant other [31]. In one study of self-monitoring during activities, Hart et al. [32] examined 18 survivors of TBI and control participants’ ability to detect errors while they performed everyday activities that increased in complexity. Participants made estimates of their performance in retrospect. Survivors detected fewer errors than controls. A modest correlation between ratings of physical ability and the number of actual errors (not detected errors) was found, whereas a weak correlation between ratings of cognitive ability and the number of actual errors was found. Hart et al. [32]


Confidence judgements after TBI

471

suggested that an impaired working memory after TBI could result in the inability to perform a cognitive activity and simultaneously self-monitor that activity. Three studies involving one or two survivors of TBI have investigated metamemory function during learning activities. Schlund [24] described a survivor of TBI who was very overconfident when making aggregated estimates of his memory. Immediately prior to (prospectively) and after the recall test (retrospectively), the survivor made aggregated estimates of the percentage that he would recall, or did recall. Estimates were also obtained 24 hours before and after each recall test. Performance and estimation feedback were provided after each test. In general, over-prediction and overconfidence declined soon after feedback was introduced and declined more in the immediate judgement condition than in the 24 hourdelayed condition. The overconfidence observed during retrospective estimations declined more substantially than the over-prediction observed during prospective predictions of future recall. However, over time, both types of estimations became either calibrated or represented under-confidence in recall. Giacino and Cicerone [18] described a survivor of TBI who reported that her memory was not seriously impaired, even though she exhibited considerable forgetfulness in her daily routine. Errors were brought to her attention during activities, providing her with direct, online feedback about memory performance. This not only resulted in improved self-monitoring during activities, but her general descriptions about her memory capability became increasingly realistic. In an earlier study, a method was used that allowed measurement of the itemby-time predictive accuracy of two survivors and two non-injured controls while learning lists of noun-pairs [33]. Judgements-of-learning ( JOL) predictions were made using a Likert rating scale of 0± 100 for each noun-pair (`How confident are you that in 8 minutes you will correctly recall the missing word, when you see this first word’? e.g., ocean - _____, Ì am _____% sure that I will recall the missing word in 8 minutes’ selected from 0, 20%, 40%, 60%, 80%, 100%) [34]. After studying each noun-pair, JOL predictions were made either immediately after studying or slightly delayed from studying. Both survivors of TBI recalled fewer noun-pairs than did either control participants. Correlations between delayed JOL predictions and recall was extremely high for control participants, while correlations between immediate JOL predictions and recall was much lower. One survivor recalled only 5.50% of noun pairs, but the correlation of predictive ratings with recall was high in both conditions. However, he was extremely over-confident when predicting recall of eventually incorrect items (average rating ˆ 83.50). The other TBI survivor recalled 36.07% of noun-pairs, but correlations between both types of predictions and recall were modest and she was moderately over-confident (average rating for incorrect items ˆ 40.00). Thus, not only does memory and metamemory accuracy vary independently, but relative and absolute measures of metamemory vary independently as well. Both measures are needed to fully describe metamemory function and its relationship to memory performance. In the current study, item-by-item retrospective confidence judgements (RCJs) were made about the accuracy of each answer, using a Likert rating scale. RCJs are similar to error detection judgements, in that they are made retrospectively (i.e., after an answer has been provided), but differ in that the scale allows individuals to indicate the degree of confidence, rather a binary choice of correct or incorrect. In general, the correlations between RCJs and recall in non-clinical populations (e.g., college students) are extremely high [35, 36]. It appears that self-monitoring during


472

M. R. T. Kennedy

RCJs reflects what the individual actually knows and is able to retrieve at the time the confidence rating is made [37]. However, when absolute measures of RCJ accuracy are used such as difference scores and calibration curves, overconfidence prevails. This trend has been repeatedly confirmed in studies in which undergraduates judge answers to general information and multiple choice questions [38± 40]. Furthermore, it appears that overconfidence can be reduced through the use of feedback. For example, Schraw and Roedel [41] provided some students with performance feedback alone, and other students with both performance feedback and feedback that described the difference between their confidence judgements and performance (i.e., difference scores). Students in the latter feedback condition demonstrated the largest reduction in overconfidence. Therefore, not only are RCJs highly correlated with recall performance (describing the relative relationship between RCJs and recall), but also overconfidence can be reduced with specific forms of feedback. The general purpose of the current study was to expand upon earlier investigations of metamemory beliefs, in an effort to describe self-monitoring during a verbal learning activity by having survivors of TBI make item-by-item confidence judgements about the accuracy of their answers. The study described here was a part of a larger project in which participants made item-by-item RCJs during a recall test, as well as immediate and delayed JOL predictions during studying [20]. The methods and results of making RCJs are reported here, although the results of JOL predictive accuracy has some bearing theoretically and clinically to RCJ accuracy. Even though survivors recalled fewer noun-pairs, they were as highly accurate as noninjured controls at predicting recall when predictions were delayed from studying, and were similarly as inaccurate when predictions were made immediately after studying (i.e., the `delayed JOL effect’). Absolute measures of predictive accuracy revealed that survivors were well-calibrated when making delayed predictions, but over-estimated when making immediate predictions, while non-injured controls under-estimated in both prediction conditions [20]. These results suggested that, under certain conditions, some aspects of metamemory remained unaffected by diffuse injury to the brain. Additionally, this finding provided evidence that survivors’ ònline’ self-monitoring is similar to how non-clinical populations self-monitor. As others have suggested, when working memory is actively engaged in initial learning (as when making immediate JOL predictions), self-monitoring is at best marginally accurate. However, when working memory is not employed and self-assessment s are made from what has been stored (as when making delayed JOL predictions), monitoring accuracy improves dramatically [20, 34, 42]. Since RCJs are similar to delayed predictions, in that they are in part based on what can be retrieved from long-term memory at the moment of self-monitoring, survivors’ confidence was expected to be highly correlated with recall performance. But, unlike predictions, RCJs are made after an overt answer is provided and so we were interested in determining if over-confidence would emerge [35± 41]. Specifically, the questions were: . Will survivors’ relative confidence judgement accuracy be similar to noninjured controls? . Will survivors’ absolute confidence judgement accuracy reflect over-confidence, under-confidence or calibration in recall?


473

Method Downloaded By: [University of Minnesota] At: 16:44 18 March 2008

Participants Thirty-six adults participated in this study, consisting of 18 survivors of TBI (11 men and seven women, mean age ˆ 37.73, SD 10.81) and 18 adults with no history of brain damage (nine men and nine women, mean age ˆ 37.22, SD 10.52). Table 1 contains demographic information and standardized test scores. No participant had a known history of learning disabilities , stroke, neurological disease, psychiatric difficulty or chemical dependency. Control participants were matched based on age, education and occupation. There was a wide range of occupations represented in both groups, including professions of construction, mechanics, business and sales, health-care and education. Survivors of TBI were recruited based on the diagnosis of moderate or severe closed TBI and not on the basis of executive dysfunction. Nine survivors had sustained moderate injuries and nine had sustained severe injuries [43, 44]. Injury information is described in detail in Kennedy and Yorkston [20]. All survivors had been hospitalized. With the exception of one participant, all were receiving outpatient cognitive rehabilitation. None had successfully returned to work at the time of the study. Estimates of participants of verbal IQ were based on criteria proposed by Barona et al. [45] and were essentially the same for the two groups. No survivors had aphasia (see aphasia quotients, Western Aphasia Battery or WAB) [46]. Reading comprehension was screened using the WAB and no survivors of TBI or controls scored less than 32 (out of a possible 40). Survivors’ receptive vocabulary scores from the Quick Test [47] were lower than controls, t…34†, p ˆ 0:001. Immediate recall scores from the Logical Memory subtest of the Wechsler Memory Scale: Revised [48] indicated that survivors recalled less information than control participants, t…34†, p ˆ 0:005. Additionally, survivors were slower than controls by Trailmaking, Part B [49], t…34†, p ˆ 0:007. General procedures As a part of the large metamemory study, participants made both JOL predictions about the likelihood of recalling each item in the future, as well as RCJs about the accuracy of their answers during recall tests. Methods for generating RCJs are described here within the context of this large metamemory study [20]. The study was conducted on a Mac Portable computer using a customized software HyperCard program. Seated adjacent to the experimenter, participants read the instructions while the experimenter read them aloud. The angle of the screen could be adjusted for comfortable viewing. A font size of 20 was used throughout the experiment. Participants practised making confidence judgements using the rating scale described below without any difficulty. List construction and study time Participants studied two different lists of 40 unrelated noun-pairs, during which they made JOL predictions during the study phase and RCJs during the delayed recall tests [20]. Eighty pairs of unrelated nouns were randomly selected from a pool by the computer. Nouns were selected based on imagery (ranging from 5.10± 6.90) and concreteness (ranging from 6.08± 7.00) [50]. Primacy effects were controlled for by

g

na na na

65.17 138.14 8± 522

Weeks postinjury

t…34† ˆ 0:13, p ˆ 0:90; b t…34† ˆ ¡1:30, p ˆ 0:20; t…34† ˆ ¡1:72, p ˆ 0:10.

37.22 10.52 21± 50

9/9

Control M SD Range

a

37.72 10.81 23± 55

Agea

11/7

Gender (M/F)

TBI M SD Range

Group

c

Table 1.

d

66.72 26.20 3rd± 97th

42.72 21.66 10th± 86th

e

75.22 15.49 45th± 98th

50.94 24.89 8th± 85th

Quick test percentile f

t…34† ˆ 3:08, p ˆ 0:006;

99.28 21.95 46± 162

149.72 66.01 69± 308

Trails B secondse

t…34† ˆ ¡2:99, p ˆ 0:005;

110.79 5.31 101.07± 118.66

107.82 8.05 98.23± 118.66

Estimated IQc

t…34† ˆ 0:54, p ˆ 0:59;

14.61 2.15 12± 18

14.11 2.25 12± 18

Education (years)b

Logical memory percentiled

Demographic characteristics of TBI and control participant groups

f

40.00 0.00 na

38.22 4.39 24± 40

Reading subtestg

t…34† ˆ ¡3:51, p ˆ 0:001;

na na na

98.62 1.48 94.4± 100

Aphasia Quotient

Western Aphasia Battery


474 M. R. T. Kennedy



475

excluding the first four noun-pairs from the recall test, which resulted in 36 nounpairs being tested, and RCJs made. Delaying the recall test with conversation such that there was at least 2 minutes between the last pair studied and the first noun-pair to be tested, controlled for recency effects. A blocking and randomization technique for studying each list of noun-pairs was employed, similar to studies in which item-by-item judgements are made. During the study phase, noun-pairs were first randomly divided into blocks of 18 and then into blocks of nine, only so that immediate and delayed JOL predictions could be made [20, 34]. Participants studied each list twice, with each pair presented at the designated time interval determined in the following manner. Preliminary testing indicated that the amount of study time would need to differ between the two groups so that floor effects would be avoided with survivors of TBI and ceiling effects would be avoided with control participants. In order to generate stable correlations between recall and confidence ratings, floor and ceiling effects had to be avoided [51]. Additionally, there is some evidence that the level of encoding influences the relative accuracy of metamemory judgements. (Weaver and Bryant [52] found that undergraduates’ metamemory accuracy was highest when reading text at a standard level of difficulty and was lowest when reading èasy’ or `difficult’ text. To avoid this, recall performance needed to be in a range of 25± 75% for both groups.) Therefore, survivors studied each item for 13 seconds, and control participants studied each item for either 3 or 5 seconds, depending on their age and years of education. Control participants who were 40 years old or younger with education beyond a high school degree, studied each noun-pair for 3 seconds. Control participants who were 40 years old or younger with a high school degree only and those who were older than 40 years, regardless of education, studied each noun-pair for 5 seconds. Making retrospective confidence judgements during the recall test Participants took a delayed, cued recall test in which the first noun of the pair provided a cue for the missing second noun (e.g., ocean ). The recall ? test was 8± 10 minutes from the time the noun-pair had been studied for the second time. This was accomplished by testing all the pairs from block 1 prior to testing the pairs from block 2. Because self-paced JOL predictions were made while studying the second time through, occasionally a participant completed the task in less than 8 minutes. When this occurred, additional time was spent conversing with the investigator. The order in which noun-pairs were tested involved re-randomizing the order within each block of 18 (in which they had been studied) by thirds. This meant that the first six noun-pairs in block 1 were tested first, followed by the second six nounpairs and then the third six noun-pairs. After all had been tested and RCJs made, pairs were tested and RCJs made from block 2 (re-randomized by thirds). This allowed for some control over the amount of time between studying, testing and making a RCJ for any one noun-pair, although the recall test was self-paced. Participants provided answers and the experimenter typed them in. Errors of omission were not allowed, which encouraged guessing. With the answer remaining on the screen, RCJs were made using the following query: ocean (participant’s recalled answer)

ocean -

tree

476

M. R. T. Kennedy


`How confident are you that your answer is correct?’ 0 20 40 60 80 100

ˆ absolutely sure the answer is incorrect, ˆ 20% sure the answer is correct, ˆ 40% sure the answer is correct, ˆ 60% sure the answer is correct, ˆ 80% sure the answer is correct, ˆ 100% sure the answer is correct.

(participant’s RCJ)

I am 80% confident.

This sequence was followed until all 36 noun-pairs had been tested and RCJs made for items on list 1. The same sequence was followed for list 2.

Reliability and data analysis All sessions were tape recorded. Twenty-two per cent (four survivors and four controls) of the experimental sessions were randomly selected to check for reliability and procedural consistency. Tape recordings were compared with test answers and RCJs, resulting in 100% item-by-item reliability for both. Additionally, experimental procedures and instructions were consistent across sessions. Because the author was interested in both relative and absolute RCJ accuracy, two measures were used: the Goodman-Kruskal gamma correlation (G) and difference scores. Gamma correlations were generated for each participant, providing a measure of each individual’s relative RCJ accuracy [51, 53]. G describes the yoked relationship between confidence in one’s answer and the accuracy of the answer compared to the confidence in another answer and the accuracy of that answer. Difference scores were also generated for each participant, by subtracting the overall per cent recalled from the average RCJ rating. Thus, difference scores measure the absolute accuracy of RCJs [42]. Scores that fall below 0 represent underconfidence and scores that fall above 0 represent over-confidence. For example, if the average RCJ rating was 60 and 30% was correctly recalled, creating a difference score of ‡30, this would suggest over-confidence in recall ability. Calibration curves were used to describe absolute RCJ accuracy for groups, by plotting per cent recalled for every RCJ rating. These curves provide a visual display of groups’ recall collapsed across subjects, along the continuum of RCJ ratings [35]. Perfect calibration is when the overall proportion recalled is the same as the RCJ rating, e.g., 40% recalled for RCJ ratings of 40% confidence. Coordinates falling below the diagonal line (perfect calibration) indicate group over-confidence, while coordinates falling above perfect calibration indicate group under-confidence.

Results Each participant’s proportion recalled, gamma correlations of RCJ ratings with recall and difference scores for both lists were generated (see table 2). A Group (survivors of TBI and controls) £ List (1 and 2) repeated measures analysis of variance (ANOVA) was calculated for proportion recalled, gamma correlations and difference scores.



Table 2.

477

Individual participant data by group across lists

Proportion recalled

Gamma correlations

Group

List 1

List 2

List 1

TBI 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 M SD

0.75 0.42 0.17 0.64 0.42 0.22 0.53 0.06 0.14 0.22 0.81 0.78 0.11 0.00 0.19 0.72 0.67 0.47 0.41 0.27

0.58 0.75 0.22 0.56 0.69 0.39 0.47 0.00 0.17 0.25 0.86 0.67 0.11 0.00 0.17 0.92 0.92 0.53 0.46 0.31

1.00 1.00 1.00 1.00 0.98 0.97 0.95 1.00 0.99 1.00 0.99 0.98 1.00 na 1.00 0.93 1.00 0.97 0.99 0.02

Control 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 M SD

0.78 0.56 0.22 0.72 0.58 0.61 0.86 0.28 0.92 0.33 0.92 0.39 0.86 0.81 0.42 0.64 0.56 0.75 0.62 0.22

0.81 0.53 0.28 0.64 0.50 0.58 0.78 0.31 0.89 0.56 0.83 0.50 0.69 0.67 0.33 0.81 0.44 0.64 0.60 0.19

0.96 0.97 1.00 0.98 1.00 1.00 0.99 0.96 1.00 1.00 1.00 0.99 0.90 0.99 1.00 1.00 1.00 1.00 0.99 0.03

List 2

Difference scores List 1

List 2

1.00 1.00 1.00 1.00 1.00 1.00 1.00 na 1.00 1.00 1.00 0.85 1.00 na 1.00 1.00 0.97 0.95 0.99 0.04

715.00 7.74 2.19 3.32 4.97 10.02 5.53 3.29 71.10 31.13 71.71 73.36 12.23 6.17 72.18 10.02 6.66 6.11 5.93 10.85

73.86 70.56 70.55 4.96 4.49 0.54 74.98 5.61 2.77 36.11 71.66 6.08 5.01 3.39 72.26 2.19 2.23 77.22 2.94 9.17

1.00 0.95 1.00 1.00 1.00 1.00 1.00 0.98 0.97 1.00 1.00 1.00 0.97 0.98 1.00 1.00 1.00 1.00 0.99 0.02

71.67 2.77 73.33 71.11 740.55 70.55 73.33 0.00 76.11 73.89 73.33 3.89 79.44 79.44 73.33 2.22 71.11 71.67 74.44 9.70

76.11 1.11 70.56 77.22 731.67 4.44 2.22 70.56 78.33 77.78 73.89 1.67 72.78 8.89 72.22 1.67 70.56 0.00 0.37 5.60

Note: `na’ indicates that the gamma correlation was indeterminable.

Recall performance Survivors of TBI recalled significantly less than control participants recalled, resulting in a main effect for Group. No significant differences were found for List or List £ Group interaction (see table 3).

478

M. R. T. Kennedy Table 3.

Repeated measures analysis of variance for recall, gamma correlations and difference scores


Source Between subjects Group Error Within subjects List List £ Group Error

df

Proportion recalled

Gamma correlation

Difference score

1 34

F 4.80* (0.12)

F 0.21 (0.01)

F 7.00** (146.19)

1 1 34

0.5 3.31 (0.01)

0.34 0.19 (0.01)

0.02 2.62 (20.10)

Note: Values in parentheses represent the mean square value for error. *p < 0:05; ** p < 0:01.

Relative RCJ accuracy Gamma correlations revealed that both groups were extremely accurate when comparing their confidence in one item to another, indicated by the near perfect correlations on both lists. No significant effects were found for Group, List, or Group £ List interaction (see table 3). Therefore, the relative accuracy of survivors’ confidence in their answers was as high as control participants’ relative accuracy, i.e. both groups were extremely accurate when judging the correctness of one answer compared to another answer. Absolute RCJ accuracy Difference scores As a group, survivors difference scores were higher than control participants’ difference scores, indicated by a significant group effect. This occurred in the absence of a List effect (see table 3). There was a tendency for control participants to move closer towards calibration on list 2 than survivors of TBI, although it was not statistically significant (see figure 1). Thus, when comparing the use of the rating scale to overall recall, the survivors group was slightly over-confident, while the control group was slightly under-confident. However, within-group variability was a large. Calibration curves As described earlier, calibration curves provide a visual display of group recall along a continuum of confidence ratings. Four curves were generatedÐ for each group and list. The equation, y ˆ 1:0x ‡ 0 describes perfect calibration, in which the per cent correctly recalled matches the rating (e.g., 20% of the items receiving a confidence rating of 20% are correctly recalled). Groups’ calibration curves were similar across lists 1 and 2. Table 4 includes regression statistics and ANOVA for regression by group and list. The ¢R 2 values ranged from 0.86± 0.95, indicating high proportions of the variance explained by the curves. While these values were similar across lists, directionality differed by group, as noted in the regression equations. Figure 2 displays calibration curves by group, collapsed across lists. When absolutely certain in the accuracy or inaccuracy of their answer, both groups were perfectly calibrated. That is, when using 0 or 100, indicating that they were absolutely certain the answer they had provided was incorrect or correct, the amount correctly recalled was 0% or 100%! However, when less than absolutely certain, both groups deviated



Figure 1.

479

TBI and control participants’ average difference scores for list 1 and list 2, with vertical bars representing standard deviations.

Table 4.

Statistics describing calibration curves of recall performance with RCJ ratings by list Analysis of variance for regression

Regression statistics Group TBI List 1 List 2 Control List 1 List 2

Equation

Multiple R

R2

Adjusted R 2

df

F

p

y ˆ 1:05x ¡ 10:86 y ˆ 1:10x ¡ 11:80

0.98 0.98

0.96 0.96

0.95 0.94

1 1

93.45 85.45

0.001 0.001

y ˆ 0:96x ‡ 17:24 y ˆ 1:03x ‡ 8:20

0.94 0.97

0.89 0.95

0.86 0.94

1 1

32.27 74.78

0.005 0.001

from perfect calibration. Survivors’ calibration of recall with confidence ratings was an inverted image of control participants’ calibration. When less than absolutely certain of recall, survivors of TBI were over-confident about the accuracy of their answers, while control participants were under- confident.

Post-hoc analyses Because survivors of TBI displayed overconfidence when unsure in their answer, it was possible that ìnterference’ from other answers may have contributed survivors selection of high RCJs. To examine this, all incorrect answers from the recall test were identified as interference or non-interference. Answers coded as ìnterference’ were words from other lists or from the same list, including words in which only the suffix was omitted (e.g., `paint’ instead of `painter’). Proportions of interference errors were calculated first. The average proportion of interference errors by survivors of TBI was 0.41, SD 0.18 and the average proportion of interference errors by


480

M. R. T. Kennedy

Figure 2. Calibration curves collapsed across lists, plotting per cent recalled for each confidence rating for TBI and control participants. Numbers at data points represent the total number of times each rating was used by each group.

Figure 3.

Average RCJ ratings for errors attributable to interference, and errors not attributable to interference for both participant groups, with vertical bars representing standard deviations.

controls was 0.44, SD 0.25, t…33† ˆ ¡0:37, p ˆ 0:71. Thus, both groups generated proportionately similar amounts of errors that could be attributed to interference. The mean RCJ rating was calculated for interference and non-interference errors (see figure 3). A Group £ Interference repeated measures ANOVA was calculated with lists collapsed (due to the similarity of the calibration curves across lists). A significant group effect, F…1; 33† ˆ 5:75, p ˆ 0:02, a significant interference effect, F…1; 33† ˆ 6:44, p ˆ 0:02, and a significant group £ interference effect, F…1; 33† ˆ 6:90, p ˆ 0:01, were found. As can be seen in figure 3, survivors of TBI assigned higher RCJ ratings to their errors overall. However, the highly sig-



481

nificant interaction indicates that survivors were more confident in interference errors than non-interference errors, while control participants’ confidence was lower and did not depend on the presence of interference. Additional post-hoc tests were performed involving demographic variables. Between severity group (moderate and severe injuries), ANOVAs were calculated on gamma correlations and difference scores for both lists. No significant effects for severity were found …p 4 0:05†. Correlations between demographic variables (age, education, estimated IQ, story recall, Trails B performance and receptive vocabulary), gamma correlations and difference scores were also calculated. Demographic variables and gamma correlations were weakly correlated, ranging from 70.09± 0.17. There was a moderately strong negative correlation between immediate story recall (WMS-R) and difference scores on list 1 and 2, respectively (70.41 and 70.35), indicating a relationship between story recall and low difference scores. Discussion Survivors of moderate and severe TBI recalled less than non-injured control participants did; however, both groups’ confidence was extremely accurate when comparing one item to another. There are several possible reasons for this finding. In comparison to prospective judgements (e.g., immediate and delayed judgements-oflearning), RCJs are made after having actively searched, retrieved and compared the strength of the answer to the belief about the correct answer. More information is available on which to base judgements made in retrospect. These additional steps prior to self-monitoring may facilitate accurate RCJs [35]. Survivors of TBI appear to benefit from these processing steps, as do non-injured adults. Another possible reason for the relative RCJ accuracy of survivors of TBI is that these participants had additional time to study the items. In an effort to protect against floor effects, survivors studied each noun-pair for 13 seconds. To protect against ceiling effects, non-injured controls studied each noun-pair for 3 or 5 seconds. Still, survivors recall was lower than that of controls. The additional time spent encoding may have enhanced the TBI group’s ability to monitor each answer. However, the aim was also to guard against simplifying the task or making it too difficult because, in a study by Weaver and Bryant [52], student’s metamemory monitoring was poorest under these two conditions. On-line metamemory studies of undergraduates using noun-pairs have employed a range of study times (4± 8 seconds) without an apparent effect on metamemory accuracy [34, 42, 54]. It is unclear without further investigation whether RCJ accuracy would be affected if encoding were made more difficult for survivors. But, it appears that under optimal encoding conditions (evidenced by recall just below 50%), relative RCJ accuracy is preserved for at least some survivors of TBI. Difference scores, which provided absolute measures of RCJ accuracy differentiated survivors of TBI from non-injured controls. As a group, survivors were slightly over-confident in their answers on both lists, while non-injured controls were slightly under-confident. However, variability within both groups was large. Some survivors were under-confident and some non-injured controls were overconfident. This provided additional support of earlier findings that while some survivors are over-confident, others are under-confident or well calibrated [21]. Calibration curves provided a means for examining certainty and uncertainty between groups. When survivors of TBI and non-injured controls were certain in


482

M. R. T. Kennedy

the accuracy of their answers (RCJ ratings of 0 and 100), recall was well calibrated. The over-confidence of survivors of TBI and the under-confidence of non-injured controls occurred when using ratings indicating uncertainty in their answers (20± 80). Although it was not the initial intent of this investigation to determine reasons for over-confidence, it appears that the ìnterference’ of words from other lists or from the same list may have contributed to TBI survivors’ overconfidence. Interference during list learning is well documented in the TBI population (e.g., [9, 11, 12]). Survivors of TBI rated errors attributed to interference with greater confidence than non-interference errors, unlike control participants. That is, survivors of TBI were more confident when the answers were interference errors, than they were when answers were not interference errors. Several researchers from cognitive psychology have suggested that when several items are activated during recall attempts, determining accuracy and assigning an appropriate confidence rating is more difficult than if just one or two alternative answers had been activated [14, 35, 55]. Thus, with answers remaining activated and ìnterfering’ during the recall test, judging confidence accurately in this study became difficult. With both groups of participants having proportionately the same amount of ìnterference’, one can only speculate why survivors assigned higher ratings especially to interference errors. It is possible that survivors of TBI made RCJ based on their isolated confidence in that particular answer, rather than considering all that the task had involved. Koriat et al. [55] suggested that individuals tend to be confident in any answer that is retrieved at the moment, when other task conditions are not considered or weighed prior to the confidence judgement. For example, only undergraduates who were required to generate alternative answers to questions showed improvement in confidence calibration by selecting lower RCJs. Students who did not generate alternative answers did not improve in their calibration [55]. In the current study, perhaps, control participants who experienced as much ìnterference’ as survivors of TBI, òverrode’ the confidence they had in their interference errors by considering the difficulty of the task prior to making a confidence judgement. Perhaps survivors of TBI did not weigh their confidence against the difficulty of the task and instead based their RCJ on their confidence in that isolated answer. Making accurate confidence judgements includes considering task conditions while searching for and retrieving an answer and comparing it to other possible answers. This type of selfmonitoring involves several executive functions such as distributing attention resources and working memory. Only future research will determine the extent to which these processes interact when self-monitoring during learning. Another explanation for over-confidence may be related to the relationship between metamemory beliefs and the use of beliefs during a learning experience [13]. Acknowledging that `my memory is not as good as it used to be’ is a statement of belief (and every TBI survivor made some such statement during this study). Several researchers have suggested that there is a disassociation between `types’ of awareness or the beliefs of poor memory and the use of these beliefs ònline’ [16, 56]. If survivors of TBI had used this metamemory belief when they were uncertain, they would have selected lower confidence ratings. It is also possible that survivors of TBI may have believed that their memory was poor and expected to perform `poorly’. But, when they began to recall more than they expected, they overcompensated by selecting high RCJ ratings.



483

This explanation may also assist in understanding controls participants’ underconfidence. Perhaps control participants had expected to recall most of the items and, when they did not, they became less confident in answers they were not absolutely certain of. They spontaneously used this new metamemory experience (recalling over ¹50%) to alter their metamemory belief about the experience (Ì’m not recalling as many as I should’) and, therefore, assigned ratings lower than their actual recall. Absolute measures of metamemory accuracy appear to reflect a combination of beliefs and task expectations [14]. Participants in this study also made judgement-of-learning ( JOL) predictions of recall described in detail in Kennedy and Yorkston [20]. Correlations between recall and delayed JOL predictions were as high as correlations between recall and RCJs for survivors of TBI and non-injured controls. RCJs are made after the answer is provided, and delayed JOL predictions are made after studying, but both are based on internal attempts to retrieve the answer from long-term memory. The underlying processes of delayed JOLs and RCJs may be very similar. Alternatively, correlations between recall and RCJs were much higher than correlations between immediate JOL predictions and recall. These findings are consistent with other studies involving non-injured adults [36, 57]. However, survivors were more closely calibrated when making delayed JOL predictions than when making RCJs, in which they were overconfident. When making immediate JOL predictions of recall, survivors were even more over-confident than they were when making RCJs. The over-confidence found in RCJs occurred during times of uncertainty, while over-confidence found in immediate JOL predictions occurred when they were absolutely certain about future recall [20, 58]. Gamma correlations in the current study and in Kennedy and Yorkston [20] identified types of metamemory monitoring ability that may be preserved following TBI. However, absolute measures were useful in identifying the differences between injured and non-injured adults. Absolute measures of accuracy are used as a group of ratings, in the form of means and/or aggregated items. They may reflect the developing or disassociated relationship between metamemory knowledge and metamemory experiences, rather than the ònline’ item-by-item comparison gained from the gamma correlation. Further empirical investigations of adults recovering from TBI using techniques such as interviews about metamemory knowledge, knowledge of recall results, and self-ratings of performance over trials could provide important clinical and theoretical information. Caution should be exercised in generalizing these results to the entire adult TBI population. Many injured adults do not recover to the same extent as these TBI participants. Although these TBI participants had sustained moderate or severe injuries and were receiving outpatient rehabilitation services, none were hospitalized at the time of the study. The findings of the current study can be generalized to adults with these levels of injury, but who have made sufficient cognitive recovery to complete the task. For these individuals, metamemory study skills are particularly important as they attempt to return to work or school. Clinical implications The findings of the current study can be generalized to recall and learning used in daily living, with caution. Whilst word lists allow for experimental control, they do not reflect easy-, standard-, and difficult-to-remember kinds of material one


484

M. R. T. Kennedy

encounters everyday. The empirical application of RCJs and other types of metamemory judgements, to a variety of functional activities, is needed before the relationship between these metamemory judgements and recall in everyday living will be fully understood e.g., [59]. Even so, identifying retrospective and prospective self-monitoring judgements that are the most and least accurate after a TBI provides valuable clinical information that could be incorporated into training individuals in self-regulated learning. Based on this study and on Kennedy and Yorkston’s [20] findings, survivors could be trained to only rely on their `confidence’ when generated from a self-quiz (e.g. confidence judgements or delayed predictions), and not when the information is being rehearsed in working memory (e.g. predictions made immediately after studying). If confidence judgements in recall or delayed JOL predictions of future recall are equally accurate, then survivors of TBI could rely on either type of monitored information to direct them to use compensatory memory techniques. However, when making confidence judgements, survivors of TBI had a false sense of confidence when they were uncertain. If even slightly uncertain about the correctness of the answer, the individual should take a `conservative’ approach and use a compensatory strategy to prevent future recall failure. Item-by-item judgements made retrospectively or prospectively provide specific self-generated feedback during learning acquisition that could assist individuals in regulating their own learning [60, 61]. Being able to accurately monitor his/her own learning does not mean that the self-monitored information will be spontaneously used to drive decisions about strategies. Currently, studies are underway to determine if survivors of TBI use item-by-item self-monitoring in the same way that non-clinical populations do, when making decisions about strategies. Clearly, further research is needed in a variety of areas, if clinicians, family members and survivors of TBI are to understand the on-going relationships between metamemory beliefs, metamemory experiences and the use of these experiences to enhance self-regulated learning. Acknowledgements This study was conducted in partial fulfillment of the author’s doctoral dissertation at the University of Washington, Seattle, WA. The author is grateful to Kathryn M. Yorkston, as advisor and mentor. Gratitude is extended to Edward Carney for his statistical consultation and to all the participants who willingly gave of their time. Portions of this study were presented at the Clinical Aphasiology Conference, 1997, Flathead Lake, Montana. References 1. Levin, H. S. and Goldstein, F. C.: Organization of verbal memory after severe closed-head injury. Journal of Clinical and Experimental Neuropsychology, 8: 643± 656, 1986. 2. Auerbach, S. H.: Neuroanatomical correlates of attention and memory disorders in traumatic brain injury: an application of neurobehavioural subtypes. Journal of Head Trauma Rehabilitation, 1: 1± 12, 1986. 3. Brooks, D. N.: Long and short-term memory in head injured patients. Cortex, ii: 329± 340, 1975. 4. Haut, M. and Shutty, M. S.: Patterns of verbal learning after closed head injury. Neuropsychology, 6: 51± 58, 1992.



485

5. Lezak, M. D.: Recovery of memory and learning functions following traumatic brain injury. Cortex, 15: 63± 72, 1979. 6. Reid, D. R. and Kelly, M. P.: Wechsler Memory Scale-Revised in closed head injury. Journal of Clinical Psychology, 49: 245± 254, 1993. 7. Roth, D. L., Conboy, T. J., Redder, K. P. et al.: Confirmatory factor analysis of the Wechsler Memory Scale-Revised in a sample of head-injured patients. Journal of Clinical and Experimental Neuropsychology, 12: 834± 842, 1990. 8. Millis, S. R. and Ricker, J. H.: Verbal learning patterns in moderate and severe traumatic brain injury. Journal of Clinical and Experimental Neuropsychology, 16: 498± 507, 1994. 9. Crosson, B., Sartor, K. J., Jenny III, A. B. et al.: Increased intrusions during verbal recall in traumatic and nontraumatic lesions of the temporal lobe. Neuropsychology, 7: 193± 208, 1993. 10. Gronwall, D. and Wrighton, P.: Memory and information processing capacity after closed head injury. Journal of Neurology, Neurosurgery, and Psychiatry, 44: 889± 894, 1981. 11. Blachstein, H., Vakil, E. and Hoofien, D.: Impaired learning in patients with closed-head injuries: an analysis of components of the acquisition process. Neuropsychology, 7: 530± 535, 1993. 12. Crosson, B., Novack, T. A., Trenerry, M. R. et al.: California Verbal Learning Test (CVLT) performance in severely head-injured and neurologically normal adults males. Journal of Clinical and Experimental Neuropsychology, 10: 754± 768, 1988. 13. Flavell, J. H.: Metacognition and cognitive monitoring: a new area of cognitive-developmental inquiry. American Psychologist, 34: 906± 911, 1979. 14. Nelson, T. O. and Narens, L.: Metamemory: a theoretical framework and new findings. The Psychology of Learning and Motivation, 26: 125± 141, 1990. 15. Boake, C., Freeland, J. C., Ringholz, G. M. et al.: Awareness of memory loss after severe closed-head injury. Brain Injury, 9: 273± 283, 1995. 16. Crosson, B., Barco, P. B., Bolesta, M. M. et al.: Awareness and compensation in postacute head injury rehabilitation. Journal of Head Trauma Rehabilitation, 4: 46± 54, 1989. 17. Fleming, J. M., Strong, J. and Ashton, R.: Self-awareness of deficits in adults: how best to measure it? Brain Injury, 10: 1± 15, 1996. 18. Giacino, J. T. and Cicerone, K. D.: Varieties of deficit unawareness after brain injury. Journal of Head Trauma Rehabilitation, 13: 1± 15, 1998. 19. Hillier, S. L. and Metzer, J.: Awareness and perceptions of outcomes after traumatic brain injury. Brain Injury, 11: 525± 536, 1997. 20. Kennedy, M. R. T. and Yorkston, K. M.: Accuracy of metamemory after traumatic brain injury: predictions during verbal learning. Journal of Speech, Language, and Hearing Research, 43: 1072± 1086, 2000. 21. Prigatano, G. P. and Altman, I. M.: Impaired awareness of behavioural limitations after traumatic brain injury. Archives of Physical Medicine and Rehabilitation, 71: 1058± 1964, 1990. 22. Schacter, D. L.: Unawareness of deficit and unawareness of knowledge in-patients with memory disorders. In: G. P. Prigatano and D. L. Schacter (editors) Awareness of Deficit after Brain Injury (New York: Oxford University Press), pp. 127± 151, 1991. 23. Sbordone, R. J., Seyranian, G. D. and Ruff, R. M.: Are the subjective complaints of traumatically brain injured patients reliable? Brain Injury, 12: 505± 515, 1998. 24. Schlund, M. W.: Self awareness: effects of feedback and review on verbal self-reports and remembering following brain injury. Brain Injury, 13: 375± 380, 1999. 25. Allen, C. C. and Ruff, R. M.: Self-rating versus neuropsychological performance of moderate versus severe head-injured patients. Brain Injury, 4: 7± 17, 1990. 26. Shimamura, A. P. and Squire, L. R.: Memory and metamemory: a study of the feeling-ofknowing phenomenon in amnesic patients. Journal of Experimental Psychology: Learning, Memory, and Cognition, 12: 452± 460, 1986. 27. Janowsky, J. S., Shimamura, A. P. and Squire,L. R.: Memory and metamemory: comparisons between patients with frontal lobe lesions and amnesic patients. Psychobiology, 17: 3± 11, 1989. 28. Ben-Yishay, Y. and Diller, L.: Cognitive rehabilitation in traumatic brain injury: update and issues. Archive of Physical Medicine and Rehabilitation, 74: 204± 213, 1993. 29. Cicerone, K. D. and Tupper, D. E.: Neuropsychological rehabilitation: treatment of errors in everyday functioning. In: D. E. Tupper, and K. D. Cicerone (editors) The Neuropsychology of Everyday Life: Issues in Development and Rehabilitation (Boston: Kluwer Academic Publishers), pp. 271± 292, 1991.


486

M. R. T. Kennedy

30. Newman, A. C., Garmoe, W., Beatty, P. et al.: Self-awareness of traumatically brain injured patients in the acute inpatient rehabilitation setting. Brain Injury, 14: 333± 334, 2000. 31. Trosset, M. W. and Kasniak, A. W.: Measures of deficit unawareness for predicted performance experiments. Journal of the International Neuropsychological Society, 2: 315± 322, 1996. 32. Hart, T., Giovannetti, T., Montgomery, M. W. et al.: Awareness of errors of naturalistic action after traumatic brain injury. Journal of Head Trauma Rehabilitation, 13: 16± 28, 1998. 33. Kennedy, M. R., Yorkston, K. M. and Rogers, M.: Self-monitoring abilitie s of two adults with traumatic brain injury during verbal learning. American Journal of Speech± Language Pathology, 4: 159± 163, 1995. 34. Nelson, T. O. and Dunlosky, J.: When peoples judgements of learning ( JOLs) are extremely accurate at predicting subsequent recall: the `delayed-JOL effect’. Psychological Science, 2: 267± 270, 1991. 35. Lichtenstein, D., Fischhoff, B. and Phillips, L. D.: Calibration of probabilities: the state of the art of 1980. In: D. Kahneman, P. Slovic and A. Tversky (editors) Judgment under uncertainty: heuristics and biases (New York: Cambridge University Press), pp. 306± 334, 1982. 36. Nelson, T. O., Graf, A. and Dunlosky, J.: Effect of acute alcohol intoxication on recall and on judgments of learning during the acquisition of new learning. In: G. Mazzoni and T. O. Nelson (editors) Metacognition and cognitive neuropsychology: monitoring and control processes (Mahwah, NJ: Lawrence Erlbaum), pp. 161± 180, 1998. 37. Costermans, J., Lories, G. and Ansay, C.: Confidence level and feeling of knowing in question answering: the weight of inferential processes. Journal of Experimental Psychology: Learning, Memory and Cognition, 18: 142± 150, 1992. 38. Fischhoff, B., Slovic, P. and Lichtenstein, S.: Knowing with certainty: the appropriateness of extreme confidence. Journal of Experimental Psychology: Human Perception and Performance, 3: 522± 564, 1997. 39. Lichtenstein, S., Fischhoff, B. and Phillips, L. D.: Calibration of probabilities: the state of the art. In: H. Jugerman and G. Dezeenn (editors) Decision making and change in human affairs (Amsterdam: D. Reidel), 1977. 40. Zakay, D. and Glicksohn, J.: Overconfidence in a multiple-choice test and its relationship to achievement. The Psychological Record, 42: 519± 524, 1992. 41. Schraw, G. and Roedel, T. D.: Test difficulty and judgment bias. Memory and Cognition, 22: 63± 69, 1994. 42. Dunlosky, J. and Nelson, T. O.: Does the sensitivity of judgments of learning ( JOLs) to the effects of various study activities depends on when the JOLs occur? Journal of Memory and Language, 33: 545± 565, 1994. 43. American Congress of Rehabilitation Medicine, Mild Traumatic Brain Injury Committee: Definition of mild traumatic brain injury. Journal of Head Trauma Rehabilitation, 8: 86± 87, 1993. 44. Levin, H. S., Hugh, W. M., Goethe, K. E. et al.: The neurobehavioural rating scale: assessment of the behavioural sequelae of head injury by the clinician. Journal of Neurology, Neurosurgery, and Psychiatry, 50: 183± 193, 1987. 45. Barona, A., Reynolds, C. R. and Chastain, R.: A demographically based index of premorbid intelligence for the WAIS-R. Journal of Consulting and Clinical Psychology, 52: 885± 887, 1984. 46. Kertesz, A.: Western Aphasia Battery (San Antonio, TX: Psychological Corporation), 1982. 47. Ammons, R. B. and Ammons, C. H.: Use and evaluation of the Quick Test. Psychological Reports, 45: 943± 946, 1979. 48. Wechsler, D.: Wechsler Memory Scale-Revised (San Antonio, TX: Psychological Corporation), 1987. 49. Reitan, R. M. and Wolfson, D.: The Halstad-Reitan Neuropsychological Test Battery: theory and clinical interpretation (Tucson, AZ: Neuropsychology Press), 1985. 50. Paivio, A., Yuille, J. C. and Madigan, S. A.: Concreteness, imagery and meaningfulness values for 925 nouns. Journal of Experimental Psychology Monograph Supplement, 76: 1± 25, 1968. 51. Gonzalez, R. and Nelson, T. O.: Measuring ordinal association in situations that contain tied scores. Psychological Bulletin, 119, 159± 165, 1996. 52. Weaver, C. A. and Bryant, D. S.: Monitoring of comprehension: the role of text difficulty in metamemory for narrative and expository text. Memory and Cognition, 23: 12± 22, 1995. 53. Nelson, T.: A comparison of current measures of the accuracy of feeling-of-knowing. Psychological Bulletin, 95: 109± 133, 1984.



487

54. Dunlosky, J. and Nelson, T. O.: Importance of the kind of cue for judgments of learning ( JOL) and the delayed JOL effect. Memory and Cognition (Special Issue: Memory and cognition applied), 20: 374± 380, 1992. 55. Koriat, A. S., Lichtenstein, S. and Fischhoff, B.: Reasons for confidence. Journal of Experimental Psychology: Human Learning and Memory, 6: 107± 118, 1980. 56. McGlynn, S. M. and Schacter, D. L.: Unawareness of deficits in neuropsychological syndromes. Journal of Clinical and Experimental Neuropsychology, 11: 143± 205, 1989. 57. Thiede, K. W. and Dunlosky, J.: Delaying students’ metacognitive monitoring improves their accuracy in predicting their recognition performance. Journal of Education Psychology, 86: 290± 302, 1994. 58. Kennedy, M. R. T.: Metamemory monitoring of adult survivors of traumatic brain injury. Dissertation Abstracts International, 57: 4338, 1997. 59. Nawrocki, M. D.: Metamemory for narratives following traumatic brain injury. Master’s thesis, unpublished, University of Minnesota, Minneapolis, MN, 2000. 60. Nelson, T. O., Dunlosky, J., Graf, A. et al.: Utilization of metacognitive judgments in the allocation of study during multitrial learning. Psychological Science, 5: 207± 213, 1994. 61. Dunlosky, J. and Hertzog, C.: Training programs to improve learning in later adulthood: helping older adults educate themselves. In: D. E. Hacker, J. Dunlosky and A. C. Graesser (editors) Metacognition in educational theory and practice (Mahweh, NJ: Lawrence Erlbaum Associates), pp. 249± 275, 1998.