In everyday life, experts are generally viewed as those best suited to helping novices, ... found in all textbooks on technical writing, namely to avoid jargon.
The Curse of Expertise: Exploring the Problem of Anticipating Readers’ Needs Leo Lentz Menno de Jong Abstract This article explores the problems faced by professional writers in attempting to anticipate the needs of their readers. On the basis of two studies (a study of the literature on expert/novice communication and an exploratory study requiring experts to assess and explain the likelihood and severity of various problems for readers), the authors distinguish cognitive shortcuts that may hinder experts in their attempts to focus on readers’ needs. Keywords Expert-focused evaluation, usability, expertise, professional writing
Introduction One of the key problems for professional writers in compiling a document is how to anticipate the difficulties that readers may experience with the text in question. Formative evaluation and usability research invariably show that documents created by professionals often contain serious and sometimes fatal problems for readers, despite all the care with which these texts may have been produced. Not one of the many document evaluations that we have conducted or supervised found the document in question to be free of problems or to contain only minor flaws. This paper therefore addresses the issue of why it is so difficult for experts to anticipate their readers’ needs. In many respects, professional writing may be seen as a form of expert/novice communication. The professional writer is the expert with the task of instructing readers – for instance, on how to operate a certain device. Target readers include both novices attempting to become more skillful or lay people who just need to know how to operate the device once. Outside the field of technical communication, and most notably in the psychological literature, interesting research into expert/novice communication may help to explain the challenges for professional writers in anticipating reader problems. In this paper, we present an overview of the literature on expert/novice communication and discuss various cognitive shortcuts used by experts that may help to explain the complexity of their task. We then elaborate on these on the basis of a study in which expert writers were asked to assess the importance of problems for readers. This study indicates an even wider range of cognitive shortcuts that hinder professional writers in anticipating their readers’ needs. Experts overestimate the knowledge of others In everyday life, experts are generally viewed as those best suited to helping novices, and it is their task to transfer knowledge across generations. However, in practice, despite or even due to their surplus of knowledge and experience, they often appear to
1
have difficulties in predicting the problems that novices will experience when performing a new task. A literature search on the topic of expert/novice communication brought to light several studies that demonstrate and explain how difficult it is for experts to predict how novices will behave. Hinds [1], a leading researcher in the field of expert/novice communication, introduced the term curse of expertise, which perfectly expresses the concept that expertise is less of an advantage than an obstacle for experts required to predict novice behavior. This was borne out by an experiment in which expert users of cell phones (i.e., salespeople), intermediate users of cell phones (i.e., users with some experience with the cell phones) and novices were asked to predict how long it would take novices to perform a set of tasks using the phone. The novices actually needed 31.5 minutes to perform these tasks. The experts were least accurate in their predictions, predicting an average of 12.9 minutes. The novices themselves were also inaccurate, underestimating the time they would need (15.7 minutes), probably due to their lack of experience with the tasks in question. The most accurate predictors were the intermediate users (20.9 minutes). According to Bühler, Griffin and Ross [2], research generally shows that people tend to underestimate how long it takes to complete a task. Nevertheless, the difference in predictions between the intermediate users with some experience and the genuine experts with far more experience was both substantial and significant: clearly, the experts’ higher level of expertise prevented them from making more accurate predictions. Given that estimating task completion times is generally recognized to be difficult, further insights may be gained by considering the prediction of other performance dimensions. Loewenstein, Moore and Weber [3] conducted an experiment in which participants were asked to predict whether people were able to solve a “spot the difference” exercise in which they needed to differentiate between two almost identical images. The experiment focused on task success rather than task completion time as the variable to be predicted. Loewenstein, Moore and Weber manipulated expertise as an independent variable by giving some participants the solution to the exercise before requiring them to make a prediction. The findings of this experiment were consistent with those of Hinds: participants who already knew the difference between the pictures were less capable of accurately predicting the success rate per image than those who had not been informed of the solution beforehand. The best predictors appeared to be the group of uninformed participants who had themselves not figured out the difference between the two images. Another factor that may affect how novices perform is the extent to which they understand professional concepts. This is reflected in the key recommendation found in all textbooks on technical writing, namely to avoid jargon. This argument is supported by the work of Bromme, Jucks and Rambow in their project on expert/novice communication. Rambow [4] demonstrated that experts in architecture overestimate how well architectural concepts are understood, particularly those concepts expressed in language halfway between everyday language and professional jargon. Jucks [5] elaborated on this finding in a complex study in which two different fragments of a software manual (Texts 1 and 2) were presented to two similar groups of experts and two similar groups of lay people. Text 1 was full of jargon, and Text 2 was a revised version in which the professional concepts had been expressed using more common terms. Both experts and lay readers were asked to read the texts and rate their comprehensibility. Whereas the experts perceived little difference between the two texts, lay readers found Text 2 much easier to understand. Jucks therefore
2
concluded that the experts were apparently unaware of the effect of professional jargon on readers’ comprehension. Bromme, Rambow and Nückles [6] tested the hypothesis that experts overestimate the knowledge of lay people in a study in which IT experts were presented with two lists, one featuring specialist concepts and the other featuring general concepts. The experts were asked to estimate the percentage of lay people who would be able to define the various concepts correctly. At the same time, a group of laypersons were asked to provide definitions for the same two lists of concepts. This study concluded that experts overestimate the knowledge of laypersons, although this effect was less clear for specialist concepts than for general knowledge, which seems to contradict Jucks’ findings. Lentz and De Jong [7] also conducted research into expert predictions within the field of technical writing. They presented two groups of experts with a brochure written for adolescents on the dangers of drinking alcohol. One group comprised 20 professional technical writers (who were therefore experts on writing and communication), and the other consisted of 35 alcohol consultants (who were therefore experts on the subject matter of the text). These experts were asked to predict the problems that readers would experience when reading the brochure. Prior to this, the text had been evaluated by a sample of 30 readers from the target audience. Both groups of experts achieved very low prediction rates: on average, they predicted less than 15% of the most serious problems encountered by readers, with an astonishing number of “unique detections” (i.e., problems detected once only). In the approximately 1,300-word brochure, the 56 experts together detected more than 400 separate problems – an average of three problems per sentence. For the majority of these problems, there was no agreement between even two experts. This lack of consensus suggests that experts may strongly rely on their individual knowledge base when trying to imagine the problems that others may experience. De Jong and Lentz [8] obtained similar findings in a study comparing experts’ predictions for a document about rent subsidies aimed at the general public. Five possible explanations In the previous section, we presented a body of empirical research showing that experts have difficulties in predicting task completion times, task success, comprehension of concepts, and specific reader problems for novices. But what is it that hinders them in detecting the obstacles faced by novices? In this section, we elaborate on a theoretical framework for expert/novice communication in order to identify specific factors that affect this curse of expertise. Based on the work of Hinds [1] and Nickerson [9], we present five explanations, framed in terms of “cognitive heuristics” or “cognitive shortcuts.” The term “cognitive heuristics” is ambiguous in the context of technical communication, as it refers to both simple decision rules used to formulate judgments (cf. Tversky and Kahneman [10]) and sets of guidelines, principles, criteria, tips and tricks that are available to support or evaluate design decisions (cf. De Jong and Van der Geest [11]). We have therefore chosen to use the second term, “cognitive shortcut,” which unequivocally refers to the first of these two meanings, signifying a simple decision rule that helps the expert to formulate a judgment. The first explanation presented by Hinds is the availability shortcut, which is closely related to Nickerson’s observation that people make predictions on the basis
3
of the information readily available in their memory. According to this explanation, cell phone experts will remember series of routine, unproblematic experiences with cell phones, and this is therefore the knowledge that they will activate when thinking about novices’ task performance. Consequently, they will overestimate that performance. Two hypotheses were formulated on the basis of this availability shortcut: (1) intermediate users will make better predictions than high-level experts, because they have more available knowledge on problems with cell phone usage; and (2) experts’ predictions will increase in accuracy when they are asked to recall their own experiences with the device (including as novices). Of these two hypotheses, only the first is supported by Hinds’ data: intermediate users did in fact make better predictions. Hinds’ second explanation is the anchoring shortcut, whereby experts fail to make use of knowledge about novice behavior that may be available to them. According to this explanation, experts anchor on their own performance and fail to adequately take into account differences between their skills and those of others. As Nickerson puts it, “People spontaneously take themselves as the reference point, or prototypical person” [9, p.745]. This explanation is supported by the above-mentioned study by Loewenstein, Moore and Weber [3], in which they provided half of all participants with the solutions to a set of exercises and found that this prior knowledge hindered them in predicting the number of correct solutions by others. This finding can be explained by the fact that these participants had to adjust their own performance to a greater extent than participants without any prior knowledge. Nickerson [9] reports on a study in which students were asked to estimate how many of them would give the right answer to a set of questions. The students gave more favorable estimates for questions to which they thought they knew the answers than for those to which they did not. Hinds tested the anchoring hypothesis by providing one group of experts with a list of problems experienced by novices using cell phones, expecting these experts to make better predictions than another group of experts, who did not have the aid of this list. Her conclusion was that experts resisted this debiasing method, which means that they did not accept the invitation to reset the anchor, so to speak. The third explanation put forward by Hinds is oversimplification of the task. The more expertise people acquire with respect to a certain task, the more carrying out that task will become automatic to them and the more they will lose sight of the underlying task components and performance details. This is related to the architecture of human cognition, in which declarative knowledge becomes procedural knowledge [12]. According to this explanation, it is not the availability of knowledge that hinders experts in predicting novice behavior, but the different structure of expert knowledge. Hinds tested this theory by asking both experts and intermediate users to describe the various subtasks required to carry out an overall task. She concluded that experts distinguished no fewer steps in the process than intermediate users, which therefore ruled out the oversimplification hypothesis. In reviewing these first three explanations – the availability shortcut, the anchoring shortcut and oversimplification – Hinds concludes that “the availability bias is the key contributor to experts’ relative inaccuracy in estimating novice performance times” [1, p.218]. The fourth and fifth explanations are provided by Nickerson [9]. The first of these is the illusion of simplicity, whereby an expert may unduly judge something to be simple only because it is familiar. This is a familiar concept in research into text comprehension, which shows that experts in a certain field mistake their familiarity
4
with the subject matter for textual comprehensibility. Similar findings have been presented by Hayes [13], who terms this phenomenon the knowledge effect. The effect is explained by the fact that people may assume that they have known something for some time, whereas they only recently acquired this knowledge. It appears to be difficult for people to imagine a state of not knowing something once they know it. This not only applies to facts, but also to opinions and attitudes: once people have changed their mind, they tend to forget the opinions they previously held. The second explanation presented by Nickerson is the false consensus effect, which refers to people’s tendency to consider themselves more representative of others than they really are. This effect has been demonstrated for beliefs, attitudes and actions. People tend to overestimate general consensus on opinions they hold themselves and to underestimate consensus on opposing opinions. This tendency appears to be stronger for opinions that matter most to the people in question. This shortcut might explain in particular why it is hard for experts to predict the problems that readers encounter with persuasive documents. In summary, we found five possible explanations or shortcuts for the low prediction rates achieved by experts when asked to think about the problems that readers may experience: (1) The availability shortcut prevents them from recalling their own history as a novice; (2) The anchoring shortcut prevents them from taking the novice’s perspective; (3) Oversimplification prevents them from having a clear view of the subtasks required by the process; (4) The illusion of simplicity prevents them from differentiating between their familiarity with a topic and the comprehensibility of the text; (5) The false consensus effect encourages them to view themselves as representative of others. As mentioned above, based on an experiment to find out which of the first three shortcuts best explains the difficulties experts have in predicting novice behavior, Hinds [1] concluded that it was the availability shortcut that hindered participants. In our view, there are serious reasons to doubt whether we should be in fact be looking for just one shortcut as an explanation. The studies described above focused on various independent variables, such as time or success rates, and they were conducted using very different methods, such as “spot the difference” exercises, phone handling and software usage. Different shortcuts might be responsible for different types of prediction task. Moreover, for one and the same task, as in our study referred to above [7], we found not only low prediction rates, but also a wide diversity in experts’ predictions: rather than all pointing in the wrong direction, their predictions all pointed in different directions. This lack of agreement may indicate that different shortcuts affect different experts. Exploratory study: Technical writers’ reasoning about reader problems Given that experts’ predictions vary so widely, we decided to conduct an exploratory study focusing on a specific aspect of the prediction process. We simplified the prediction task by offering participants a set of detected problems and concentrating on the assessment of these problems. In this section, we present the findings of this
5
study, exploring the relationship between the cognitive shortcuts discussed above and the assessment of the problems detected. In our view, these judgments are at the core of the prediction process; this can be explained using a model of the revision process provided by Hayes [14]. According to this model, the revision of a text is guided by three fundamental processes: (1) Text interpretation (reading and comprehending the text); (2) Reflection (detecting problems, making decisions and solving problems); (3) Text production. These processes are controlled by the writer’s task schema, and they rely on both the writer’s working memory and long-term memory. Predicting reader problems involves the first two fundamental processes. When asked to predict reader problems, experts need to first read and comprehend the text and then reflect on it. During this reflection, the experts try to detect problems they believe readers may have with the text and assess how serious these problems will be. This process is guided by decision rules, along the lines of the cognitive shortcuts referred to by Hinds [1]. One such decision rule might be as follows: “If I experience a problem while reading, many other readers will also experience that problem.” This rule is a reflection of the false consensus effect discussed above. The same shortcut may lead to the opposite decision rule: “If I do not experience any problem, neither will other readers.” An investigation of the decision rules experts apply while reflecting on problem detections might shed more light on the cognitive shortcuts that experts activate when asked to predict reader problems. Design In this exploratory study, we asked 18 communication professionals to reflect on 10 reader problems collected in our usability study on a brochure about the dangers of alcohol [7]. We asked them to assess for each of these problems (a) how likely it was that the problem would be experienced by readers, and (b) how severely it would impair the effectiveness of the document. The study was designed using the Delphi method as described by Linstone and Turoff [15]. This method is specifically designed for research into decision processes and to investigate whether it is possible to reach consensus on the topics investigated. The study was divided into three stages: (1) First, the experts rated the likelihood and impact of each problem detected on a five-point scale, giving an argument for each rating. (2) Next, the experts received feedback on the arguments given and were then asked to provide new ratings and arguments. We termed this stage the enrichment stage, as the experts’ assessments were enriched by other arguments they may not have previously considered. (3) Finally, the experts received feedback on the mean scores of all ratings and a selection of the main arguments given, after which they rated the problems once again and provided arguments for their rating. We termed this stage the convergence stage, as all experts were informed about the main tendencies in the scores of the entire group of experts.
6
Each of the 18 experts was sent a mail package consisting of a fragment of the alcohol brochure, the set of 10 problems detected, instructions to rate every detection in terms of both likelihood and impact, and a request to provide arguments for each rating. For each of the 10 problems, the paragraph in which the problem was found was indicated, a description of the paragraph’s function in the brochure was given, the problem itself was described, the two scales indicating the likelihood and impact of each problem were given, and space to write down arguments was provided. Like Nielsen, we made a distinction between likelihood and impact in order to enable us to differentiate between the assessment of the probability of a problem occurring and the assessment of the negative impact of that problem on the document’s effectiveness [16]. Findings Did the feedback that we provided on arguments and ratings in the three stages lead to consensus between experts? Our criterion for consensus was that 80% of the 18 experts had to agree on both the likelihood and severity of a problem detected, rated on a five-point scale. Only one of the 10 problems met this criterion, and only in Stage 1. We therefore conclude that the process did not lead to consensus between experts. Was there an increase in consensus between Stages 1 and 3? We analyzed the differences in standard deviation of mean scores for all 10 problems (separating likelihood and impact), as a decrease in standard deviation may be seen as an increase in agreement among participants. Table 1 shows the results of this analysis. Stage 1 Ö 2 Stage 2 Ö 3
SD decrease 12 15
SD increase 8 5
Mean SD decrease 1.51-1.48 = 0.03 1.48-1.27 = 0.21
Significance p = 0.12 p < 0.05
Table 1. Effect of Stages 2 and 3 on mean standard deviation (SD) of likelihood and severity scores for 10 problems detected (N=18) Only a small decrease in standard deviation (which appeared to be significant) was found between Stages 2 and 3 (1.48 – 1.27). We therefore conclude that the overview of all arguments as given in Stage 2 did not help at all to achieve consensus. However, the information about scoring tendencies did have some harmonizing influence on the experts’ ratings. Nevertheless, consensus among experts remained low. What caused this diversity in judgments? And how did the experts reason about the likelihood and impact of the problems presented to them? An analysis of their arguments helped us to gain deeper insight into the cognitive shortcuts that experts use when thinking about reader problems. In total, the experts provided 385 arguments for statements about the likelihood and severity of 10 problems detected; this represents a mean score of approximately 19 arguments for likelihood and 19 arguments for severity per problem. We analyzed these verbal statements by means of a process of open coding, on the basis of which we identified eight categories of argument. Once these categories had been defined, both authors
7
coded all arguments independently by category. These scores were then analyzed for reliability, which resulted in a fairly low Cohen’s Kappa (0.49). As our attempts to increase reliability by redefining categories were unsuccessful, no further statistical analysis was performed. We will restrict ourselves here to presenting the various categories and giving examples of experts’ arguments to illustrate shortcuts discovered in the data. These examples will also demonstrate why a clear-cut categorization is difficult to achieve. The categories are presented in Table 2. Category Text-focused
Definition Experts anchor on the text as something autonomous. Pedantic Experts anchor on their expertise in communication or linguistics. Blame the reader Experts blame the reader for failing to understand. Correction first Experts anchor on the role of an editor. Elsewhere in the Experts dissociate the fragment containing text a problem detected from the rest of the text as an argument for claiming that “no problem exists.” Praise the reader Experts dissociate readers who experience the problem from others who are said not to have any problems with the fragment. Interpretational Experts focus on readers’ interpretations. Functional Experts focus on the function of a fragment.
Shortcut Anchoring Anchoring Anchoring Anchoring Dissociation
Dissociation Likelihood Severity
Table 2. Eight categories of argument for the likelihood or severity of problems detected Four of these categories appear to be related to the anchoring shortcut discussed above, which prevents experts from taking the reader’s perspective. The other four categories indicate decision rules that do not seem related to any of the shortcuts found in the literature. In the next section, we discuss each of these categories in order to gain deeper insights into the curse of expertise. Cognitive shortcuts in reasoning about problem detections One of the most frequent categories of argument was the text-focused category, whereby experts anchored on the text not as a means of communication, but rather as something autonomous. For example, one of the problems detected concerned a statement in the brochure that black coffee and cold showers do not help to accelerate the breakdown of alcohol in the bloodstream. The function of this fragment was to refute the popular misconception that it is possible to sober up quickly by drinking black coffee or taking a cold shower. The reader needs to be persuaded that alcohol breaks down slowly in the body and that there is no way of accelerating this process. The problem detected here was summed up as follows: “They say these things don’t work, but I think they do. After a cup of coffee or a shower, you always feel better!”
8
Below are some of the reasons that were advanced for giving the problem a low likelihood rating: (1) “It is not the text that creates this problem; it is the readers’ own experience.” (2) “The text presents a scientific fact in a very clear way. A technical writer should not stretch the truth. A text must be comprehensible, attractive and inviting; for the rest, it should just tell the truth.” (3) “This text is perfectly clear.” All three of these arguments refer to the text as something autonomous, and contain a hidden assumption about the allocation of responsibilities between the reader, the text and the writer. The writer’s role is simply to provide clear information about scientific facts; what happens during the reading process is not the writer’s responsibility. The experts clearly commit themselves to the text as an autonomous product, and are willing to defend it against problems detected by readers. In all three cases, the experts refuse to adopt the readers’ perspective. Experts using this kind of reasoning must be expected to have difficulty in anticipating readers’ needs, as they focus on the text itself rather than on the effects it should have on its readers’ beliefs and attitudes. A second category of arguments indicated use of the same anchoring shortcut, but they were linked to the experts’ expertise in communication and linguistics rather than to any loyalty to the text. We termed these pedantic arguments. They focus on stylistic issues popular among the literary in-crowd, or try to come across as original, sharp-witted and knowledgeable. Rather than concentrating on the readers’ efforts to make sense of a fragment, they instead seize the opportunity to demonstrate their own expertise. This can be illustrated by looking at the arguments regarding another problem detected, concerning the heading of a fragment on two popular misconceptions about alcohol. The heading was “Twice the biggest nonsense.” Some readers had indicated that they did not understand this heading, and others considered it unclear. The following are examples of pedantic arguments given: (4) (5) (6) (7)
“You cannot multiply nonsense.” “Only one thing can be the biggest; this will therefore confuse readers.” “If you multiply two negatives, you get a positive.” “This sentence is incomplete, as it has no verb: this makes it difficult to understand.” (8) “Headings should always invite readers to continue reading.”
These arguments all show that experts reason in the same way as teachers evaluating a student’s text, mixing up mathematics, grammar and logic with the interpretation processes of young readers. This type of argument is slightly different to the textfocused argument: the experts refer not so much to their loyalty to text and author, but rather to their knowledge about textual features and communication processes in general. The third category of argument, termed blame the reader, also relates to the anchoring shortcut, and is illustrated by the following three statements: (9) (10) (11) (12)
“It is your own responsibility if you refuse to take the text seriously.” “I’m sorry, but the reader is talking nonsense here. “If the reader thinks they know better, they should do it themselves.” “If readers say such absurd things, authors won’t dare to write another word.”
9
In these arguments, the experts also anchor on the text and show loyalty to the author, but they go a step further by turning their back on the reader and increasing the distance between reader and writer/expert. The writer/expert does not want to take any responsibility for such apparently unwilling readers, and therefore more or less dismisses them. These experts seem to believe that if readers react in this way, one might as well stop worrying about trying to make contact with them at all. The interesting thing here is that the experts apparently did not feel the need to investigate the cause of the problem at hand. The readers are blamed for their reaction and no further analysis is deemed necessary. The fourth category of argument related to the anchoring shortcut is termed correction first, and is illustrated by the following statements: (13) “I can’t think of a way to say more clearly that you cannot speed up the process of alcohol breakdown.” (14) “I would prefer to offer some more alternatives in this fragment.” (15) “I would have written that a glass of beer contains less alcohol than a glass of whisky.” This phenomenon may be described as the tendency to detect only those problems that can be solved. If the expert cannot see how to solve the problem, then it simply does not exist. Many examples of this shortcut were found in the arguments given by experts. These arguments took two basic forms: (a) This is not a serious problem, because I do not know how to revise it. (b) This is a serious problem: the text should be revised as follows. Experts who advance arguments of this type are also anchoring on the author and the text, but they are specifically taking a revision perspective. Hacker et al. [17] called this the correction first hypothesis, whereby people tend to restrict themselves to those problems that they know how to solve. The four categories of argument discussed so far illustrate different aspects of the anchoring shortcut. What they have in common is that they reveal an egocentric tendency and neglect the reader’s perspective. In all these arguments, the anchor is located within the expert’s ego, but its exact location varies. In the text-focused arguments, the experts anchor on their own interpretation of the text, which is presented as an absolute statement (e.g., “This text is perfectly clear.”). This category of argument reflects a view of the text as something autonomous, and the arguments are egocentric in that they reflect an overgeneralization of the expert’s individual interpretation. In the pedantic arguments, the experts anchor on their expertise in linguistics and communication. When they blame the reader, experts identify with the author and turn their back on the reader. Finally, when they use correction first arguments, they anchor on the role of an editor. Both these author and editor roles reflect the expert’s personal position. In practice, it was often difficult to place the experts’ statements in just one of these categories. For example, a pedantic argument was often linked to a correction first argument, while a text-focused statement might be combined with a remark blaming the reader. The fifth and sixth categories of argument (praise the reader and elsewhere in the text) reflect a tendency to downgrade the importance of problem detections by means of dissociation. Praise the reader arguments dissociate those readers who
10
detected a problem from all the other readers who did not detect that problem. By disparaging the former and praising the latter, they emphasize their optimism about the effectiveness of the document. The following statement is an example of this type of argument: (16) “Most readers will know that alcohol does not suddenly disappear from the body.” Elsewhere in the text arguments dissociate the fragment containing the problem detected from the rest of the document. In these arguments, the strength of the rest of the text is emphasized in order to downgrade the importance of the problem. Experts using this category of argument show a high degree of optimism about the readers’ willingness and ability to resolve any misunderstanding. The following statement illustrates this type of argument: (17) “Everything is made clear in the following sentences of the text.” In the examples discussed above, experts utilize all types of reasoning techniques as decision rules for assessing the importance of problem detections in a document. In our view, all these rules reflect an aspect of the curse of expertise. They did not help experts to gain a clear view of the likelihood and severity of reader problems. The experts failed to anchor on complex reading processes, but instead anchored on the writer’s or editor’s perspective and on their own textual expertise. The diversity of these decision rules may explain why experts do not agree when detecting reader problems or assessing their severity. This raises the following question: What type of decision rule would help to assess problem detections adequately? In answering this question, we will discuss two adequate decision rules, one for assessing the likelihood and the other for assessing the severity of reader problems. These are the interpretational and functional categories respectively. In order to assess the likelihood of problem detection, experts require an interpretational decision rule. This requires that a statement in the text be analyzed in relation to the reader’s interpretation of that statement, as exemplified in Box 1. “Remember this. Nothing, absolutely nothing can accelerate the breakdown of alcohol in your body. All that talk about fresh air, black coffee, cold showers, vitamins or medicines that sober you up – don’t believe it.” Reader’s comment: “They say these things don’t work, but I think they do. After a cup of coffee or a shower, you always feel better!” Box 1. Fragment about alcohol breakdown and problem detection The crux of the proposition here is that “nothing can accelerate the breakdown of alcohol in your body.” The problem detection, however, presents a counterargument: namely, that you do feel better after a cup of coffee or a shower. The exact relationship between the proposition and counterargument is rooted in both the statement about the breakdown of alcohol on the one hand and the reader’s experience of feeling better on the other hand. The reader infers that feeling better is an indication of alcohol breakdown. In assessing likelihood, we therefore need to reconstruct the following process of interpretation: 11
(a) The topic of sobering up activates readers’ memories of experiences such as drinking coffee and taking a shower after drinking too much alcohol. (b) Feeling better is interpreted as an indication of accelerated alcohol breakdown. (c) The experience of feeling better after drinking coffee therefore conflicts with the statement about the impossibility of accelerating the breakdown of alcohol. (d) In the event of a conflict between the statement in the text and experience, a young reader will prefer to stick to the belief based on experience. We consider this reconstructed interpretation process to be highly plausible. The crucial relationship between the specific experience of feeling better and the abstract concept of alcohol breakdown is obvious. Such a clear diagnosis of the problem is often the key to successfully revising the text – in this case, conceding that, although you may feel better after a shower or coffee, the blood alcohol level remains the same. In short, the decision rule for assessing the likelihood of problem detections requires the conscientious reconstruction of the reader’s interpretation process. For this purpose, the following questions need to be answered: (a) What is the relationship between the problem detected and the text fragment? (b) What interpretation of the text fragment might give rise to this problem? (c) What kind of knowledge about the world in general, the reader’s experiences, or the specific communication situation will readers relate to this fragment? (d) How plausible is this interpretation of the text fragment? In assessing the severity of a problem detected, we need another, very different decision rule: one that focuses not on the interpretation process of the reader, but instead on the goals and intentions of the author. The main issue here concerns the negative effect a problem might have, even if it seems highly unlikely to occur. This requires an analysis of the function of the fragment in relation to the effect as presented in the problem detected. Let us illustrate this by analyzing the problem discussed above. In order to specify the function of the text fragment, we rely on the theory of functional analysis as presented by Lentz and Pander Maat [18]. The function of this fragment can be defined as follows: Young adults who tend to drink alcohol frequently realize that alcohol breaks down in the blood rather slowly (1 hour per glass of alcohol) and that this process cannot be accelerated by cold showers or black coffee. This should help them to know when to stop drinking alcohol, and thus lead to fewer injuries and accidents. The way in which the reader formulates the problem detection (i.e., the reader’s comment in Box 1) shows that the effect of the text fragment is to trigger resistance to the proposition put forward in the text fragment, namely that the breakdown process cannot be accelerated by black coffee and cold showers. This resistance appears to be strong, as it is based on experiences that the reader brings to the fore. This impairs the effectiveness of this fragment, and as long as readers cling to this belief, the text will not achieve its primary objective. There is only one possible conclusion: the problem detected by the reader is a serious one.
12
To summarize, the decision rule for assessing the severity of a problem detection requires an analysis of the effect of the text fragment with regard to the function of this fragment. For this purpose, the following questions need to be answered: (a) (b) (c) (d)
What is the function of this fragment of the document? What is the effect of the interpretation as reflected in the problem detection? Does this effect impair the function of the fragment in any way? How serious is this impairment?
Conclusion Experts in professional communication experience the curse of expertise in the same way as experts in other professions, as shown by their low prediction rates for reader problems and their lack of consensus in assessing the detection of problems. However, it is not the availability shortcut that prevents them from anticipating the readers’ perspective. In fact, the most frequently used decision rules were related to the anchoring shortcut. Further analysis of these decision rules showed that experts anchor on different aspects of expertise: (1) their own reading experiences; (2) their expertise in communication processes; (3) their loyalty to the author and text, and (4) an editor’s perspective with a focus on revision of the text. Two other decision rules could not be related to the shortcuts found in the literature. These were based on (5) dissociation between trouble-seeking readers and trouble-free readers, and (6) dissociation of the problematic fragment from other parts of the document. Two other shortcuts presented by Nickerson [9], the illusion of simplicity and the false consensus effect, are both related to the anchoring decision rules found in our data. Experts who present their own comprehension of a fragment as an argument for the improbability of a problem detected anchor on their own reading experience and demonstrate both the illusion of simplicity and the false consensus shortcut. Two of the shortcuts found in the literature seem to be irrelevant in the context of communication experts and reader problems. The availability shortcut, which prevents experts from recalling their own history as a novice, is irrelevant, because in the context of professional communication most experts do not have any experience with the specific text as a novice. And the oversimplification shortcut, which prevents them from having a clear view of the subtasks needed for the process, is often irrelevant because the main function of many documents (such as the text about alcohol) is not task-oriented. The other decision rules hindered experts in focusing on the needs of the reader. However, it seems appropriate to also consider decision rules that do help to achieve the right focus. We presented one decision rule for assessing the likelihood of problem detections and another for assessing the severity of problem detections. Further research is required to find out whether the use of such rules in training experts would improve their prediction of reader problems and increase consensus in their assessments of these problems.
13
References [1] [2] [3]
[4 [5]
[6]
[7] [8] [9] [10] [11] [12] [13] [14] [15] [16]
P.J. Hinds, “The Curse of Expertise: The Effects of Expertise and Debiasing Methods on Predictions of Novice Performance,” Journal of Experimental Psychology: Applied, vol. 5, pp. 205-221, 1999. R. Buehler, D. Griffin and M. Ross, “Exploring the ‘Planning Fallacy’: Why People Underestimate Their Task Completion Times,” Journal of Personality and Social Psychology, vol. 67, pp. 366-381, 1992. G. Loewenstein, D.A. Moore and R.A. Weber, “Paying $1 to Lose $2: Misperceptions of the Value of Information in Predicting the Performance of Others,” Carnegie Mellon Behavioral Decision Research working paper no. 301, 2002. http://ssrn.com/abstract=323221. R. Rambow, Experten-Laien-Kommunikation in der Architectur (Expert Lay Communication in Architecture). Münster: Waxmann, 2000. R. Jucks (2001), Was verstehen Laien? Die Verständlichtkeit von Fachtexten aus der Sicht von Computer-Experten (What Do Laypersons Understand? The Comprehensibility of Professional Documents from the Perspective of Computer Experts). Münster: Waxmann, 2001. R. Bromme, R. Rambow and M. Nückles, “Expertise and Estimating What Other People Know: The Influence of Professional Experience and Type of Knowledge,” Journal of Experimental Psychology: Applied, vol.7, pp. 317-330, 2001. L. Lentz and M. de Jong, “The Evaluation of Text Quality: Expert-Focused and Reader-Focused Methods Compared,” IEEE Transactions on Professional Communication, vol. 40, pp.224-233, 1997. M. de Jong and L. Lentz, “Expert Judgments versus Reader Feedback: A Comparison of Text Evaluation Techniques,” Journal of Technical Writing and Communication, vol. 26, 4, pp. 507-520, 1996. R.S. Nickerson, “How We Know – And Sometimes Misjudge – What Others Know: Imputing One’s Own Knowledge to Others,” Psychological Bulletin, vol. 125, pp. 737-759, 1999. A. Tversky and D. Kahnemann, “Judgment under Uncertainty: Heuristics and Biases,” Science, vol. 185, 1124-1131, 1974. M. de Jong and T. van der Geest, “Characterizing Web Heuristics,” Technical Communication, vol. 47, pp. 327-340, 2000. J.R. Anderson, The Architecture of Cognition. Cambridge, MA: Harvard University Press, 1983. J.R. Hayes, “Writing Research: The Analysis of a Very Complex Task,” in Complex Information Processing: The impact of Herbert A. Simon, D. Klahr and K. Kotovsky, Eds. Hillsdale: Lawrence Erlbaum, 1989, 209-234. J.R. Hayes. “What Triggers Revision?” in Revision: Cognitive and Instructional Processes, L. Allal, L. Chanquoy and P. Largy, Eds. Dordrecht: Kluwer, 2004, p. 9-20. H.A. Linstone and M. Turoff (Eds.), The Delphi Method: Techniques and Applications. Massachusetts: Addison Wesley, 1975. J. Nielsen, “Heuristic Evaluation,” in Usability Inspection Methods, J. Nielsen and R.L. Mack, Eds. New York: John Wiley, 1994, 25-62.
14
[17] D.J. Hacker, C. Plumb, E.C. Butterfield, D. Quathamer and E. Heineken, “Text Revision: Detection and Correction of Errors,” Journal of Educational Psychology, vol. 86, pp. 65-78, 1994. [18] L. Lentz and H. Pander Maat, “Functional Analysis for Document Design,” Technical Communication, vol. 51, pp. 387-398, 2004.
15
