Book reviews

10 downloads 102195 Views 527KB Size Report
(Studies in Language Testing, Volume 4. Series Editor: Michael Milanovic.) ISBN 0 521 56708 4. A minefield mapped. Caroline Clapham's 1996 volume in the ...
Book reviews

Clrpham, C. 199,6: The darcloprnent of IELTS: a sndy of the effect of background lonwledge on reading comprehension. Cambridge: Univemity of Cambridge Local Examinations Syndicate and Cambridge University Press. (Studies in Language Testing, Volume 4. Series Editor: Michael Milanovic.) ISBN 0 521 56708 4.

A minefield mapped Caroline Clapham's 1996 volume in the UCLES series examines a notion which has a lot of courmon sense; namely, to administer to second-language univenity strde,nts an entrance examination not only in the L2 but also in the specific acade,mic discipline of the studenl Science students should face reading passages about science. Business strdents should write essays on business topics. Humanities students should have oral inten'iews about topics

ftom the humanities. The,re is an inher€nt logical appeal to these claims. Things are not so simple. Clapham's study shows the reader that this seem-

ingly clear conce,pt is deceptively complex. tn effect, this simple terrain is a minefiel{ and her work has served to map the mines. The test at issue in Clapham's work is the Inte,rnational English t ansuage Testing Service (IELTS), which now has a yearly candidaturc of about 70,(n0 worldwide (Alderson, personal communication). As she e4plains in Chapt€r 4, the IELTS and ie precu$ors have long been targeted at university entrance for EFL sfirdents at English-medium schools. Towards the end of the 1970s, the managenrent of this exam carne to rcahzn, that their test should reflect the the,n-recent needs analyses of its examinees, and in so doing, that the test should also incorporate some form of English for Specific Purposes (ESP) te$ting. Clapham's work began not long thereafter, and the project reported in the prese,lrt volume incorporated (asentially) the second ESP test in TELTS htory; i.e., this book reports on the exam developed after the ELTS Validation Study was completed in 1988. In Chapter I, Clapham overviews the history of ESP testing with particular focus on the skill of her project: reading comp'rehe,nsion. The basic lesson she draws ftom this review foreshadows her own findings: First" language proficiency levels seem to play at least as iryortant a role as background knowledge in the comp'rehension of reading texts. Secondln background knowledge iself is not easily asse.sseq a strdent ufro is in Business Studies may well have prreviously worked in another discipline such as

Langnge lestirg

19!18 15 (2) 289-301

O265-5322(98)LT151XX @

1S

Arnold

Book reviews 289 Science, or rnfly have scientific inrcrese in his or her spare time. Thirdly, ahhough the above studies were in many ways inconcfusivg there did seem to be a tendency for science shdens to perform better ttan other students on science-based tesb, but to perform as well as humanities stud€nts on humanities-based ones. FinallS the level of specificity of the zubject-based texts prob ably varied widely in the diffemnt studies, hS this was not fulty taken into account in the studies (p. l0).

In addition to this review of ESP testing of reading, Clapham also provides a review both of first language and of second language reading. This lite,rature leads her to theoretical perspectives cnrcial to any l,anguage for Specific Purposes (LSP) testing. Schema theory, given aew analysis of text, Minsky's 'frame syst€m ft€ory', and the notion of 'sclipts' all serve as background to the project reported he,re. The basic theme to this literanre is that an LSP test for shrdents in a specific field facilitat€s a match betwee'n the examinee's field of study and the topic and stnrcture of the text (see pp. 13-21), The shrdent's comprehension is affected by his or her ubryry -to ilraw on speciamd knowledge bases. Clapham has chosen the term 'background knbwldge' as her own technical t€rm for this notion, although she notes that background knowledge is a notoriously difficult t€rm to assess and thenefore circumscribe. Hence, the first mine in the minefield is definitional. Her literature review (Chryt€rs 2 and 3) contains an overview of methodological iszues and rmresolved zubstantive questions in ESP testing both of which serned to stnrcture her project. Chaptff 4 describes the IELTS history already cite4 but also serves as a 'validity narrative' or extended discoltrse of evidence about the developrme,nt and rhinking that constituted the test rmder analysis (Lpch and Davidson, 1997). Chapt€r 4 is quite rich in its detail about the spocifications of the IELTS, feedback on the test ftom many int€rested parties, and its early piloting. This is the most cnrcial cla1> t€r iil the book,-because the reader must accept the basic stnrcture of the IELTS: three subject-specific tests of Business and Social Science (BSS), Medical Scie,nces Physical Science and Technologr (PST) and Life and 'T* early s!1ge of (LMS). A second mine may have been detected ev€n ut the book: despite the many years of work (from 1980 to the early 1990s), despite the great cost in personnel and srryport, have the IELTS team scGUrately dividd lp the academic world? One wonde,rs wheth€r the tripartite ta:ronomy in the IELTS test 'modtrles' (BSS, PST and LMS) nuly demarcate separable -CnAtersacademic disciplines. 5 to l0 r€,port on the strdy itself, both a pllot project (Chapt€r 5) and fte extensive ;ain study that forms the bulk of the book (Chapten GlQ). Appendices provide the te.st instnrments, sample specifications, ffid various other useful background information for both the pilot and main studies, The pilot sfirdy was a crtrcial tryout of procedures to detect the effect of students' field of shrdy on performance on the IELTS reading tast. These early data show'almost no evidence that students were disaftrantaged if they took an unzuitable subject module' (p. 86). Results of the pilot were necejsarily inconclusive due to a small sample size; the purpose of the pilot

was to wort out the analytical pocedures. The main shrdy was a richer, full€r, larger project.

It took

advantage

of

290

Book reviews

mines mapped and lessons learned in the pilot shrdy.

It examined

eigbt

research questions on effect of background knowledge, timing of the student

specialty (e.g., does one classiff a student as 'BSS' hcause he or she is planning to study business or because he or she has alrcady studied it?), ditrerences betrreen and rmdergraduatesn characteristics of the reading passages in the module tests, formal study vs informal shrdy of a particular module's area (e.g., is a PST stude,nt who reads in LMS b€n€fited?), and an examination of ge,ne,ral language proficiency both as a s€parafe variable and in conjrmction with background knowledge. The reading passages cove,red an array of topics within each module,

for

example, 'The Purposes of Continuing Education' (BSS), 'The Mystery of Declining Tooth Decay' (LMS), and 'Life Without a Sunscreen' (LMS). The texts wer€ from a variety of sources, including published malerial, gove,mment documens, ild one rmknoum textbook sourice. Readability analysis and conte,nt ratings were done of the passages and of their items. The conte,nt analpis was a modified form of Bachman's analytical system (see Bachman, Davidson and Milanovic, 1996; Bachman, 1990). Subjects represe,nted a range of academic levels, from pre-rmdeqgraduate, undergraduate, postgraduarc, t€chnical training, teachs training, and 'not sure'. There were 787 sfirdents in the main sfirdy, each declarcd prinarily

to be in either a BSS, PST or LMS field of study (with 65 listed as 'combined'). Subjects also took a grammar test and a lenglhy self-report

questionnaire; the former was ope,rationalized as a measnrre of language proficiency while the latter was a collection of questions about prior study, background knowledge, otrd familiarity with the test's topics. A}IOVA, item-level lt€m Response Analyses Theory bias analysis, regression, and various statistics on the itein/passage Bachman-style content ratings. Results revealed that some stude,nt grorrys were somewhat advantaged by faking a test within their onn field of study.

included@

Wher smaller units of analysis were examined (e.9., individual passages), texts 'seeined to vary in their specificity ftom ones which were highly specific to ones which w€re equally appropriate for students in all three subject areas' (p. 131), The questionnaire se,lrred to disentangle ftis passage-level variety. By far the most fascinating part of the book is formd as Clapham charts new te,rritory to explain how somepassages seem to advantage groups for whom they we,re nominally chosen (e.g., an LMS passage for the LMS goup), some do not and some seem decidedly inconclusive, Clapham

employed test review not only by applied linguists for the Bachman-slyle content analysis, but also (wisely) by subjet-area specialists. She surnolof,izes (pp. 199-200) the inconclusive, ofte,n conflicting, results of subjectspecialist and applied-linguist conte,nt analysis of the passqges. The basic dilemma she illusfiates is this: just because a passage is ostensibly labelled as belonging to a particular academic discipline does not mean that it will yield eryirical data to veriff it as a specific-field test task What is moren reading comprehension is an inte,raction of passage and item - the tnre complexity is at the netrus between the examinee, the passage, and the question being asked. Therre are seneral mines in the passage-choice sector

Book revisvss 291

of the minefiel4 possibly stacked one on top of the other

such that

uncovering one does not mean that the digging is done: subt€sb vary widety in their specificity. This variation is not necessarily obvious to t€st writers and users, and may well explain some of the contadictory results of earlisr reserch into the effect of background knowledge on reading

coryreheirsion (p. 189).

Other dilemmas in special-field t€sting relate to obtaining accurate data about background knowldge, accurate information about students' present and future field of study (itself investigated in the main study), and accurate readings from the examinee about self-pe,rceived effect of his or her knowledge about particular it€ms. All these features impact on Clapham's findings

in

co'mplex and sometimes contradictory ways. Nwertheless, Claphm always mrys the mmy different mines she was able to detect in clear md concise detail. What is going on? Why have,n't these specific-field tests worked more clearly? Possibly the answer does not lie in specific-field testing b_ut in the way Clapham and the IELTS team approached general nonspecific langutesting. 4ge -

The IELTS project chose to op€rdionalize general lmguage pnoficie'ncy as a score on a grammar test How would Clapham's results have ditrered had they had a largern richer definition of general ability in language? For example, what if they had included a very wide array of more general reading ta.sks coupled even with listening and speaking tests? Clearly, there wet€ practical conshain8 on the number of tests that each stude,lrt could sit, but adeitionat assessmenb would have provided a richer ope,rational definition of general lmgrrage ability. Had Claphm and her t€m done so, p€,!hE)s she would have seen that the IELTS module tests correlate ev€n better with ge,neral abillty. Perhaps she would have discovered that a gen€,ral ability test would have seirred the IELTS needs as well as the module tests. The minefietd mapped in this book serves not so much to describe specific-fields testing as to remind the reader that it night be acceptable to find another route and to avoid the minefield altogether. This conclusion would have been stnength€nd had Clapham had a wider arrdy of general proficienry t€$ts. Nonetheless, Clapham does appear to come to this conclusion herself. Chapt€r ll comes full sircle and generally advises against subjectspecific tests. This is most poignant in the following rcma* very near the elrd of the book, where Clapham seern to vent a very deep academic frustration: I have shown horn'difficult it is to make sure that [English for Specific Academic Purpoc€sl texts are specific to their designat€d subjec{ area, and what corylex procedures have to be followed to ensure that the t€xts arc specific. The inclusion of a general reading pas$age in what is stry'posedly a zubject

specific test may not matter to the $hd€nts as long as it does not place the'm at a disadvant4ge, but it destroys the whole purpos€ of having tesb in diffe'rent subject areas. All the effort of producing zubject specific t€sb is wasted if such tests turn out to consist of general reading fsages. However, even if the texts are specific or highly specific, it is not clear from my shrdy how many stud€nts

would profit or suffer from taking reading modtrles in different subjat areas.

292

Book revievtts

It

therefore seems advisable not to give academic shd€nt$ subject specific readngmodules, but to give them an [Fnglish for General Academic Purposes] r€ading test instead (pp.20G{l).

Caroline Clapham has unitten a major, seminal book- She has examined

a dangerous field of landmines, detected them, and disarmed them. Thi$ book will serve as a map of that minefield for years to come. Highereducation language de,parhents (this reviewer's rmiversity included) who are seriously considering special-fields testing should read this book sare-

fully. They will the,lr face two choices: travel the fiel4 disarm the same mines Clapham di4 and possibly come to the sarne inconclusive end result, or simply avoid the field altogether and rely on general test-tasks md topics. The latter option is certainly cheaper. References

Bachman, L.F. 1990: Funda nennl cowiderations in language testing. Onford: Oxford University Press. Bachman, L.F., I)avidson, F. and Milenovic, NL 1996: The use of test method characteristics in the content analysis and design of EFL proficie,ncy tests. Langwge Testing 13, 125-50. Lynch, B. and Davidson, F. 1997: Is my test valid? Paper presented at the 31st Annual TESOL Convention, Orlando, Florida" March. Fred Davidson Univercity of lllinois at Urbana-Chonpaigr