ABSTRACT: To examine to what degree argumentation skills are mastered by pupils who attend the vocational, general and academic streams in Dutch secondary ... investigates: (1) what education is offered in secondary schools towards.
Empirical Research on the Identification of Singular, Multiple and Subordinate Argumentation R. J. OOSTDAM University of Amsterdam Centrefor EducationalResearch GroteBickersstraat72 1013 KS Amsterdam The Netherlands
ABSTRACT: To examine to what degree argumentation skills are mastered by pupils who attend the vocational, general and academic streams in Dutch secondary education various subtests were constructed. The theoretical study of argumentation as exposed by Van Eemeren and Grootendorst was the basis for this test construction. In this article tests for the identification of singular, multiple and subordinate argumentation are described. Also an account is given of a pretest of these three subtests. KEY WORDS: Argumentation, structures of argumentation, speech act, disputes, critical discussion, empirical research, facet-designs.
1. INTRODUCTION
It is important that language users have well developed argumentation skills at their command. After all, argumentation is encountered in many daily situations and often one must sustain a point of view with arguments. It's therefore important that language users are able to propound argumentation for their own point of view and can recognize what point of view an opponent refers to in a dispute and what arguments are given by the opponent to support his point of view. At the Centre for Educational Research of the University of Amsterdam the research-project Assessment of the teaching and learning of argumentation skills in secondary education is carried out. This study investigates: (1) what education is offered in secondary schools towards the development of argumentation skills, and (2) to what degree receptive (reading) and productive (writing) argumentation skills are mastered by pupils ending the education process. To measure the receptive and productive skills of pupils tests were constructed each focussing on a particular subskill.1 Starting-point for this test-construction was the theoretical approach and conceptual analysis of argumentation, as expounded by Van Eemeren and Grootendorst in their thesis Speech Acts in Argumentative Discussions (1984). Most of the tests Argumentation 4: 223-234, 1990. © 1990 KluwerAcademic Publishers.Printedin the Netherlands.
224
R. J. OOSTDAM
are constructed on the principle of a facet-design in which several facets can be systematically varied. In this article the construction and pretest of three tests aimed at the identification of singular, multiple and subordinate argumentation are described. Relevant questions to be answered are: (1) is the reliability of the test satisfactory? (2) are the tests appropriate in measuring the various levels of pupils' command of skills in identifying argumentation? (3) what is the effect of the manipulated facets on the identification of argumentation?
2. STRUCTURES OF ARGUMENTATION
Disputes and the argumentation advanced for a point of view can have various degrees of complexity. Van Eemeren and Grootendorst (1984) define the speech act that language users sustain with argumentation as the expressed opinion. The point of view is someone's attitude towards an expressed opinion. A positive point of view for an expressed opinion such as 'actors are vain' is 'to my opinion actors are vain'. A negative point of view is 'I don't believe that actors are vain'. Analogous to this distinction four types of dispute can be identified. The first two types are single and multiple disputes. Single disputes concern with one and no more than one expressed opinion and multiple disputes have to do with more than one expressed opinion. Furthermore it is necessary to discern simple and compound disputes. In simple disputes only one (positive or negative) point of view is advanced in respect of an expressed opinion and in compound disputes two different points of view (one positive and one negative) are propounded. It may be clear that different combinations can occur: simple single disputes, compound single disputes, simple multiple disputes and compound multiple disputes. Further distinction has to be made between the principal discussion, related to the initial expressed opinions and initial points of view, and subdiscussions occasioned by subdisputes. Subdisputes are related to subordinate expressed opinions and subordinate points of view. Argumentation given for the initial expressed opinion is defined as the principal argumentation and argumentation sustaining the subordinate expressed opinion as the subargumentation. The complexity of the argumentation for an expressed opinion can differ. The following possibilities exist: (a) the argumentation consists of one (principal) argumentation; (b) the argumentation consists of two or more principal argumentations each of which is individually sufficient to justify the initial expressed opinion, for which however neither is individually necessary;
EMPIRICAL RESEARCH ON ARGUMENTATION
225
(c) the argumentation consists of two or more principal argumentations, each of which is individually necessary to justify the initial expressed opinion, for which however they are sufficient only in combination; (d) the argumentation consists of one or more principal argumentations and one or more subargumentations. To examine to what extent pupils are able to identify structures of argumentation, three subtests were constructed for the above mentioned forms (a), (b) and (d). No test was constructed for form (c) in this survey, because the difference between (b) and (c) is based upon a personal judgement of the strength of the advanced arguments. If pupils can identify form (b) it is most likely that they can also identify form (c).
3. TEST-FORM
Each subtest has the character of a pencil-and-paper test. This means that pupils must answer a great number of items: short texts with a multiple choice question. This test-form has two great advantages. First the test is very appropriate for large-scale surveys because of the objective rating. Secondly it is very easy to present texts in a natural way. The pencil-and-paper tests are constructed in such a way that certain factors which may influence the identification of argumentation vary systematically among the test. Factors that may interfere are kept under control, such as: style, sentence-length, the tension of the points of view etc. The instruction for the test can be easily read by pupils without any explanation from the teacher. The concept of argumentation is defined on the basis of examples (ostensive definition). Then the specific task is explained, for example the identification of subordinate argumentation, and some example-items are given to demonstrate how the questions must be answered. In the instruction it is emphasized that the test is not a speed test. Every pupil will get enough time to finish the test.
4. THE FACET-DESIGNS
Each subtest contains a large number of items, short texts constructed according to the specific form of argumentation language users have to identify. In addition to these argumentation-items some items with no argumentation were incorporated in the tests. The items have to comply with different requirements. First they must cover the object in view, which means that in the constructed texts the
226
R. J. OOSTDAM
different combinations of points of view with arguments must appear. Secondly it is essential that the items differ in the level of complexity, because the tests must measure the various levels of command in identifying argumentation. Earlier studies about the identification of singular argumentation carried out at the University of Amsterdam (Braas, Jungslager, & Oostdam, 1985; Van Eemeren, Grootendorst, & Meuffels, 1984) can benefit this survey. The results of these studies gave abundant evidence that the following factors influence the identification of argumentation: (a) presence of indicators of argumentation as 'this', 'so', 'because' makes identification easier; (b) position of the argument (before or after the point of view) has significant influence: argumentation with the sequence point of view/ argument is less difficult to identify than the sequence argument/point of view; (c) when indicators of point of view as 'in my opinion' and 'according to me' are present language users identify the argumentation much better. In the constructed subtests for this study these three factors are varied among some others (see diagrams 1-3). The different combinations of factors can be classified in a facetted design (Lingoes, 1979). One of the important profits of such a design is its role in relating substantive theory and observations. In a facetted design the varied facets form several cells or structuples. Each structuple represents a form of appearance. Using a facet-design is one method to optimize the content-validity of a test (by verifying if the test includes all manifestations of the object in view). Below an overview is presented of the varied factors in the three tests. Furthermore a schematic reproduction is given of the facet-designs. An example-item illustrates the identification-task. In subtest 1 the identification of singular arguments the following factors are manipulated: -
-
presence of indicators for the point of view; presence of indicators for the argumentation; presence of indicators for the point of view in the formulation of the argument (the expectation is that the identification of the argument becomes more difficult); position of the argument: before (a) or after (b) the point of view.
Beside the items with singular argumentation 16 items with non-argumentation were incorporated, so the total number of items in the test is 48.
227
EMPIRICAL RESEARCH ON ARGUMENTATION Diagram 1. Facet-design subtest 1, the identification of singular argumentation (32 items; 16 structuples; 2 items per structuple). indicator point of view
yes
indicator argument
no
yes /
no
X
/
indicator point of view in argument
yes
position argument
ba ha
no
yes
yes
X
/
no
ha ha
yes
no
X
/
no
X
yes
no
ba ba ba ba
The following item is an example of the structuple indicator point of view (yes), indicator argument (yes), indicator point of view (yes), indicator point of view in argument (yes), position argument (a): John is rather clumsy, so in my opinion you must not employ him as a service-engineer. Argumentation? O Yes O No O Don't know If argumentation, underline the argument.
In subtest 2 the identification of multiple argumentation the manipulated factors are: -
presence of indicators for the point of view; presence of indicators for the argumentation; position of the argument: before (b) or after (a) the point of view.
The test contains also 8 items with singular argumentation and 10 items with non-argumentation. The total number of items in the test is 42. Diagram 2. Facet-design subtest 2, the identification of multiple argumentation (24 items; 8 structuples; 3 items per structuple). indicator point of view
yes
indicator argument position posargument argument
no
yes / b
no a a
/ b
yes a a
/ b
\
no a a
/ b
aon a
228
R. J. OOSTDAM
An example of the structuple multiple argumentation, indicator point of view (yes), indicator argument (yes), position argument (a) is: In my opinion Joyce must sign up for a correspondence course, because it will increase her chance on an appointment and it gives her thepossibility to meet some people. Argumentation? O Yes O No O Don't know If argumentation, underline the argument.
The factors in subtest 3 the identification of subordinate argumentation are: -
-
position of the initial point of view (IP), the subordinate point of view (SP), and the subargument (SA). The next sequences were included: IP-SP-SA, SA-SP-IP, SP-SA-IP, and IP-SA-SI; presence of indicators for the subordinate point of view; presence of indicators between subordinate point of view and subargument.
The total number of items in the test is 50. A number of 12 items with singular argumentation and 14 items with non-argumentation were added. Diagram 3. Facet-design subtest 3, the identification of subordinate argumentation (24 items; 12 structuples; 2 items per structuple). sequence points of view/ argumentation
SP-SA-IP
SA-SP-IP IP-SP-SA
indicator subordinate point of view
yes
yes
no
no
IP-SA-SP
yes
no
indicator argument
/ / \ yes no yes no
/ \ / \ yes no yes no
/ / yes no ydicator
argument
yes no yes no
yes no yes no
yes no yes no
An example of the structuple subordinate argumentation, position points of view and the argumentation (IP-SP-SA), indicator subordinate point of view (yes), indicator argument (yes) is: I better visit the hairdresser. In my opinion I must look good, because my girlfriend appreciates outward appearance. Argumentation? O Yes O No O Don't know If argumentation, underline the argument.
EMPIRICAL RESEARCH ON ARGUMENTATION
229
5. RESULTS OF THE PRETEST
In the period October-November 1988 the tests were presented to pupils in the pre final year of the four major school types in Dutch secondary education: junior vocational education (LBO), lower general secondary education (MAVO), higher secondary education (HAVO) and academic secondary education (VWO).2 Two forms of seven schools participated in the pretest. In each form two tests were made without any time limit. When there was time left an extra test was filled in. In Table II the number of testees for each school type are stated. Table I. Number of Respondents per School Type and per Subtest. School Type
Subtests 1
LBO MAVO HAVO VWO Total
2
3
88 51 46 77
45 96 37 46
39 54 35 46
262
224
174
The estimated reliabilities of the subtests are sufficient. As standard for the reliability the Gulliksen splithalf is used. This standard rates the lower limit of the reliability. The reliabilities indicate that the tests are appropriate to distinguish the different degrees in which pupils master the skill of identifying argumentation. These reliabilities also demonstrate that disturbing factors were kept under control and that the distinctions in the test scores can not be attributed to low reliability. Table II. Gulliksen Splithalfs per Test for the Argumentation and Non-argumentation Items, Divided into Multiple Choice and Underline Questions. Items
Subtests 1
2
3
Multiple Choice: Argu Non-argu
0.92 0.88
0.88 0.80
0.92 0.82
Underline: Argu Non-argu
0.95 0.86
0.95 0.78
0.86 0.86
230
R. J. OOSTDAM
Table II gives an overview of the rated reliabilities for the argumentative and non-argumentative items, divided into multiple choice questions (Argumentation yes/no) and underline-questions (If argumentation, underline the argument). Underneath only the results of the underline-questions will be discussed. These questions are the best indicators of a person's skill in identifying argumentation, since a correct underlined argument shows that not only the text is correctly identified as argumentation but also that the statement with the function of an argument is correctly underlined. The analyses were carried out by means of analysis of variance with a repeated measurement design. As the sample segmentation is unknown the results will be discussed on the basis of the F-ratio: the bigger the F-ratio the stronger the effect of the manipulated factors. For each test and school type the percentage correct identified argumentation-items is given. Subsequently the percentages correct identified argumentation-items are presented for each varied factor. Significant interactions will be mentioned as much as possible. Table III shows the results of subtest 1 (the identification of singular argumentation). The test differentiates well between the four school types. As expected the test is more difficult for pupils of the junior vocational education (LBO) and the lower general secondary education (MAVO). Pupils of the academic secondary education score very well. The fact that the score of LBO is somewhat higher than the score of MAVO must be described to school effects. Table III. Analysis Facet-design Subtest 1: The Identification of Singular Argumentation. Facets
Percentage correctly identified argumentation LBO
MAVO
HAVO
VWO
Whole group
Overall score
52.79
44.36
55.84
84.24
59.31
Point of view: With indicator Without indicator
55.16 50.41
48.09 40.63
57.20 54.48
85.46 83.02
61.48 57.13
17.27
Argument: With indicator Without indicator
55.98 49.59
49.59 39.13
60.89 50.82
88.99 79.48
63.86 54.76
55.35
Indicator point of view in argument: With indicator Without indicator
48.78 56.79
44.16 44.57
57.60 54.08
82.47 86.01
58.25 60.36
2.83
Position argument: Before point of view After point of view
41.44 64.13
40.49 48.23
47.01 64.67
78.80 89.67
51.94 66.68
32.64
F-ratio main effect
231
EMPIRICAL RESEARCH ON ARGUMENTATION
The factors 'indicators point of view' and 'indicators argument' have a strong effect on the identification of singular argumentation. Argumentation presented with an indicator for the point of view or the argument is better identified than argumentation without these indicators. Such an effect also obtains for the factor 'position argument'. Argumentation with the sequence point of view/argument is better recognized than the sequence argument/point of view. These results confirm the results of former studies (Braas et al., 1985; Van Eemeren et al., 1984, 1985). For the factor 'indicator point of view in argument' no main effect was established. On the other hand an interaction effect was observed with school type (F = 13.60). In particular, pupils of the junior vocational schools are deceived by this factor: they identify argumentation in which the argument has an indicator for the point view less often correct. For pupils of the higher and academic secondary education the identification of multiple argumentation is somewhat easier than the identification of singular argumentation. For the other two school types the scores are almost the same (see Table IV). Table IV. Analysis Facet-design Subtest 2: The Identification of Multiple Argumentation. Facets
Percentage correctly identified argumentation LBO
MAVO
HAVO
VWO
Whole group
F-ratio main effect
Overall score
41.21
51.69
73.82
93.92
65.16
Point of view: With indicator Without indicator
49.66 32.77
56.42 46.96
80.41 67.23
95.27 92.57
70.44 59.88
44.53
Argument: With indicator Without indicator
49.32 33.11
55.74 47.64
76.35 71.28
92.57 95.27
68.50 61.82
20.84
Position argument: Before point of view After point of view
32.77 49.66
47.30 56.08
63.51 84.12
91.55 96.28
58.78 71.54
33.15
The results of subtest 2 also reveals effects of the factors 'indicator point of view', 'indicator argument' and 'position argument'. Indicators for the point of view or for the argument simplify the identification and argumentation with the sequence point of view/arguments is better recognized than argumentation where the point of view is presented after the arguments. There is also a strong interaction effect between the factors 'indicator argument' and 'position argument' (F = 19.25): the factor 'indicator argument' has the strongest effect on the identification of multiple argumentation when the arguments precede the point of view. The hypothesis that the identification of subordinate argumentation is
232
R. J. OOSTDAM
more difficult than the identification of singular and multiple argumentation is supported by the results of subtest 3 (see Table V). Pupils of all four school types have problems identifying subordinate argumentation. Even pupils of the academic secondary education identify no more than 39 percent of the argumentation items correctly. Table V. Analysis Facet-design Subtest 3: The Identification of Subordinate Argumentation. Facets
Percentage correctly identified argumentation LBO
MAVO
HAVO
VWO
Whole group
Overall score
22.26
17.98
27.38
38.57
26.55
Sequence point of view/argumentation: SP/SA/IP IP/SP/SA, SA/SP/IP IP/SA/SP
9.28 42.50 15.00
8.57 32.86 12.50
18.21 40.71 23.21
37.14 48.57 30.00
18.30 41.16 20.18
47.86
Indicator subordinate point of view: With indicator Without indicator
22.86 21.67
17.62 18.33
26.19, 28.57
35.24 41.91
25.48 27.62
2.85
Indicator between subordinate point of view and subargument: With indicator Without indicator
17.68 24.29
17.62 18.33
31.19 23.57
39.29 37.86
27.08 26.01
0.83
F-ratio main effect
It is evident that these bottom-effects give no cause for finding maineffects. Only the factor 'position point of view/argumentation' shows a rather strong effect. The linear sequence with the initial point of view before or after the argumentation and the subordinate point of view in the middle (IP-SP-SA and SA-SP-IP) is recognized best. The other two sequences are very difficult to identify, particularly for pupils of the junior vocational and lower general secondary education.
6. DISCUSSION
The results of the pretest demonstrate that it is possible to construct subtests to measure the identification of argumentation structures. The high reliabilities indicate that the developed tests are appropriate in distinguishing the different degrees in which pupils master the skill of identify-
EMPIRICAL RESEARCH ON ARGUMENTATION
233
ing singular, multiple and subordinate argumentation. The reliabilities also testify that disturbing factors were kept under control. Proof was found that the manipulated factors have a strong effect on the identification task. This confirms the results of former studies on the identification of singular argumentation (Braas et al., 1985; Van Eemeren et al., 1984, 1985). These studies gave ample evidence that factors such as 'presence of indicators for the argumentation', 'indicators for the point of view' and 'position of the argument' strongly influence the identification task. In this study evidence was also found that the presence of factors such as 'indicators point of view' and 'indicators argument' have a strong effect on the identification of both singular and multiple argumentation. Argumentation with indicators of this kind was better identified than argumentation without these. The same effect obtains for the factor 'position argument': singular and multiple argumentation with the argumentation given before the point of view is easier to identify than the sequence in which the point of view precedes the argumentation. There is a strong interaction effect between the factors 'indicator argument' and 'position argument'. Pupils recognize singular and multiple argumentation better when the argument is placed before the point of view and when the argument includes an indicator. The results of subtest 1 'the identification of singular argumentation' prove that the 'indicator point of view in argument' is an influencing factor. In particular pupils of the junior vocational schools were deceived by this factor. They identify singular argumentation in which the argument has an indicator for the point of view less often. Their judgement regarding whether or not a statement has the function of an argument is strongly dominated by aspects of the formulation. Finally it is clear that the identification of subordinate argumentation is rather problematic for pupils of all four school types. Especially sequences with the subargument in the middle, often occurring in daily situations, are difficult to identify.
NOTES I For the measurement of the receptive argumentation skills a total number of 10 subtests was constructed. Apart from the three subtests for the identification of singular, multiple and subordinative compound argumentation tests were constructed for (a) the identification of points of view in single and multiple disputes, (b) the reconstruction of unexpressed premisses, (c) the analysis of concealed arguments (arguments formulated as rhetorical questions), (d) the evaluation of employed argumentation schemes and (e) the evaluation of the reliability of argumentation. For a detailed description of these tests see Oostdam et al., 1989. 2 The average age of pupils of the four school types differ. To a large extent this explains
234
R. J. OOSTDAM
distinctions in the various levels of command of pupils of the four school types. The junior vocational education (LBO) and the lower general secondary education (MAVO) have four grade levels. The third grade can be compared with the ninth grade of high school. The higher secondary education (HAVO) has five grade levels. Grade level four is comparable with the tenth grade. Academic secondary education (VWO) has six grades. The fifth grade is similar to grade eleven of high school.
BIBLIOGRAPHY Braas, C. W. P., F. S. Jungslager, and R. J. Oostdam: 1985, 'Empirisch onderzoek naar de identificatie van argumentatie', Taalbeheersing in theorie en praktijk, Foris, Dordrecht, pp. 29-37. Eemeren, F. H., R. Grootendorst, and B. Meuffels: 1984, 'Het identificeren van enkelvoudige
argumentatie', Tijdschrift voor Taalbeheersing6, 297-310. Eemeren, F. H. van and R. Grootendorst: 1984, Speech Acts in Argumentative Discussions:
A Theoretical Modelfor the Analysis of DiscussionsDirected TowardsSolving Conflicts of Opinion, Foris, Dordrecht. Eemeren, F. H., R. Grootendorst, and B. Meuffels: 1985, 'Gedifferentieerde replicaties van identificatie-onderzoek', Tijdschrift voor Taalbeheersing 7, 241-257. Lingoes, J. (ed.): 1979, Geometric Representationsof RelationalData, Mathesis, Ann Arbor. Oostdam, R. J. and M. J. Couzijn: 1989, 'Argumentatie in de peiling: instrumenten voor het meten van receptieve argumentatievaardigheden', Tijdschrift voor Taalbeheersing 11, 177-196.
