data set. The Kappa statistic is shown to be a more discern- ing statistical tool for assessing the classification accuracy of different classifiers and has the added ...
REMOTE SENS. ENVIRON. 47:362-368 (1994)
Assessing the Classification Accuracy of Multisource Remote Sensing Data R. W. Fitzgerald* and B. G. Lees* Classification accuracy has traditionally been expressed by the overall accuracy percentage computed from the sum of the diagonal elements of the error, confusion, or misclassification matrix resulting from the application of a classifier. This article assesses the adequacy of the overall accuracy measure and demonstrates that it can give misleading and contradictory results. The Kappa test statistic assesses interclassifier agreement and is applied in assessing the classification accuracy of two classifiers, a neural network and a decision tree model on the same data set. The Kappa statistic is shown to be a more discerning statistical tool for assessing the classification accuracy of different classifiers and has the added advantage of being statistically testable against the standard normal distribution. It gives the analyst better interclass discrimination that the overall accuracy measure. The authors recommend that the Kappa statistic be used in preference to the overall accuracy as a means of assessing classification accuracy.
INTRODUCTION In his timely article on assessing classification accuracy, Congalton (1991) reviewed a number of interrelated aspects of classification accuracy. The aspects reviewed by Congalton include site- and non-site-specific accuracy, statistical techniques (the Kappa test statistic) for comparing between classifiers, ground truthing errors, the discreteness of classification schemes, spatial autocorrelation, and sampling issues. Like Congalton, we believe that the assessment of errors in the classification of remote sensing and GIS data has been poorly examined and deserves intensive scrutiny.
*Geography Department, Australian National University, Canberra. Address correspondence to R. W. Fitzgerald, Geography Dept., Australian National Univ., Canberra ACT 0200, Australia. Received 29 April 1992; revised 8 June 1993.
362
The article examines two of the issues raised by Congalton (1991): the assessment of site-specific accuracy by the application of a statistical technique which was designed to test interclassifier agreement and the Kappa test statistic itself. In doing so, we demonstrate that the accepted method of assessing classification accuracy, the overall accuracy percentage, is misleading especially so when applied at the class comparison level. The Kappa statistic is shown to be a statistically more sophisticated measure of interclassifier agreement than the overall accuracy and gives better interclass discrimination than the overall accuracy. The article advances the application of the Kappa statistic by using it to compare the classification results of two supervised classifiers, a neural network and decision tree classifier applied to the same input data set. Each classifier successfully fused multisource remote sensing and GIS images into a single classified floristic land cover image. These two fused images were compared to highlight the relative utility of the Kappa statistic over the overall accuracy percentage especially at the interclass level. A backpropagation ne~aral network (Fitzgerald and Lees, 1992) and a decision tree model (Lees and Ritman, 1991) were applied to the task of floristic land cover classification. Both classification techniques come from the artificial intelligence field and have been applied in many disciplines where traditional statistical classifiers have been found wanting. Decision trees are a rule-based classifier and apply top-down induction to the input data to partition each input record into a class at the end of the branches from user-defined decision rules. The resulting classification is robust against the underlying statistical distributions of the input variables and easily mixes continuous (remote sensing data) and categorical variables (GIS attribute data). It does, however, require large numbers of training sites as a squared function of the number of decision nodes. The most popular and robust neural network is 0034-4257 / 94 / $7.00 ©Elsevier Science Inc., 1994 655 Avenue of the Americas, New York, NY 10010
Assessment of Classification Accuracy
,
r'oint
:: ~ ~ Brush •y Islant {Nondnra Point
Kiolo~
:
~: ~
c , ....~-~....LO°HaraPoint Snapper Point
Durras Mountain ~
I
Figure 1. Location of the study area, Kioloa, NSW, Australia.
the three-layer backpropagation network. T h e n e t w o r k creates a set of weights which m a p the input variables to a known output class via the hidden layer. The network progressively learns to minimize the classification error
of the input to output mapping. By contrast to a decision tree, neural networks have no a priori implicit rules and require fewer training sites while remaining robust classifiers. T h e study area for both exercises was identical and covered a 225 sq km area a r o u n d Kioloa in southeastern Australia (Fig. 1). T h e study area is complex, comprising a mixture of disturbed and partially cleared dry sclerophyll forest and rainforest on rough terrain with variable geology. Rainforest gullies and coastal heath are found mixed together along the coastal fringe, thus adding to the floristic complexity confronted by the classifiers. This area and the data sets used have b e e n previously discussed by Moore et al. (1990) and Lees and Ritman (1991). T h e 15 input variables for the decision tree model comprised r e m o t e sensing data derived from Landsat-4 TM Bands 1, 2, 4, 5, and 7. The GIS data included elevation, azimuth (classed aspect), position on slope index, catchment, geology, slope, downhill index, uphill index, horizon and steepness [see Moore et al. (1990) for details describing the variables in the Kioloa data set q u o t e d above]. The decision tree algorithm identified eight of these
Table 1. Agreement Analysis for the Neural Network Classifier ~ N
Agreement Level
121,0 99,6 10.8 104.9 138.4 75.5 85.6 164.0 208.7
62727 62727 62727 62727 62727 62727 62727 62727 62727
g p p g g p p g e
0.002
253.9
62727
g
0.455 0.412 0.126 0.274 0.431 0.216 0.285 0.950
0.030 0.030 0.019 0.030 0.029 0.024 0.030 0.030
15.3 13.9 6.6 9.3 14,9 9.1 9.5 31.7
1116 1116 1116 1116 1116 1116 1116 1116
g g p p g p p e
0.395
0.014
29.1
1116
p
Class
OA
~(
SE
Overall Dry sclerophyU E. botroyoides Lower slope wet Wet E. maculata Dry E. maculata RF Ecotone Rainforest Paddocks Sea
0.992 0.998 0.997 0.996 0.997 0.998 0.998 0.999 0.992
0.457 0.382 0.037 0.416 0.541 0.244 0.342 0.609 0.820
0.004 0.004 0.003 0.004 0.004 0.003 0.004 0.004 0.004
0.984
0.617
0.766 0.922 0.954 0.765 0.813 0.912 0.893 0.996 0.511
Overall table Land Dry sclerophyll E. botroyoides Lower slope wet Wet E. maculata Dry E. maculata RF Ecotone Rainforest Paddocks Overall table
363
Z
a Agreement analysis-- L x 9 (ground truth) vs. neural networks classified image; estimated Kappa values and agreement levels. (1) All the I~ values are statistically significant at p < 0.001. (2) Notation: OA = overall agreement proportion, I~ = estimated value of Kappa, the agreement test statistic, SE = the standard error of Kappa, Z = the standard normal variable, N = the grand total number of pixels, Agreement level: p = poor (I~ < 0.4) g = good (0.4 ~ K < 0.75) e = excellent (1~ > 0.75)
364 Fitzgerald and Lees
Table 2. Agreement Analysis for Lees and Ritman Decision Tree ModeP Class
OA
~
SE
Z
N
Agreement Level
Land Dry sclerophyll E. botroyoides Lower slope wet Wet E. maculata Dry E. maculata RF Ecotone Rainforest Paddocks
0.790 0.929 0.950 0.738 0.813 0.914 0.896 0.989
0.505 0.307 0.229 0.287 0.297 0.416 0.312 0.873
0.028 0.028 0.026 0.028 0.028 0.028 0.028 0.028
18.0 11.0 8.9 10.2 10.5 15.0 11.1 31.0
1257 1257 1257 1257 1257 1257 1257 1257
g p p g g p p g
0.509
0.391
0.013
30.3
1264
g
Overall table
Agreement analysis--L × 9 [ground truth] vs. Lees and Ritman; estimated Kappa values and agreement levels. (1) All the I~ values are statistically significant at p < 0.001. (2) Notation as in Table 1. (3) Lees and Ritman's model did not predict a sea class. The sea was masked out.
input variables as being important. The eight input variables used for the neural n e t w o r k analysis w e r e Landsat-4 TM Bands 2 (0.52-0.6/zm), 4 (0.76-0.9/zm), and 7 ( 1 0 . 4 - 1 2 . 6 / z m ) , aspect (a surrogate of the azimuth and horizon data sets), elevation, c a t c h m e n t (a surrogate of the position on slope, downhill and uphill indices), geology, and slope. The neural n e t w o r k application developed in this study p e r f o r m e d a m o r e floristically plausible classification of the entire Kioloa subscene than the corresponding decision tree m o d e l of Lees and Ritman (1991) with less input variables. T h e sample size of the training set used is 22 times smaller than used by Benediktsson et al. (1990) and 78 times smaller than Ritter and H e p n e r (1990). T h e quality of the results is attributed to both the mix of remotely sensed data and environmental variables as well as the quality and low noise levels of the Kioloa data set. A n u m b e r of authors have applied neural networks
to land cover classification with claimed classification accuracies of 80% (Benediktsson et al., 1990; Ritter and H e p n e r , 1990; Decatur, 1989; H e e r m a n n and Khazenie, 1991; Yin and Xu, 1991). These authors have delineated gross structures such as forests, urban areas, water, and o p e n paddocks. Only one author g r o u p cited, Benediktsson et al. (1990), has w o r k e d at the floristic classification level.
METHOD T h e essence of this article is to extend some important aspects of Congalton's (1991) article on errors in r e m o t e sensing classification by m o r e rigorously defining the use of the Kappa test statistic. The Kappa statistic tests the null hypothesis that two i n d e p e n d e n t classifiers do not agree on the rating or classification of the same physical object, in this case the class of a g r o u n d truth site. The two i n d e p e n d e n t classifiers are the h u m a n
Table 3. Agreement Analysis for the Neural Network vs. Lees-Ritman (1991) Decision Tree Classifier ~ Class
OA
K
SE
Land Dry sclerophyll E. botroyoides Lower slope wet Wet E. maculata Dry E. maculata RF Ecotone Rainforest Paddocks Overall table
0.746 0.903 0.971 0.693 0.764 0.954 0.850 0.951 0.415
0.372 0.207 0.130 0.238 0.245 0.107 0.077 0.593 0.275
0.002 0.002 0.002 0.002 0.002 0.002 0.002 0.002 0.001
Z 185.4 116.3 63.0 123.6 119.1 52.0 40.2 294.2 289.4
N
Agreement Level
233609 233609 233609 233609 233609 233609 233609 233609 233609
p p p p p p p g p
Agreement analysis--Lees and Ritman vs. neural networks classified image; estimated Kappa values and agreement levels. (1) All the I~ values are statistically significant at p < 0.001. (2) Notation as in Table 1. (3) Lees and Ritman's model did not predict a sea class. The sea was masked out. a
Assessment of Classification Accuracy
365
Table 4. Error Matrix for the Neural Network Classifier~" L × 9 Training sites NN Classified Dry sclerophyll
E. botroyoides Lower slope wet Wet E. maculata
Dry E. maculata RF ecotone Rainforest Paddocks Sea
1 1 2 3 4 5 6 7 8 9
Marginal p.i Column n Overall accuracy OA (land)
2
3
215 24 1 23 49 3 1 1 0
25 36 1 5 1 0 0 0 0
28 12 4 4 3 0 1 0 0
0.005 317
0.001 68
0.001 52
4
5
68 3 0 94 67 2 17 2 0
6
28 3 0 24 124 I 2 0 0
0.004 253
8 2 1 23 13 16 40 O 1
0.003 182
0.002 104
9
Marg p~
Row n
2 11 0 22 18 5 31 0 0
7 0 0 0 2 0 0 0 50 0
8
242 29 148 0 0 0 0 59 61,132
0.010 0.002 0.002 0.003 0.004 0.00O 0.001 0.002 0.975
616 120 155 197 175 27 92 112 61133
0.001 89
0.001 52
0.982 61610
1.000 62727
0.984 0.511
Agreement analysis--error matrix of L × 9 with neural networks classified image.
analyst who originally determined the floristic class for each ground truth site and an automated classifier such as the neural network and decision tree classifiers used in this article. The comparison of the ground truth classes with the automated classifier constitutes what Congalton (1991) terms site-specific accuracy assessment. By this he means that the locational and classification accuracy are both assessed together. In this respect, the agreement analysis results reported for the neural network (Table 1) and the decision tree model (Table 2) are site-specific accuracy assessments because they not only compare the class agreement, but do it at each specific ground truth site's location. By contrast, Congalton (1991) refers to non-sitespecific accuracy assessment as that applying to the class classifications while ignoring their locational accuracy.
The comparison of the neural network and decision tree classifiers in Table 3 is an example of this gross level non-site-specific accuracy assessment. The neural network and the decision tree model were both used to produce a ground truth site classification (site-specific accuracy assessment) and an entire subscene classification. The error matrices for the ground truth site classifications of each classifier are contained in Tables 4 and 5, respectively. From these error matrices, the overall accuracy percentages and the agreement analyses using the Kappa test statistic were created (Tables 1 and 2). The Kappa test statistic itself is discussed fully in Chapter 13 of Fleiss (1981). The computation of the Kappa statistic is based on a two-way cross-classification table (error matrix) which has k categories (see Table 6). From this error matrix, the overall proportion of
Table 5. Error Matrix for Lees and Ritman (1991) Decision Tree Classifiera L × 9 Training sites
CART Classified Dry sclerophyll
E. botroyoides Lower slope wet Wet E. maculata Dry E. maculata RF ecotone Rainforest Paddocks Sea Marginal pi. column n Overall accuracy OA (land)
1 1 2 3 4 5 6 7 8 9
2
3
4
250 11 4 32 39 4 11 7 0
18 23 5 11 3 7 9 0 0
25 6 11 7 3 2 6 0 0
63 6 3 140 43 11 20 0 0
0.284 357
0.061 77
0.047 59
0.226 285
5 47 2 2 59 81 4 10 0 0 0.163 205
6 5 3 1 41 12 46 9 0 0 0.093 117
7 0 9 1 33 12 7 38 0 0 0.080 100
8
9
Marg p~.
0 0 0 1 0 2 4 51 0
0 0 0 0 0 0 0 0 0
0.323 0.048 0.020 0.257 0.154 0.067 0.085 0.047 0.000
0.047 59
0.000 O
1.000
Row n 406 60 26 323 193 84 106 59 0 1257
0.509 0.509
a Agreement analysis-- error matrix Lees and Ritman; true site class (columns) and CART classification (rows). Column and row n are a proportional estimate from the 1991 article. Rounding error may occur.
366 Fitzgerald and Lees
Table 6. Error Matrix Expressed as Joint Proportions of Ratings by Two Independent Classifiers for k Classes Ground Truth Classes Classifier
1
2
1 2
pn p21
k Total
k
Total
pl2 p22
plk p~
pl . p2.
pkt
pk~
pa
pk-
p. 1
p' 2
p.~,
1
agreement po is calculated from Eq. (1) by summing all the diagonal elements p~i of the error matrix: k
po = E p , , .
(1)
i=1
The overall proportion of chance-expected agreement, pe, is computed from Eq. (2) by the sum of the marginal proportions p~. and p~: k
P, = EP"P'"
(2)
The estimated value of the Kappa statistic is computed from ~, =po--pe
1 -pe'
(3)
Finally, the standard error (SE) of the Kappa statistic is computed from 1 z k -- , r- [po + pe -- ~p,.p.i(p,. + p.,)l 1/2. (4) s.e.(g) = (1 - pe)~ln [ ,~1 The Kappa test statistic is used to test the null hypothesis that there is no (H0:K--0) agreement between the two independent classifiers. The Kappa test statistic is converted to the standard normal score Z
g Z= s.e.(I~----~)
(5)
and tested against the normal distribution. K ranges in value from 0 (no agreement) through 1 (complete agreement). Landis and Koch (1977) have suggested ranges of agreement for the Kappa test statistic. These are poor (K < 0.4), good (0.4 ~< K ~< 0.75) and excellent (K > 0.75). These ranges have been adopted in this article. RESULTS
Neural Network Classification Accuracy Table 4 contains the error matrix for the neural network floristic classifier. The overall accuracy is 0.984 or 98.4%. For the land portion of the subscene, the overall accuracy reduces to 51.1%. This overall land classification accuracy of 51% compares well with the rates
..
achieved by Benediktsson et al. (1990), 52%, Ritter and Hepner (1990), 69%, and Decatur (1989), 76%. When the error matrix is subject to an agreement analysis using the Kappa test statistic, the results of the neural network classification are less impressive. Referring to Table 1, the level of agreement for the overall error matrix (land and sea) as measured by the Kappa statistic is only 0.617, only a good level of agreement according to the criteria of Landis and Koch (1977). The Kappa statistic indicates that the level of interclassifier agreement is not as impressive as indicated by the overall accuracy of 98.4% quoted above for the neural network classifier in this study. When the land portion of the error matrix for the neural network classifier is tested with the Kappa statistic, the level of inter classifier agreement falls to a marginally good level of 0.395 (Table 1). The disparity between the overall accuracy of 51.1% for the land portion and the Kappa statistic value (0.395) is now not as great as for the overall land and sea comparison above. This is due in part to the dominance of the sea class which constitutes 97% of the overall result in Table 1. Using the agreement level criteria for the Kappa test statistic from Landis and Koch (1977) for the land portion in Table 1, the neural network had good agreement with the ground truth values for the floristic classes dry sclerophyll, E. botroyoides, dry E. maculata, and excellent agreement for the paddocks. All other classes agreed poorly. Also note from Table 1 that all the reported Kappa test statistics are statistically significant at the 0.001 level. Thus all the reported Kappa values are statistically nonzero values. Of particular interest in the agreement analysis is the disparity in the relative rankings of the overall accuracy values and the Kappa values in Table 1. The Kappa test statistic values for each class give more consistent answers by comparison to the corresponding overall accuracy values. As an example, for the land portion in Table 1, the classes with good agreement range in Kappa values from 0.412 through 0.455 while their corresponding overall accuracy values range from 0.766 through 0.922. We feel that the Kappa statistic is a more discriminating measure of class agreement than the overall accuracy percentage especially since it can
Assessment of Classification Accuracy 367
be tested for statistical significance against the standard normal distribution. The Lees-Ritman (1991) Decision Tree Classifier Accuracy Table 5 is the error matrix for the Kioloa sub scene generated by Lees and Ritman's (1991) decision tree model. Their model only dealt with terrestrial pixels and the overall accuracy was 50.9%. This compares well to the neural networks overall accuracy of 51.1% for the same land portion (see Table 4). The corresponding agreement analysis for the decision tree classifier (Table 2) shows an overall poor level of agreement (K--0.391) for this classification technique and in this respect is the same as the neural network result (Table 1, K = 0.395). Once again, a comparison of the overall accuracy values and their corresponding Kappa test statistics for each class demonstrates that the Kappa test statistic is better at assessing interclassifier agreement than the overall accuracy values. The overall accuracy values range upwards from 73 %, suggesting good to excellent levels of interclassifier agreement. This contrasts with that of the Kappa test statistic which shows that only three classes had good to excellent agreement and the bulk had poor levels of agreement (see Table 2). The decision tree model had nearly twice the number of input variables (15) of the neural network (8) and in this light could be considered to have performed poorly by comparison to the neural network classifier. A comparison of the Kappa statistic based agreement levels between the neural network (Table 1) and the decision tree classifier (Table 2) demonstrates the following: i. The two classifiers achieved good agreement levels but on different sets of classes. The decision tree classifier performed poorly on all classes except dry sclerophyll, rainforest ecotone, and paddocks. The neural network performed poorly on all classes except dry sclerophyll, E. botroyoides, dry E. maculata, and paddocks. ii. The two classifiers performed equally on dry sclerophyll and paddocks. A Comparison between the Neural Network and Decision Tree Land Cover Classification Each classifier was used to classify the entire Kioloa sub scene. The resulting images were then compared and the resultant agreement levels examined (see Table 3). This constitutes Congalton's (1991) non-site-specific accuracy assessment where only the overall class classification accuracy is considered while ignoring their locational accuracy. The overall level of agreement between the two classifiers is very poor as measured by the Kappa test statistic
(0.275) and by the overall accuracy of 41%, which is a little more optimistic in its assessment (Table 3). This result may in part be due to the much larger number of pixels being compared (n=233,609 compared to n = 1116 and n = 1257 in the previous Tables 1 and 2). All classes except paddocks have very poor levels of agreement (Table 3). The disparity between the overall accuracy values at the class level which range from 69% upwards when compared to the Kappa test statistic results is highlighted again in this Table 3. The best example is rainforest ecotone where the overall accuracy value is 95% while the Kappa value is 0.107, indicating a very poor level of agreement. All the Kappa test statistic values are statistically significant at the 0.001 level, thus indicating that we can reject the null hypothesis that the observed Kappa statistic values are 0.0. Where poor class classification agreement is found, a number of possibilities exist: i. The ground truth data is poor and thus misclassifled from the start. ii. There is a class definition problem. Congalton (1991) refers to this as classification scheme error: The classes used may not be discrete and may overlap. The ground truth data is good, but the original classification of quadrat data carried out by Davey (1989) generated classes in a continuum of vegetation communities from rainforest, through wet sclerophyll to dry sclerophyll. These classes do not overlap, but they are contiguous. Both classifiers are very successful at paddocks, suggesting that at least this class is distinct based on spectral and attribute criteria. iii. There is a classifier technique problem: The classifier may be unsuited to the task or it requires more input variables. As Lees and Ritman (1991) point out, the high level of misclassification of lower slope wet and wet E. maculata with dry E. maculata and dry sclerophyll reflects the problems of detecting understorey changes with remote sensing in these very disturbed environments. In these environments, the environmental domain components of the data set are not being constrained by the reflectance data. From a spatial perspective, the neural network generated image produces a more natural and sophisticated result than that of the decision tree model. Floristic types interfinger and mingle in the manner experienced in the field in the neural networks classification. CONCLUSIONS This article has taken two of Congalton's (1991) classification accuracy assessment issues (the Kappa test statistic and non-site-specific accuracy) and applied them to
368 Fitzgerald and Lees
the floristic classification errors generated by a neural network and decision tree classification of the Kioloa subscene. The following conclusions emerge from the agreement analysis applied to this subscene: a. The analysis demonstrates that the overall accuracy percentage is a poor index of classification accuracy, especially at the class level. Here the disparity between it and the Kappa test statistic values are most marked. b. The Kappa test statistic is a more rigorous and discerning statistical tool for measuring the classification accuracy of different classifiers. It is able to test the null hypothesis that there is no agreement between the two classifiers (ground truth and automated classifier) and, at the class level, test the strength and significance of interclassifier agreement. c. From the agreement analysis, the neural network discriminated slightly more classes at good and excellent agreement level than the decision tree model. At the overall agreement level both classifiers performed equally well (K = 0.4). These results testify to the good design of the Kappa statistic. It was specifically designed as a statistical tool for the evaluation of interclassifier problems. It is the appropriate tool for the job and is superior to the overall accuracy percentage. The range of values exhibited by the Kappa statistic are typically twice that of the overall accuracy. This helps considerably in discerning interclass differences especially when combined with the significance testing available for the Kappa statistic. From the analysis above, we propose that the Kappa test statistic be used in preference to the overall accuracy as a means of testing classification accuracy based on error matrices. The Kappa statistic is relatively easy to compute and interpret and was designed specifically for evaluating interclassifier problems.
REFERENCES
Benediktsson, J. A., Swain, P. H., and Ersoy O. K. (1990), Neural network approaches versus statistical methods in
classification of multi-source remote sensing data, in 9th Annual Int. Geoscience and Remote Sensing Symposium-IGARSS "89, May, 1990, College Park, MD, IEEE, Piscataway, NJ, pp. 489-492. Congalton, R. G. (1991), A review of assessing the accuracy of classification of remotely sensed data, Remote Sens. Environ. 37:35-46. Davey, S. M. (1989), The environmental relationships of arboreal marsupials in a eucalypts forest: a basis for Australian wildlife management, PhD thesis, Forestry Dept., Australian National University, Canberra, Australia. Decatur, S. E. (1989), Applications of neural networks to terrain classification, International Joint Conference on Neural Networks (IJCNN "89), Vol. 1, IEEE, Piscataway, NJ, pp. 151153. Fleiss, J. L. (1981), Statistical Methods for Rates and Proportions, Wiley, New York. Fitzgerald, R. W., and Lees, B. G. (1992), The application of neural networks to the floristic classificationof remote sensing and GIS data in complex terrain, in Proceedings of the 17th ISPRS, Washington, DC, 2-14 August, 1992, ASPRS, Bethesda, MD. Heermann, P. D., and Khazenie, N. (1991), Application of neural networks for classification of multi-source multispectral remote sensing data, in lOth Annual International Geoscience and Remote Sensing Symposium, IGARSS gO, College Park, MD, IEEE, Piscataway, NJ, pp. 1273-1275. Hepner, G. F., Logan, T., Ritter, N., and Bryant, N. (1990), Artificial neural network classification using a minimal training set-comparison to conventional supervised classification, Photogramm. Eng. Remote Sens. 56(4):469-473. Landis, J. R., and Koch, G. C. (1977), The measurement of observer agreement for categorical data, Biometrics 33:159174. Lees, B. G., and Ritman, K. (1991), Decision tree and rule induction approach to integration of remotely sensed and GIS data in mapping vegetation in disturbed or hilly environments, Environ. Manage. 15:823-831. Moore, D. M., Lees, B. C., and Davey, S. M. (1990), A new method for predicting vegetation distribution using decision tree analysis in geographic information systems, Environ. Manage. 15:50-71. Ritter, N. D., and Hepner, G. F. (1990), Application of an artificial neural network to land-cover classification of Thematic Mapper imagery, Comput. Geosci. 16(6):873-880. Yin, Y., and Xu, X. (1991), Applying neural net technology for multi-objective land use planning, J. Environ. Manage. 32:349-356.
