Ensemble Learning with Decision Tree for Remote Sensing Classification

In recent years, a number of works proposing the combination of multiple classifiers to produce a single classification have been reported in remote sensing literature. The resulting classifier, referred to as an ensemble classifier, is generally found to be more accurate than any of the individual classifiers making up the ensemble. As accuracy is the primary concern, much of the research in the field of land cover classification is focused on improving classification accuracy. This study compares the performance of four ensemble approaches (boosting, bagging, DECORATE and random subspace) with a univariate decision tree as base classifier. Two training datasets, one without ant noise and other with 20 percent noise was used to judge the performance of different ensemble approaches. Results with noise free data set suggest an improvement of about 4% in classification accuracy with all ensemble approaches in comparison to the results provided by univariate decision tree classifier. Highest classification accuracy of 87.43% was achieved by boosted decision tree. A comparison of results with noisy data set suggests that bagging, DECORATE and random subspace approaches works well with this data whereas the performance of boosted decision tree degrades and a classification accuracy of 79.7% is achieved which is even lower than that is achieved (i.e. 80.02%) by using unboosted decision tree classifier.

Authors:

• Mahesh Pal

Keywords:

• Ensemble learning
• decision tree
• remote sensingclassification.

References:

[1] G. Giacinto and F. Roli, Ensembles of neural networks for soft
classification of remote sensing images. Proceedings of the European
Symposium on Intelligent Techniques, European Network for Fuzzy
Logic and Uncertainty Modelling in Information Technology, Bari,
Italy, 166-170, 1997.
[2] J. A.,Benediktsson, J. R.,Sveinsson, O. K.,Ersoy and P. H., Swain,
Parallel consensual neural networks. IEEE Trans. Neural Networks, 8,
1997, 54-65.
[3] F. Roli, G. Giacinto and G. Vernazza, Comparison and combination
of statistical and neural networks algorithms for remote-sensing
image classification. Neurocomputation in Remote Sensing Data
Analysis, Austin, J., Kanellopoulos, I., Roli, F. and Wilkinson G.
(Eds.), Berlin: Springer-Verlag, 117-124, 1997.
[4] L., Breiman, Bagging predictors, Machine Learning, 26, 1996, 123-
140.
[5] Y.Freund and R. Schapire, Experiments with a new boosting
algorithm. Machine Learning: Proceedings of the Thirteenth
International conference, 148-156, 1996.
[6] M. A. Friedl, C. E. Brodley, and A. H. Strahler, Maximizing land
cover classification accuracies produced by decision tree at
continental to global scales. IEEE Transactions on Geoscience and
Remote Sensing. 37, 1999, 969-977.
[7] M. Pal and P. M. Mather, Decision tree classifiers and land use
classification. Proceedings of the 27th Annual Conference of the
Remote Sensing Society, 12-14 September, London, UK, 2001.
[8] G. J., Briem, J. A.,Benediktsson, and J. R., Sveinsson, Multiple
Classifiers Applied to Multisource Remote Sensing Data IEEE
Transactions on Geoscience and Remote Sensing, 40, 2002, 2291-
2299.
[9] P. Melville and R. Mooney, Constructing diverse classifier ensembles
using artificial training examples. In Proceedings of the Eighteenth
International Joint Conference on Artificial Intelligence, 505-510,
2003, Acapulco, Mexico, August.
[10] T.K.Ho, The Random Subspace Method for Constructing Decision
Forests. IEEE Transactions on Pattern Analysis and Machine
Intelligence 20, 1998, 832-844.
[11] L. Hansen and P. Salamon, Neural network ensembles, IEEE
Transactions on Pattern recognition and Machine intelligence, 12,
1990, 993-1001.
[12] A. Krogh and J. Vedelsby, Neural networks ensembles, cross
validation and active learning. In D.S. Touretzky, G. Tesauro, and
T.K. Leen, editors, Advances in Neural Information Processing
Systems, volume 7, pages 107-115, 1995, MIT Press, Cambridge,
MA.
[13] L.,Breiman, J.H. Friedman, R.A.,Olshen and C.J.,Stone,
Classification and Regression Trees, Wadsworth, Monterey, CA,
1984.
[14] S. K.Murthy, S. Kasif and S. Salzberg, A system for induction of
oblique decision trees. Journal of Artificial Intelligence Research, 2,
1994, 1-32.
[15] I. Kononenko and J. S. Hong, Attribute selection for modelling.
Future Generation Computer Systems, 13, 1997, 181-195.
[16] J. Mingers, An empirical comparison of selection measures for
decision tree induction. Machine Learning, 3, 1989, 319-342.
[17] J. R. Quinlan, C4.5: Programs for Machine Learning. San Mateo:
Morgan Kaufmann, San Francisco, 1993.