An ensemble of Weighted Support Vector Machines for Ordinal Regression

Instead of traditional (nominal) classification we investigate the subject of ordinal classification or ranking. An enhanced method based on an ensemble of Support Vector Machines (SVM-s) is proposed. Each binary classifier is trained with specific weights for each object in the training data set. Experiments on benchmark datasets and synthetic data indicate that the performance of our approach is comparable to state of the art kernel methods for ordinal regression. The ensemble method, which is straightforward to implement, provides a very good sensitivity-specificity trade-off for the highest and lowest rank.

Authors:

• Willem Waegeman
• Luc Boullart

Keywords:

• Ordinal regression
• support vector machines
• ensemblelearning.

References:

[1] A. Agresti, Categorical Data Analysis, 2nd version. John Wiley and
Suns Publications, 2002.
[2] J. DelCoz, G. Bayn, J. Dez, O. Luaces, A. Bahamonde, and C. Saudo,
"Trait selection for assessing beef meat quality using non-linear svm,"
in Proceedings of the conference on Neural Information Processing
Systems, Vancouver, Canada, 2004, pp. 321-328.
[3] R. Herbrich, "The trueskill(tm) ranking system," 2005,
http://www.research.microsoft.com/mlp/trueskill/.
[4] S. Kramer, G. Widmer, B. Pfahringer, and M. Degroeve, "Prediction
of ordinal classes using regression trees," Fundamenta Informaticae,
vol. 24, pp. 1-15, 2000.
[5] K. Cao-Van, "Supervised ranking, from semantics to algorithms," Ph.D.
dissertation, Ghent University, Belgium, 2003.
[6] R. Potharst and J. Bioch, "Decision trees for ordinal classification,"
Intelligent Data Processing, vol. 4, no. 2, 2000.
[7] W. Cohen, R. Schapire, and Y. Singer, "Learning to order things," in
Advances in Neural Information Processing Systems 10, 1998.
[8] K. Crammer and Y. Singer, "Pranking with ranking," in Proceedings of
the conference on Neural Information Processing Systems (NIPS), 2001.
[9] E. Harrington, "Online ranking/collaborative filtering using the perceptron
algorithm," in Proceedings of the 20th International Conference on
Machine Learning, Washington, USA, 2003.
[10] R. Herbrich, T. Graepel, and K. Obermayer, "Large margin rank boundaries
for ordinal regression," in Advances in Large Margin Classifiers,
2000, pp. 115-132.
[11] J. Shawe-Taylor and N. Cristianini, Kernel Methods for Pattern Analysis.
Cambridge University Press, 2004.
[12] A. Shashua and A. Levin, "Ranking with the large margin principle:
two approaches," Advances in Neural Information Processing Systems,
vol. 15, 2003.
[13] W. Chu and S. Keerthi, "New approaches to support vector ordinal
regression," in Proceedings of the 22th International Conference of
Machine Learning, Bonn, Germany, 2005, pp. 321-328.
[14] T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical
Learning. Springer, 2001.
[15] B. Scholkopf and A. Smola, Learning with Kernels, Support Vector
Machines, Regularisation, Optimization and beyond. The Mit Press,
2002.
[16] E. Frank and M. Hall, "A simple approach to ordinal classification," in
Proceedings of the European Conference on Machine Learning, 2000.
[17] S. Abe and T. Inoue, "Fuzzy support vector machines for multiclass
problems," in European Symposium on Artificial Neural Networks, 2002,
pp. 113-118.
[18] C.-C. Chang and C.-J. Lin, "LIBSVM: a library for support vector
machines," 2001, http://www.csie.ntu.edu.tw/ cjlin/libsvm.