An ensemble of Weighted Support Vector Machines for Ordinal Regression
Instead of traditional (nominal) classification we investigate
the subject of ordinal classification or ranking. An enhanced
method based on an ensemble of Support Vector Machines (SVM-s)
is proposed. Each binary classifier is trained with specific weights
for each object in the training data set. Experiments on benchmark
datasets and synthetic data indicate that the performance of our
approach is comparable to state of the art kernel methods for
ordinal regression. The ensemble method, which is straightforward
to implement, provides a very good sensitivity-specificity trade-off
for the highest and lowest rank.
[1] A. Agresti, Categorical Data Analysis, 2nd version. John Wiley and
Suns Publications, 2002.
[2] J. DelCoz, G. Bayn, J. Dez, O. Luaces, A. Bahamonde, and C. Saudo,
"Trait selection for assessing beef meat quality using non-linear svm,"
in Proceedings of the conference on Neural Information Processing
Systems, Vancouver, Canada, 2004, pp. 321-328.
[3] R. Herbrich, "The trueskill(tm) ranking system," 2005,
http://www.research.microsoft.com/mlp/trueskill/.
[4] S. Kramer, G. Widmer, B. Pfahringer, and M. Degroeve, "Prediction
of ordinal classes using regression trees," Fundamenta Informaticae,
vol. 24, pp. 1-15, 2000.
[5] K. Cao-Van, "Supervised ranking, from semantics to algorithms," Ph.D.
dissertation, Ghent University, Belgium, 2003.
[6] R. Potharst and J. Bioch, "Decision trees for ordinal classification,"
Intelligent Data Processing, vol. 4, no. 2, 2000.
[7] W. Cohen, R. Schapire, and Y. Singer, "Learning to order things," in
Advances in Neural Information Processing Systems 10, 1998.
[8] K. Crammer and Y. Singer, "Pranking with ranking," in Proceedings of
the conference on Neural Information Processing Systems (NIPS), 2001.
[9] E. Harrington, "Online ranking/collaborative filtering using the perceptron
algorithm," in Proceedings of the 20th International Conference on
Machine Learning, Washington, USA, 2003.
[10] R. Herbrich, T. Graepel, and K. Obermayer, "Large margin rank boundaries
for ordinal regression," in Advances in Large Margin Classifiers,
2000, pp. 115-132.
[11] J. Shawe-Taylor and N. Cristianini, Kernel Methods for Pattern Analysis.
Cambridge University Press, 2004.
[12] A. Shashua and A. Levin, "Ranking with the large margin principle:
two approaches," Advances in Neural Information Processing Systems,
vol. 15, 2003.
[13] W. Chu and S. Keerthi, "New approaches to support vector ordinal
regression," in Proceedings of the 22th International Conference of
Machine Learning, Bonn, Germany, 2005, pp. 321-328.
[14] T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical
Learning. Springer, 2001.
[15] B. Scholkopf and A. Smola, Learning with Kernels, Support Vector
Machines, Regularisation, Optimization and beyond. The Mit Press,
2002.
[16] E. Frank and M. Hall, "A simple approach to ordinal classification," in
Proceedings of the European Conference on Machine Learning, 2000.
[17] S. Abe and T. Inoue, "Fuzzy support vector machines for multiclass
problems," in European Symposium on Artificial Neural Networks, 2002,
pp. 113-118.
[18] C.-C. Chang and C.-J. Lin, "LIBSVM: a library for support vector
machines," 2001, http://www.csie.ntu.edu.tw/ cjlin/libsvm.
[1] A. Agresti, Categorical Data Analysis, 2nd version. John Wiley and
Suns Publications, 2002.
[2] J. DelCoz, G. Bayn, J. Dez, O. Luaces, A. Bahamonde, and C. Saudo,
"Trait selection for assessing beef meat quality using non-linear svm,"
in Proceedings of the conference on Neural Information Processing
Systems, Vancouver, Canada, 2004, pp. 321-328.
[3] R. Herbrich, "The trueskill(tm) ranking system," 2005,
http://www.research.microsoft.com/mlp/trueskill/.
[4] S. Kramer, G. Widmer, B. Pfahringer, and M. Degroeve, "Prediction
of ordinal classes using regression trees," Fundamenta Informaticae,
vol. 24, pp. 1-15, 2000.
[5] K. Cao-Van, "Supervised ranking, from semantics to algorithms," Ph.D.
dissertation, Ghent University, Belgium, 2003.
[6] R. Potharst and J. Bioch, "Decision trees for ordinal classification,"
Intelligent Data Processing, vol. 4, no. 2, 2000.
[7] W. Cohen, R. Schapire, and Y. Singer, "Learning to order things," in
Advances in Neural Information Processing Systems 10, 1998.
[8] K. Crammer and Y. Singer, "Pranking with ranking," in Proceedings of
the conference on Neural Information Processing Systems (NIPS), 2001.
[9] E. Harrington, "Online ranking/collaborative filtering using the perceptron
algorithm," in Proceedings of the 20th International Conference on
Machine Learning, Washington, USA, 2003.
[10] R. Herbrich, T. Graepel, and K. Obermayer, "Large margin rank boundaries
for ordinal regression," in Advances in Large Margin Classifiers,
2000, pp. 115-132.
[11] J. Shawe-Taylor and N. Cristianini, Kernel Methods for Pattern Analysis.
Cambridge University Press, 2004.
[12] A. Shashua and A. Levin, "Ranking with the large margin principle:
two approaches," Advances in Neural Information Processing Systems,
vol. 15, 2003.
[13] W. Chu and S. Keerthi, "New approaches to support vector ordinal
regression," in Proceedings of the 22th International Conference of
Machine Learning, Bonn, Germany, 2005, pp. 321-328.
[14] T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical
Learning. Springer, 2001.
[15] B. Scholkopf and A. Smola, Learning with Kernels, Support Vector
Machines, Regularisation, Optimization and beyond. The Mit Press,
2002.
[16] E. Frank and M. Hall, "A simple approach to ordinal classification," in
Proceedings of the European Conference on Machine Learning, 2000.
[17] S. Abe and T. Inoue, "Fuzzy support vector machines for multiclass
problems," in European Symposium on Artificial Neural Networks, 2002,
pp. 113-118.
[18] C.-C. Chang and C.-J. Lin, "LIBSVM: a library for support vector
machines," 2001, http://www.csie.ntu.edu.tw/ cjlin/libsvm.
@article{"International Journal of Engineering, Mathematical and Physical Sciences:50135", author = "Willem Waegeman and Luc Boullart", title = "An ensemble of Weighted Support Vector Machines for Ordinal Regression", abstract = "Instead of traditional (nominal) classification we investigate
the subject of ordinal classification or ranking. An enhanced
method based on an ensemble of Support Vector Machines (SVM-s)
is proposed. Each binary classifier is trained with specific weights
for each object in the training data set. Experiments on benchmark
datasets and synthetic data indicate that the performance of our
approach is comparable to state of the art kernel methods for
ordinal regression. The ensemble method, which is straightforward
to implement, provides a very good sensitivity-specificity trade-off
for the highest and lowest rank.", keywords = "Ordinal regression, support vector machines, ensemblelearning.", volume = "1", number = "12", pages = "570-5", }