Extended Least Squares LS–SVM

Among neural models the Support Vector Machine (SVM) solutions are attracting increasing attention, mostly because they eliminate certain crucial questions involved by neural network construction. The main drawback of standard SVM is its high computational complexity, therefore recently a new technique, the Least Squares SVM (LS–SVM) has been introduced. In this paper we present an extended view of the Least Squares Support Vector Regression (LS–SVR), which enables us to develop new formulations and algorithms to this regression technique. Based on manipulating the linear equation set -which embodies all information about the regression in the learning process- some new methods are introduced to simplify the formulations, speed up the calculations and/or provide better results.

Authors:

• József Valyon
• Gábor Horváth

Keywords:

• Function estimation
• Least–Squares Support VectorMachines
• Regression
• System Modeling

References:

[1] V. Vapnik, "The Nature of Statistical Learning Theory", New-York:
Springer-Verlag., 1995
[2] J. A. K. Suykens, V. T. Gestel, J. De Brabanter, B. De Moor, J.
Vandewalle, "Least Squares Support Vector Machines", World
Scientific, 2002
[3] J. A. K. Suykens, L. Lukas, and J. Vandewalle, "Sparse approximation
using least squares support vector machines", IEEE International
Symposium on Circuits and Systems ISCAS'2000, 2000
[4] J. A. K. Suykens, L. Lukas, and J. Vandewalle, "Sparse least squares
support vector machine classifiers", ESANN'2000 European Symposium
on Artificial Neural Networks, 2000, pp. 37-42.
[5] J.A.K. Suykens, J. De Brabanter, L. Lukas, and J. Vandewalle,
"Weighted least squares support vector machines: robustness and sparse
approximation", Neurocomputing, 2002. pp. 85-105
[6] B. Schölkopf and A. Smola: Learning with Kernels. Support Vector
Machines, Regularization, Optimization, and Beyond. The MIT Press,
Cambridge, MA, 2002.
[7] J. Valyon and G. Horv├íth, ÔÇ×A generalized LS-SVM", SYSID'2003
Rotterdam, 2003, pp. 827-832.
[8] J. Valyon and G. Horv├íth, ÔÇ×A Sparse Least Squares Support Vector
Machine Classifier", Proceedings of the International Joint Conference
on Neural Networks IJCNN 2004, 2004, pp. 543-548.
[9] P. W. Holland, and R. E. Welsch, "Robust Regression Using Iteratively
Reweighted Least-Squares," Communications in Statistics: Theory and
Methods, A6, 1977, pp. 813-827.
[10] P. J. Huber, Robust Statistics, Wiley, 1981.
[11] B. Schölkopf, S. Mika, C.J.C. Burges, P. Knirsch, K.-R. M├╝ller, G.
R├ñtsch, and A. Smola, ÔÇ×Input space vs. feature space in kernel-based
methods". IEEE Transactions on Neural Networks, 1999, 10(5), pp.
1000-1017.
[12] G. Baudat and F. Anouar, "Kernel-based methods and function
approximation". In International Joint Conference on Neural Networks,
pages 1244-1249, Washington DC, 2001. July 15-19.
[13] Yuh - Jye Lee and O. L. Mangasarian, "RSVM: Reduced Support
Vector Machines", Proceedings of the First SIAM International
Conference on Data Mining, Chicago, 2001. April 5-7.
[14] W. H. Press, S. A. Teukolsky, W. T. Wetterling and B. P. Flannery ,
"Numerical Recipes in C", Cambridge University Press, Books On-Line,
Available: www.nr.com, 2002
[15] H. Golub and Charles F. Van Loan, Matrix Computations", Gene Johns
Hopkins University Press, 1989