Motivated Support Vector Regression using Structural Prior Knowledge
It-s known that incorporating prior knowledge into support
vector regression (SVR) can help to improve the approximation
performance. Most of researches are concerned with the incorporation
of knowledge in the form of numerical relationships. Little work,
however, has been done to incorporate the prior knowledge on the
structural relationships among the variables (referred as to Structural
Prior Knowledge, SPK). This paper explores the incorporation of SPK
in SVR by constructing appropriate admissible support vector kernel
(SV kernel) based on the properties of reproducing kernel (R.K).
Three-levels specifications of SPK are studied with the corresponding
sub-levels of prior knowledge that can be considered for the method.
These include Hierarchical SPK (HSPK), Interactional SPK (ISPK)
consisting of independence, global and local interaction, Functional
SPK (FSPK) composed of exterior-FSPK and interior-FSPK. A
convenient tool for describing the SPK, namely Description Matrix
of SPK is introduced. Subsequently, a new SVR, namely Motivated
Support Vector Regression (MSVR) whose structure is motivated
in part by SPK, is proposed. Synthetic examples show that it is
possible to incorporate a wide variety of SPK and helpful to improve
the approximation performance in complex cases. The benefits of
MSVR are finally shown on a real-life military application, Air-toground
battle simulation, which shows great potential for MSVR to
the complex military applications.
[1] V. Vapnik, The Nature of Statistical Learning Theory. New York:
Springer-Verlag, 1995.
[2] G. Bloch, F. Lauer, G. Colin, and Y. Chamaillard, "Support vector regression
from simulation data and few experimental samples," Information
Sciences, vol. 178, pp. 3813-3827, 2008.
[3] J.-B. Gao, S. R. Gunn, and C. J. Harris, "Mean field method for the
support vector machine regression," Neurocomputing, vol. 50, pp. 391-
405, 2003.
[4] M. A. Mohandes, T. O. Halawani, S. Rehman, and A. A. Hussain, "Support
vector machines for wind speed prediction," Renewable Energy,
vol. 29, no. 6, pp. 939-947, 2004.
[5] W.-W. He, Z.-Z. Wang, and H. Jiang, "Model optimizing and feature
selecting for support vector regression in time series forecasting,"
Neurocomputing, vol. 73, no. 3, pp. 600-611, 2008.
[6] F. Pan, P. Zhu, and Y. Zhang, "Metamodel-based lightweight design of
b-pillar with twb structure via support vector regression," Computers
and Structures, vol. 88, pp. 36-44, 2010.
[7] A. J. Smola, T. Friess, and K.-R. Mller, "Semiparametric support vector
and linear programming machines," Advances in neural information
processing systems, vol. 11, pp. 585-591, 1998.
[8] O. L. Mangasarian, J. Shavlik, and E. W. Wild, "Knowledge-based
kernel approximation," Journal of Machine Learning Research, vol. 5,
pp. 1127-1141, 2004.
[9] M. Lzaro, F. Prez-Cruz, and A. Arts-Rodriguez, "Learning a function
and its derivative forcing the support vector expansion," IEEE Signal
Processing Letters, vol. 12, pp. 194-197, 2005.
[10] F. Lauer and G. Bloch, "Incorporating prior knowledge in support vector
regression," Mach. Learn., vol. 70, pp. 89-118, 2008.
[11] O. L. Mangasarian and E. W. wild, "Nonlinear knowledge in kernel
approximation," IEEE Transactions on Neural Networks, vol. 18, pp.
300-306, 2007.
[12] B. Sch¨olkopf, P. Simard, A. J. Smola, and V. Vapnik, "Prior knowledge
in support vector kernels," in Advanced in Kernel Method-Support Vector
Learning, C. B. A. S. Sch¨olkopf, B., Ed. Cambridge, England: MIT
Press, 1998.
[13] A. J. Smola and B. Sch¨olkopf, "A tutorial on support vector regression,"
Statistics and Computing, vol. 14, no. 3, pp. 199-222, 2004.
[14] P. K. Davis and J. H. Bigelow, "Motivated metamodels: Synthesis of
cause-effect reasoning and statistical metamodeling," RAND, Tech. Rep.
MR-1570, 2003.
[15] W. Zhang, X. Zhao, Y.-F. Zhu, and X.-J. Zhang, "A new composition
method of admissible support vector kernel based on reproducing
kernel," in International Conference on Computer and Applied Mathematics,
Rio de Janeiro, March 2010.
[16] N. Aronszajn, "Theory of reproducing kernels," Transactions of the
American Mathematical Society, vol. 68, no. 3, pp. 337-404, 1950.
[17] D. Haussler, "Convolution kernels on discrete structures," University of
California, Sana Cruz, CA,, Tech. Rep. UCSC-CRL-99-10, 1999.
[18] G. F. Smits and E. M. Jondaan, "Improved svm regression using mixtures
of kernels," in Proceeding of the 2002 international joint conference on
neural networks, vol. 3. Honolulu, Hawaii, USA: IEEE, 2002.
[19] Y. Tan and J. Wang, "A support vector machine with a hybrid kernel
and minimal vapnik-chervonenkis dimension," IEEE Transactions on
Knowledge and Data Engineering, vol. 16, pp. 385-395, 2004.
[20] B. Sch¨olkopf, K.-K. Sung, C. J. Burges, F. Girosi, P. Niyogi, T. Poggio,
and V. Vapnik, "Comparing support vector machines with gaussian
kernels to radial basis function classifiers," IEEE Transactions on Signal
Processing, vol. 45, pp. 2758-2765, 1997.
[21] G. Wahba, Spline Models for Observational Data, SIAM, Philadephia,
1990.
[22] J. St¨ockler, "Multivariate bernoulli splines and the periodic interpolation
problem," Constr. Approx., vol. 7, pp. 105-120, 1991.
[23] G. Wahba, "Support vector machines,reproducing kernel hilbert spaces
and randomized gacv," in Advances in Kernel Methods - Support Vector
Learning, B. Sch¨olkopf, C. J. Burges, and A. J. Smola, Eds. Cambridge,
England: MIT Press, 1999, pp. 69-88.
[24] A. Berlinet and C. Thomas-Agnan, Reproducing Kernel Hilbert Spaces
in Probability and Statistics. Boston, Dordrecht, London: Kluwer
Academic Publishers Group, 2003.
[25] R. Schaback, "A unified theory of radial basis functions native hilbert
spaces for radial basis functions ii," Journal of Computational and
Applied Mathematics, vol. 121, pp. 165-177, 2000.
[26] J. Mercer, Ed., Functions of positive and negative type and their
connection with the theory of integral equations, ser. Philosophical
Transactions of the Royal Society, London, 1909, vol. A, 209.
[27] P. K. Davis, "Introduction to multiresolution, multiperspective modeling
(mrmpm) and exploratory analysis," RAND, Tech. Rep. WR-224, 2005.
[28] R. Horn, Topics in Matrix Analysis. Cambridge, MA: Cambridge
University Press, 1994.
[29] C. J. Burges, "A tutorial on support vector machines for pattern recognition,"
Data Mining and Knowledge Discovery, vol. 2, pp. 121-167,
1998.
[30] Y.-J. Park, "Application of genetic algorithms in response surface
optimization problems," Doctor of Philosophy, Arizona State University,
December 2003.
[31] S. S. Chaudhry and W. Luo, "Application of genetic algorithms in production
and operations management: A review," International Journal
of Production Research, vol. 43, pp. 4083-4101, 2005.
[32] C. R. Houck, J. A. Joines, and M. G. Kay, "A genetic algorithm for
function optimization: A matlab implementation," Tech. Rep. NCSU-IE
TR 95-09, 1995.
[33] B. P. Zeigler, H. Praehofer, and T. G. Kim, Theory of Modeling
and Simulation: Integrating Discrete Event and Continuous Complex
Dynamic Systems, second edition ed. CA, USA: Academic Press, 2000.
[1] V. Vapnik, The Nature of Statistical Learning Theory. New York:
Springer-Verlag, 1995.
[2] G. Bloch, F. Lauer, G. Colin, and Y. Chamaillard, "Support vector regression
from simulation data and few experimental samples," Information
Sciences, vol. 178, pp. 3813-3827, 2008.
[3] J.-B. Gao, S. R. Gunn, and C. J. Harris, "Mean field method for the
support vector machine regression," Neurocomputing, vol. 50, pp. 391-
405, 2003.
[4] M. A. Mohandes, T. O. Halawani, S. Rehman, and A. A. Hussain, "Support
vector machines for wind speed prediction," Renewable Energy,
vol. 29, no. 6, pp. 939-947, 2004.
[5] W.-W. He, Z.-Z. Wang, and H. Jiang, "Model optimizing and feature
selecting for support vector regression in time series forecasting,"
Neurocomputing, vol. 73, no. 3, pp. 600-611, 2008.
[6] F. Pan, P. Zhu, and Y. Zhang, "Metamodel-based lightweight design of
b-pillar with twb structure via support vector regression," Computers
and Structures, vol. 88, pp. 36-44, 2010.
[7] A. J. Smola, T. Friess, and K.-R. Mller, "Semiparametric support vector
and linear programming machines," Advances in neural information
processing systems, vol. 11, pp. 585-591, 1998.
[8] O. L. Mangasarian, J. Shavlik, and E. W. Wild, "Knowledge-based
kernel approximation," Journal of Machine Learning Research, vol. 5,
pp. 1127-1141, 2004.
[9] M. Lzaro, F. Prez-Cruz, and A. Arts-Rodriguez, "Learning a function
and its derivative forcing the support vector expansion," IEEE Signal
Processing Letters, vol. 12, pp. 194-197, 2005.
[10] F. Lauer and G. Bloch, "Incorporating prior knowledge in support vector
regression," Mach. Learn., vol. 70, pp. 89-118, 2008.
[11] O. L. Mangasarian and E. W. wild, "Nonlinear knowledge in kernel
approximation," IEEE Transactions on Neural Networks, vol. 18, pp.
300-306, 2007.
[12] B. Sch¨olkopf, P. Simard, A. J. Smola, and V. Vapnik, "Prior knowledge
in support vector kernels," in Advanced in Kernel Method-Support Vector
Learning, C. B. A. S. Sch¨olkopf, B., Ed. Cambridge, England: MIT
Press, 1998.
[13] A. J. Smola and B. Sch¨olkopf, "A tutorial on support vector regression,"
Statistics and Computing, vol. 14, no. 3, pp. 199-222, 2004.
[14] P. K. Davis and J. H. Bigelow, "Motivated metamodels: Synthesis of
cause-effect reasoning and statistical metamodeling," RAND, Tech. Rep.
MR-1570, 2003.
[15] W. Zhang, X. Zhao, Y.-F. Zhu, and X.-J. Zhang, "A new composition
method of admissible support vector kernel based on reproducing
kernel," in International Conference on Computer and Applied Mathematics,
Rio de Janeiro, March 2010.
[16] N. Aronszajn, "Theory of reproducing kernels," Transactions of the
American Mathematical Society, vol. 68, no. 3, pp. 337-404, 1950.
[17] D. Haussler, "Convolution kernels on discrete structures," University of
California, Sana Cruz, CA,, Tech. Rep. UCSC-CRL-99-10, 1999.
[18] G. F. Smits and E. M. Jondaan, "Improved svm regression using mixtures
of kernels," in Proceeding of the 2002 international joint conference on
neural networks, vol. 3. Honolulu, Hawaii, USA: IEEE, 2002.
[19] Y. Tan and J. Wang, "A support vector machine with a hybrid kernel
and minimal vapnik-chervonenkis dimension," IEEE Transactions on
Knowledge and Data Engineering, vol. 16, pp. 385-395, 2004.
[20] B. Sch¨olkopf, K.-K. Sung, C. J. Burges, F. Girosi, P. Niyogi, T. Poggio,
and V. Vapnik, "Comparing support vector machines with gaussian
kernels to radial basis function classifiers," IEEE Transactions on Signal
Processing, vol. 45, pp. 2758-2765, 1997.
[21] G. Wahba, Spline Models for Observational Data, SIAM, Philadephia,
1990.
[22] J. St¨ockler, "Multivariate bernoulli splines and the periodic interpolation
problem," Constr. Approx., vol. 7, pp. 105-120, 1991.
[23] G. Wahba, "Support vector machines,reproducing kernel hilbert spaces
and randomized gacv," in Advances in Kernel Methods - Support Vector
Learning, B. Sch¨olkopf, C. J. Burges, and A. J. Smola, Eds. Cambridge,
England: MIT Press, 1999, pp. 69-88.
[24] A. Berlinet and C. Thomas-Agnan, Reproducing Kernel Hilbert Spaces
in Probability and Statistics. Boston, Dordrecht, London: Kluwer
Academic Publishers Group, 2003.
[25] R. Schaback, "A unified theory of radial basis functions native hilbert
spaces for radial basis functions ii," Journal of Computational and
Applied Mathematics, vol. 121, pp. 165-177, 2000.
[26] J. Mercer, Ed., Functions of positive and negative type and their
connection with the theory of integral equations, ser. Philosophical
Transactions of the Royal Society, London, 1909, vol. A, 209.
[27] P. K. Davis, "Introduction to multiresolution, multiperspective modeling
(mrmpm) and exploratory analysis," RAND, Tech. Rep. WR-224, 2005.
[28] R. Horn, Topics in Matrix Analysis. Cambridge, MA: Cambridge
University Press, 1994.
[29] C. J. Burges, "A tutorial on support vector machines for pattern recognition,"
Data Mining and Knowledge Discovery, vol. 2, pp. 121-167,
1998.
[30] Y.-J. Park, "Application of genetic algorithms in response surface
optimization problems," Doctor of Philosophy, Arizona State University,
December 2003.
[31] S. S. Chaudhry and W. Luo, "Application of genetic algorithms in production
and operations management: A review," International Journal
of Production Research, vol. 43, pp. 4083-4101, 2005.
[32] C. R. Houck, J. A. Joines, and M. G. Kay, "A genetic algorithm for
function optimization: A matlab implementation," Tech. Rep. NCSU-IE
TR 95-09, 1995.
[33] B. P. Zeigler, H. Praehofer, and T. G. Kim, Theory of Modeling
and Simulation: Integrating Discrete Event and Continuous Complex
Dynamic Systems, second edition ed. CA, USA: Academic Press, 2000.
@article{"International Journal of Information, Control and Computer Sciences:49613", author = "Wei Zhang and Yao-Yu Li and Yi-Fan Zhu and Qun Li and Wei-Ping Wang", title = "Motivated Support Vector Regression using Structural Prior Knowledge", abstract = "It-s known that incorporating prior knowledge into support
vector regression (SVR) can help to improve the approximation
performance. Most of researches are concerned with the incorporation
of knowledge in the form of numerical relationships. Little work,
however, has been done to incorporate the prior knowledge on the
structural relationships among the variables (referred as to Structural
Prior Knowledge, SPK). This paper explores the incorporation of SPK
in SVR by constructing appropriate admissible support vector kernel
(SV kernel) based on the properties of reproducing kernel (R.K).
Three-levels specifications of SPK are studied with the corresponding
sub-levels of prior knowledge that can be considered for the method.
These include Hierarchical SPK (HSPK), Interactional SPK (ISPK)
consisting of independence, global and local interaction, Functional
SPK (FSPK) composed of exterior-FSPK and interior-FSPK. A
convenient tool for describing the SPK, namely Description Matrix
of SPK is introduced. Subsequently, a new SVR, namely Motivated
Support Vector Regression (MSVR) whose structure is motivated
in part by SPK, is proposed. Synthetic examples show that it is
possible to incorporate a wide variety of SPK and helpful to improve
the approximation performance in complex cases. The benefits of
MSVR are finally shown on a real-life military application, Air-toground
battle simulation, which shows great potential for MSVR to
the complex military applications.", keywords = "admissible support vector kernel, reproducing kernel,structural prior knowledge, motivated support vector regression", volume = "4", number = "5", pages = "862-10", }
