An Evolutionary Statistical Learning Theory

Statistical learning theory was developed by Vapnik. It is a learning theory based on Vapnik-Chervonenkis dimension. It also has been used in learning models as good analytical tools. In general, a learning theory has had several problems. Some of them are local optima and over-fitting problems. As well, statistical learning theory has same problems because the kernel type, kernel parameters, and regularization constant C are determined subjectively by the art of researchers. So, we propose an evolutionary statistical learning theory to settle the problems of original statistical learning theory. Combining evolutionary computing into statistical learning theory, our theory is constructed. We verify improved performances of an evolutionary statistical learning theory using data sets from KDD cup.

Authors:

• Sung-Hae Jun
• Kyung-Whan Oh

Keywords:

• Evolutionary computing
• Local optima
• Over-fitting
• Statistical learning theory

References:

[1] A. Ben-Hur, A. D. Horn, H. Siegelmann, V. Vapnik, "Support Vector
Clustering," Journal of Machine Learning Research 2, 2001, pp.
125-137.
[2] L. Breiman, J. H. Friedman, R. A. Olshen, C. J. Stone, Classification and
Regression Trees, Wadsworth Inc., 1984.
[3] J. Cannady, "Artificial Neural Networks for Misuse Detection. National
Information Systems," Proceedings of Security Conference, 1998.
[4] G. Casella, R. L. Berger, Statistical Inference, Duxbury Press, 1990.
[5] V. Cherkassky, F. Mulier, Learning From Data Concepts, Theory, and
Methods, John Wiley & Sons, 1998.
[6] R. Cooley, B. Mobasher, J. Srivastava, "Web Mining: Information and
Pattern Discovery on the World Wide Web," Proceeding of the 9th IEEE
International Conference on Tools with Artificial Intelligence, 1997.
[7] R. Cooley, P. N. Tan, J. Srivastava, "Discovery of interesting usage
patterns from web data," Technical Report TR 99-022, University of
Minnesota, 1999.
[8] H. Debar, M. Becke, D. Siboni, "A Neural Network Component for an
Intrusion Detection System," Proceedings of the IEEE Computer Society
Symposium on Research in Security and Privacy, 1992, pp. 240-250.
[9] H. Debar, B. Dorizzi, "An Application of a Recurrent Network to an
Intrusion Detection System," Proceedings of the International Joint
Conference on Neural Networks, 1992, pp 78-483.
[10] A. E. Eiben, J. E. Smith, Introduction to Evolutionary Computing,
Springer, 2003.
[11] S. M. Emran, M. Xu, N. Ye, Q. Chen, X. Li, "Probabilistic techniques for
intrusion detection based on computer audit data," IEEE Transactions on
Systems, Man and Cybernetics, Part A, vol.31, 2001, pp.266-274.
[12] D. Fisher, K. Hildrum, J. Hong, M. Newman, M. Thomas, R. Vuduc,
"SWAMI: A Frame-work for Collaborative Filtering Algorithm
Development and Evaluation," Proceeding of SIGIR 2000, ACM Press,
2000.
[13] D. B. Fogel, Evolutionary Computation, IEEE Press, 1995.
[14] L. J. Fogel, A. J. Owens, M. J. Walsh, Artificial Intelligence through
Simulated Evolution, Wiley, Chichester, UK, 1996.
[15] A. K. Ghosh, Learning Program Behavior Profiles for Intrusion
Detection, USENIX, 1999.
[16] J. W. Haines, R. P. Lippmann, D. J. Fried, M. A. Zissman, E. Tran, S. B.
Boswell, "1999 DARPA Intrusion Detection Evaluation: Design and
Procedures," Technical Report 1062, Lincoln Laboratory, MIT, 2001.
[17] S. Haykin, Neural Networks, Prentice Hall, 1999.
[18] S. Huet, A. Bouvier, M. A. Poursat, E. Jolivet, Statistical Tools for
Nonlinear Regression, Springer Series in Statistics, Springer, 2003.
[19] S. H. Jun, "Hybrid Statistical Learning Model for Intrusion Detection of
Networks," The KIPS Transaction: Part C, vol. 10-C, no. 6, 2003, pp.
705-710.
[20] S. H. Jun, "Web Usage Mining Using Support Vector Machine," Lecture
Note in Computer Science, vol. 3512, 2005, pp. 349-356.
[21] S. Kumar, E. H. Spafford, "An Application of Pattern Matching in
Intrusion Detection," Technical Report CSD-TR-94-013, Purdue
University, 1994.
[22] W. Lee, S. J. Stolfo, K. W. Mok, "A data mining framework for building
intrusion detection models," Proceedings of the 1999 IEEE Symposium
on Security and Privacy, 1999, pp.120-132.
[23] R. J. A. Little, D. B. Rubin, Statistical Analysis with Missing Data, Wiley
Inter-Science, 2002.
[24] B. Liu, "Fuzzy Random Chance-Constrained Programming," IEEE
Transactions on Fuzzy Systems, vol. 9, Issue 5, 2001, pp. 713-720.
[25] J. Luo, S. M. Bridges, "Mining Fuzzy Association Rules and Fuzzy
Frequency Episodes for Intrusion Detection," International Journal of
Intelligent Systems, John Wiley & Sons, 2000, pp. 687-703.
[26] G. Mclachlan, D. Peel, Finite Mixture Models, John Wiley & Sons, Inc.,
2000.
[27] T. M. Mitchell, Machine Learning, McGraw-Hill, 1997.
[28] T. M. Mitchell, An introduction to Genetic Algorithms, MIT Press, 1998.
[29] S. Mukkamala, G. Janoski, A. Sung, "Intrusion Detection Using Neural
Networks and Support Vector Machines," Proceedings of International
Symposium on Applications and the Internet Technology, 2000, pp.
209-216
[30] R. H. Myers, Classical and Modern Regression with Applications,
Duxbury Press, 1990.
[31] A. T. Quang, Q. L. Zhang, X. Li, "Evolving Support Vector Machine
Parameters," Proceedings of the First International Conference on
Machine Learning and Cybernetics, 2002, pp. 548-551.
[32] J. Ryan, M. J. Lin, R. Miikkulainen, "Intrusion Detection with Neural
Networks," Advances in Neural Information Processing Systems 10,
Cambridge, MA: MIT Press, 1998.
[33] A. J. Smola, Regression estimation with support vector learning
machines, Master-s thesis, Technische University, 1996.
[34] V. Vapnik, Statistical Learning Theory, John Wiley & Sons, Inc., 1998.
[35] X. Yao, "Evolving Artificial Neural Networks," Proceedings of the IEEE,
vol. 87, Issue 9, 1999, pp. 1423-1447.
[36] http://www.ecn.purdue.edu/KDDCUP
[37] http://www.ll.mit.edu/IST/ideval/data