Artificial Neural Network Development by means of Genetic Programming with Graph Codification
The development of Artificial Neural Networks
(ANNs) is usually a slow process in which the human expert has to
test several architectures until he finds the one that achieves best
results to solve a certain problem. This work presents a new
technique that uses Genetic Programming (GP) for automatically
generating ANNs. To do this, the GP algorithm had to be changed in
order to work with graph structures, so ANNs can be developed. This
technique also allows the obtaining of simplified networks that solve
the problem with a small group of neurons. In order to measure the
performance of the system and to compare the results with other
ANN development methods by means of Evolutionary Computation
(EC) techniques, several tests were performed with problems based
on some of the most used test databases. The results of those
comparisons show that the system achieves good results comparable
with the already existing techniques and, in most of the cases, they
worked better than those techniques.
[1] S. Haykin, Neural Networks (2nd ed.), Englewood Cliffs, NJ: Prentice
Hall, 1999.
[2] J. R. Rabu├▒al and J. Dorado, (eds.) Artificial Neural Networks in Real-
Life Applications, Idea Group Inc, 2005.
[3] J. R. Koza, Genetic Programming: On the Programming of Computers
by Means of Natural Selection, Cambridge, MA, MIT Press, 1992.
[4] J.R. Rabu├▒al, J. Dorado, A. Pazos, J. Pereira and D. Rivero, "A New
Approach to the Extraction of ANN Rules and to Their Generalization
Capacity Through GP". Neural Computation, vol. 16, n. 7. 2004. pp.
1483-1523.
[5] M. Bot, "Application of Genetic Programming to Induction of Linear
Classification Trees", Final Term Project Report, Vrije Universiteit,
Amsterdam, 1999.
[6] J. R. Rabu├▒al, J. Dorado, J. Puertas, A. Pazos, A. Santos and D. Rivero,
"Prediction and Modelling of the Rainfall-Runoff Transformation of a
Typical Urban Basin using ANN and GP", Applied Artificial
Intelligence, 2003.
[7] R. S. Sutton, "Two problems with backpropagation and other steepestdescent
learning procedure for networks", Proc. 8th Annual Conf.
Cognitive Science Society, Hillsdale, NJ: Erlbaum, 1986, pp. 823-831.
[8] D. J. Janson and J. F. Frenzel, "Training product unit neural networks
with genetic algorithms", IEEE Expert, vol. 8, 1993, pp. 26-33.
[9] G. W. Greenwood, "Training partially recurrent neural networks using
evolutionary strategies", IEEE Trans. Speech Audio Processing, vol. 5,
1997, pp. 192-194.
[10] E. Alba, J. F. Aldana and J. M. Troya, "Fully automatic ANN design: A
genetic approach", Proc. Int. Workshop Artificial Neural Networks
(IWANN-93), Lecture Notes in Computer Science, vol. 686. Berlin,
Germany: Springer-Verlag, 1993, pp. 399-404.
[11] H. Kitano, "Designing neural networks using genetic algorithms with
graph generation system", Complex Systems, vol. 4, 1990, pp. 461-476.
[12] X. Yao and Y. Liu, "Toward designing artificial neural networks by
evolution", Appl. Math. Computation, vol. 91, no. 1, 1998, pp. 83-90.
[13] S. A. Harp, T. Samad and A. Guha, "Toward the genetic synthesis of
neural networks", Proc. 3rd Int. Conf. Genetic Algorithms and Their
Applications, J.D. Schafer, Ed. San Mateo, CA: Morgan Kaufmann,
1989, pp. 360-369.
[14] S. Nolfi and D. Parisi, "Evolution of Artificial Neural Networks",
Handbook of brain theory and neural networks, Second Edition,
Cambridge, MA: MIT Press, 2002, pp. 418-421.
[15] P. Turney, D. Whitley and R. Anderson, "Special issue on the
baldwinian effect", Evolutionary Computation, vol. 4, no. 3, 1996, pp.
213-329.
[16] A. Zomorodian, 1995. "Context-free Language Induction by Evolution
of Deterministic Push-down Automata Using Genetic Programming", in
Working Notes of the Genetic Programming Symposium, AAAI-95, Eric
Siegel and John Koza, chairs. AAAI Press. 1995.
[17] Z. Fan, K. Seo, R. C. Rosenberg, J. Hu and E. D. Goodman, "Exploring
Multiple Design Topologies Using Genetic Programming And Bond
Graphs". GECCO 2002: Proceedings of the Genetic and Evolutionary
Computation Conference. Springer-Verlag. 2002, pp. 1073-1080
[18] Z. Fan, K. Seo, J. Hu, R. C. Rosenberg and E. D. Goodman, "System-
Level Synthesis of MEMS via Genetic Programming and Bond Graphs",
Genetic and Evolutionary Computation -- GECCO-2003. Vol. 2724.
2003, pp. 2058-2071.
[19] F. Gruau, "Genetic micro programming of neural networks", in Kinnear,
Jr., K. E., editor, Advances in Genetic Programming, chapter 24, MIT
Press, 1994, pp. 495-518.
[20] S. Luke and L. Spector, "Evolving Graphs and Networks with Edge
encoding: Preliminary Report". In Late Breaking Papers at the Genetic
Programming 1996 Conference (GP96). J. Koza, ed. Stanford: Stanford
Bookstore, 1996, pp. 117-124.
[21] A. Teller, "Evolving Programmers: The Co-evolution of Intelligent
Recombination Operators", in Advances in Genetic Programming II, P.
Angeline and K. Kinnear, editors. Cambridge: MIT Press., 1996.
[22] W. Kantschik, P. Dittrich, M. Brameier and W. Banzhaf,
"MetaEvolution in Graph GP", Proceedings of EuroGP'99, LNCS, Vol.
1598. SpringerVerlag, 1999, pp. 15-28.
[23] R. Poli "Evolution of Graph-like Programs with Parallel Distributed
Genetic Programming", Genetic Algorithms: Proceedings of the Seventh
International Conference, 1997.
[24] W. Kantschik, W. Banzhaf, "Linear-Graph GP - A new GP Structure",
in Proceedings of the 4th European Conference on Genetic
Programming, EuroGP 2002, 2002.
[25] A. Teller A. and M. Veloso, "Internal reinforcement in a connectionist
genetic programming approach", Artificial Intelligence. Vol. 120, N. 2,
2000, pp. 165-198.
[26] D. J. Montana, "Strongly typed genetic programming", Evolutionary
Computation, Vol. 3, No. 2, 1995, pp. 199-200.
[27] C. J. Mertz and P. M. Murphy, UCI repository of machine learning
databases. http://www-old.ics.uci.edu/pub/machine-learning-databases,
2002
[28] E. Cant├║-Paz and C. Kamath, "An Empirical Comparison of
Combinations of Evolutionary Algorithms and Neural Networks for
Classification Problems", IEEE Transactions on systems, Man and
Cybernetics - Part B: Cybernetics, 2005, pp. 915-927.
[29] T. G. Dietterich, "Approximate statistical tests for comparing supervised
classification learning algorithms", Neural Computation, Vol. 10, No. 7,
1998, pp. 1895-1924.
[1] S. Haykin, Neural Networks (2nd ed.), Englewood Cliffs, NJ: Prentice
Hall, 1999.
[2] J. R. Rabu├▒al and J. Dorado, (eds.) Artificial Neural Networks in Real-
Life Applications, Idea Group Inc, 2005.
[3] J. R. Koza, Genetic Programming: On the Programming of Computers
by Means of Natural Selection, Cambridge, MA, MIT Press, 1992.
[4] J.R. Rabu├▒al, J. Dorado, A. Pazos, J. Pereira and D. Rivero, "A New
Approach to the Extraction of ANN Rules and to Their Generalization
Capacity Through GP". Neural Computation, vol. 16, n. 7. 2004. pp.
1483-1523.
[5] M. Bot, "Application of Genetic Programming to Induction of Linear
Classification Trees", Final Term Project Report, Vrije Universiteit,
Amsterdam, 1999.
[6] J. R. Rabu├▒al, J. Dorado, J. Puertas, A. Pazos, A. Santos and D. Rivero,
"Prediction and Modelling of the Rainfall-Runoff Transformation of a
Typical Urban Basin using ANN and GP", Applied Artificial
Intelligence, 2003.
[7] R. S. Sutton, "Two problems with backpropagation and other steepestdescent
learning procedure for networks", Proc. 8th Annual Conf.
Cognitive Science Society, Hillsdale, NJ: Erlbaum, 1986, pp. 823-831.
[8] D. J. Janson and J. F. Frenzel, "Training product unit neural networks
with genetic algorithms", IEEE Expert, vol. 8, 1993, pp. 26-33.
[9] G. W. Greenwood, "Training partially recurrent neural networks using
evolutionary strategies", IEEE Trans. Speech Audio Processing, vol. 5,
1997, pp. 192-194.
[10] E. Alba, J. F. Aldana and J. M. Troya, "Fully automatic ANN design: A
genetic approach", Proc. Int. Workshop Artificial Neural Networks
(IWANN-93), Lecture Notes in Computer Science, vol. 686. Berlin,
Germany: Springer-Verlag, 1993, pp. 399-404.
[11] H. Kitano, "Designing neural networks using genetic algorithms with
graph generation system", Complex Systems, vol. 4, 1990, pp. 461-476.
[12] X. Yao and Y. Liu, "Toward designing artificial neural networks by
evolution", Appl. Math. Computation, vol. 91, no. 1, 1998, pp. 83-90.
[13] S. A. Harp, T. Samad and A. Guha, "Toward the genetic synthesis of
neural networks", Proc. 3rd Int. Conf. Genetic Algorithms and Their
Applications, J.D. Schafer, Ed. San Mateo, CA: Morgan Kaufmann,
1989, pp. 360-369.
[14] S. Nolfi and D. Parisi, "Evolution of Artificial Neural Networks",
Handbook of brain theory and neural networks, Second Edition,
Cambridge, MA: MIT Press, 2002, pp. 418-421.
[15] P. Turney, D. Whitley and R. Anderson, "Special issue on the
baldwinian effect", Evolutionary Computation, vol. 4, no. 3, 1996, pp.
213-329.
[16] A. Zomorodian, 1995. "Context-free Language Induction by Evolution
of Deterministic Push-down Automata Using Genetic Programming", in
Working Notes of the Genetic Programming Symposium, AAAI-95, Eric
Siegel and John Koza, chairs. AAAI Press. 1995.
[17] Z. Fan, K. Seo, R. C. Rosenberg, J. Hu and E. D. Goodman, "Exploring
Multiple Design Topologies Using Genetic Programming And Bond
Graphs". GECCO 2002: Proceedings of the Genetic and Evolutionary
Computation Conference. Springer-Verlag. 2002, pp. 1073-1080
[18] Z. Fan, K. Seo, J. Hu, R. C. Rosenberg and E. D. Goodman, "System-
Level Synthesis of MEMS via Genetic Programming and Bond Graphs",
Genetic and Evolutionary Computation -- GECCO-2003. Vol. 2724.
2003, pp. 2058-2071.
[19] F. Gruau, "Genetic micro programming of neural networks", in Kinnear,
Jr., K. E., editor, Advances in Genetic Programming, chapter 24, MIT
Press, 1994, pp. 495-518.
[20] S. Luke and L. Spector, "Evolving Graphs and Networks with Edge
encoding: Preliminary Report". In Late Breaking Papers at the Genetic
Programming 1996 Conference (GP96). J. Koza, ed. Stanford: Stanford
Bookstore, 1996, pp. 117-124.
[21] A. Teller, "Evolving Programmers: The Co-evolution of Intelligent
Recombination Operators", in Advances in Genetic Programming II, P.
Angeline and K. Kinnear, editors. Cambridge: MIT Press., 1996.
[22] W. Kantschik, P. Dittrich, M. Brameier and W. Banzhaf,
"MetaEvolution in Graph GP", Proceedings of EuroGP'99, LNCS, Vol.
1598. SpringerVerlag, 1999, pp. 15-28.
[23] R. Poli "Evolution of Graph-like Programs with Parallel Distributed
Genetic Programming", Genetic Algorithms: Proceedings of the Seventh
International Conference, 1997.
[24] W. Kantschik, W. Banzhaf, "Linear-Graph GP - A new GP Structure",
in Proceedings of the 4th European Conference on Genetic
Programming, EuroGP 2002, 2002.
[25] A. Teller A. and M. Veloso, "Internal reinforcement in a connectionist
genetic programming approach", Artificial Intelligence. Vol. 120, N. 2,
2000, pp. 165-198.
[26] D. J. Montana, "Strongly typed genetic programming", Evolutionary
Computation, Vol. 3, No. 2, 1995, pp. 199-200.
[27] C. J. Mertz and P. M. Murphy, UCI repository of machine learning
databases. http://www-old.ics.uci.edu/pub/machine-learning-databases,
2002
[28] E. Cant├║-Paz and C. Kamath, "An Empirical Comparison of
Combinations of Evolutionary Algorithms and Neural Networks for
Classification Problems", IEEE Transactions on systems, Man and
Cybernetics - Part B: Cybernetics, 2005, pp. 915-927.
[29] T. G. Dietterich, "Approximate statistical tests for comparing supervised
classification learning algorithms", Neural Computation, Vol. 10, No. 7,
1998, pp. 1895-1924.
@article{"International Journal of Information, Control and Computer Sciences:60382", author = "Daniel Rivero and Julián Dorado and Juan R. Rabuñal and Alejandro Pazos and Javier Pereira", title = "Artificial Neural Network Development by means of Genetic Programming with Graph Codification", abstract = "The development of Artificial Neural Networks
(ANNs) is usually a slow process in which the human expert has to
test several architectures until he finds the one that achieves best
results to solve a certain problem. This work presents a new
technique that uses Genetic Programming (GP) for automatically
generating ANNs. To do this, the GP algorithm had to be changed in
order to work with graph structures, so ANNs can be developed. This
technique also allows the obtaining of simplified networks that solve
the problem with a small group of neurons. In order to measure the
performance of the system and to compare the results with other
ANN development methods by means of Evolutionary Computation
(EC) techniques, several tests were performed with problems based
on some of the most used test databases. The results of those
comparisons show that the system achieves good results comparable
with the already existing techniques and, in most of the cases, they
worked better than those techniques.", keywords = "Artificial Neural Networks, Evolutionary
Computation, Genetic Programming.", volume = "2", number = "9", pages = "3152-6", }
