Improved Back Propagation Algorithm to Avoid Local Minima in Multiplicative Neuron Model

The back propagation algorithm calculates the weight changes of artificial neural networks, and a common approach is to use a training algorithm consisting of a learning rate and a momentum factor. The major drawbacks of above learning algorithm are the problems of local minima and slow convergence speeds. The addition of an extra term, called a proportional factor reduces the convergence of the back propagation algorithm. We have applied the three term back propagation to multiplicative neural network learning. The algorithm is tested on XOR and parity problem and compared with the standard back propagation training algorithm.

Authors:

• Kavita Burse
• Manish Manoria
• Vishnu P. S. Kirar

Keywords:

• Three term back propagation
• multiplicative neuralnetwork
• proportional factor
• local minima.

References:

[1] Y. H. Zweiri, J. F. Whidborne, K Althoefer and L. D. Seneviratne, "A
three term back propagation algorithm," Neurocomputing, vol. 50, pp.
305-318, 2003.
[2] R. J. Edward: An Introduction to Neural Networks. A White paper.
United States of America, Visual Numerics Inc. 2004.
[3] Y. F. Yam and T.W.S. Chow, "Extended back propagation algorithm,"
Electronics Letters, vol. 29(19), pp. 1701-1702, 1993.
[4] B. K. Verma and J. J. Mulawka, "A modified back propagation
algorithm," in Proc. IEEE World Congress on Computational
Intelligence, pp. 840-844, 1994.
[5] G. P. Drago, M. Morando and S. Ridella," An adaptive momentum back
propagation, Neural Computing and Application, vol. 3, pp. 213-221,
1995.
[6] Y. Q. Chen, T. Yin and H. A. Babri, "A stochastic back propagation
algorithm for training neural networks," in proc. International
Conference on Information, Communications and Signal Processing, pp.
703-707, 9-12 September, 1997, Singapore.
[7] S. C. Ng, S.H. Leung and A. Luk, "Fast convergent generalized back
propagation algorithm with constant learning rate", Neural Processing
Letters, vol. 9, pp. 13-23, 1999.
[8] J. W. Wen, J. L. Zhao, S. W. Luo and Z. Han, "The improvements of BP
neural network learning algorithm," in Proc. of ICSP2000, pp. 1647-
1649, 2000.
[9] X. G. Wang, Z. Tang, H. Tamura, M. Ishii and W. D. Sun, "An
improved back propagation algorithm to avoid the local minima
problem," Neurocomputing, vol. 56, pp. 455-460, 2004.
[10] C. H. Wang, C.H. Kao, and W. H. Lee, "A new interactive model for
improving the learning performance of back propagation neural
network," Automation in Construction, vol. 16, no. 6, pp.745-758, 2007.
[11] B. Mel, "Information processing in dendritic trees," Neural Computing,
vol.6, pp.1031-1085, 1994.
[12] B. Mel, "Information processing in dendritic trees," Neural Computing,
vol.6, pp.1031-1085, 1994.
[13] R. N. Yadav, P. K. Kalra and J. John, "Time series prediction with
single multiplicative neuron model," Applied soft computing, vol. 7 pp
1157-1163, 2007.
[14] R. N. Yadav, V. Singh and P. K. Kalra, "Classification using single
neuron," in Proc. IEEE Int. Conf. on Industrial Informatics, pp. 124-
129, 21-24 August, 2003, Banff, Alberta, Canada.
[15] C-C Yu, and B-D Liu, "A back propagation algorithm with adaptive
learning rate and momentum coefficient," in Proc. of the International
Joint Conference on Neural Networks, IJCNN 2002, pp.1218-1223, May
2007.