Improving the Convergence of the Backpropagation Algorithm Using Local Adaptive Techniques

Since the presentation of the backpropagation algorithm, a vast variety of improvements of the technique for training a feed forward neural networks have been proposed. This article focuses on two classes of acceleration techniques, one is known as Local Adaptive Techniques that are based on weightspecific only, such as the temporal behavior of the partial derivative of the current weight. The other, known as Dynamic Adaptation Methods, which dynamically adapts the momentum factors, α, and learning rate, η, with respect to the iteration number or gradient. Some of most popular learning algorithms are described. These techniques have been implemented and tested on several problems and measured in terms of gradient and error function evaluation, and percentage of success. Numerical evidence shows that these techniques improve the convergence of the Backpropagation algorithm.

• Z. Zainuddin
• N. Mahat
• Y. Abu Hassan

• Backpropagation
• Dynamic Adaptation Methods
• Local Adaptive Techniques
• Neural networks.

[1] Magoulas, G.D., Plagianakos, V.P., & Vrahatis, M.N. Globally
Convergent Algorithms with Local Learning Rates. In IEEE
Transactions on Neural Networks, Vol 13, No 3, 774 - 779, 2002.
[2] Silva F.M. & Almeida L.B., Acceleration techniques for the
backpropagation algorithm. Neural Networks EURASIP Workshop,
Sesim, 1990.
[3] Jacobs R. A., Increased rates of convergence through learning rate
adaptation, Neural Networks, 1(4), 295-308, 1988.
[4] Tollenaere, T., SuperSab:fast adaptive backpropagation with good
scaling properties. Neural Networks, 3(5), 1990.
[5] Fahlman, S.E., An empirical study of learning speed in backpropagation
networks, Technical Report, CMU-CS-88-162, 1988
[6] Vrahatis, M.N., Magoulas, G.D. & Plagaianakos, V.P. Convergence
Analysis of the Quickprop Method. University of Patras and Athens,
1999.
[7] Riedmiller, M. & Braun, H., A direct adaptive method for faster
backpropagation learning. The RPROP algorithm. In Proceedings of the
IEEE International Conference on Neural Networks (ICNN) (Ruspini,
H. ed), p. 586-591, 1993, San Francisco.
[8] Evans, D.J. and Zainuddin, Z., Acceleration of the backpropagation
through dynamic adaptation of the momentum. Neural, Parallel &
Scientific Computations, 5(3), 1997, 297-308. (see also Internal Report
No.1028, PARC, Loughborough University of Tech., U.K., 1996).
[9] Zainuddin Z. & Evans D.J., Acceleration of the backpropagation through
dynamic adaptation of the learning rate, Int. Journal of Computer
Mathematics, 334, 1-17, 1997 (see also Internal report No. 1029, PARC,
Loughborough University of Tech., U.K. 1996).
[10] Zainuddin Z. & Sathasivam S. , Modeling nonlinear relationships in
ecology and biology using neural networks, Proceedings of the National
Workshop in Ecological and environmental modeling (ECOMOD), Sept.
3-4, 2001.
[11] Zainuddin Z. & Ahmad Fadzil, M.H., Training Feedforward Neural
Networks: An Algorithm Giving Improved Convergence. Proceedings of
research & Development in Computer Science & its Application,p.98-
102, 1998, Penang, Malaysia.