Oscillation Effect of the Multi-stage Learning for the Layered Neural Networks and Its Analysis

This paper proposes an efficient learning method for the layered neural networks based on the selection of training data and input characteristics of an output layer unit. Comparing to recent neural networks; pulse neural networks, quantum neuro computation, etc, the multilayer network is widely used due to its simple structure. When learning objects are complicated, the problems, such as unsuccessful learning or a significant time required in learning, remain unsolved. Focusing on the input data during the learning stage, we undertook an experiment to identify the data that makes large errors and interferes with the learning process. Our method devides the learning process into several stages. In general, input characteristics to an output layer unit show oscillation during learning process for complicated problems. The multi-stage learning method proposes by the authors for the function approximation problems of classifying learning data in a phased manner, focusing on their learnabilities prior to learning in the multi layered neural network, and demonstrates validity of the multi-stage learning method. Specifically, this paper verifies by computer experiments that both of learning accuracy and learning time are improved of the BP method as a learning rule of the multi-stage learning method. In learning, oscillatory phenomena of a learning curve serve an important role in learning performance. The authors also discuss the occurrence mechanisms of oscillatory phenomena in learning. Furthermore, the authors discuss the reasons that errors of some data remain large value even after learning, observing behaviors during learning.

• Isao Taguchi
• Yasuo Sugai

• data selection
• function approximation problem
• multistage leaning
• neural network
• voluntary oscillation.

[1] Makoto Motoki´╝îSeiichi Koakutsu´╝îHironori Hirata:"A Supervised Learning
Rule Adjusting Input-Output Pulse Timing for Pulsed Neural Network",
The transactions of the Institute of Electronics, Information and
Communication Engineers, Vol.J89-D-II, No.12, pp.2744-2756 (2006)
[2] Noriaki Kouda´╝îNobuyuki Matsui´╝îHaruhiko Nishimura:"A Multi-
Layered Feed-Forward Network Based on Qubit Neuron Model", The
transactions of the Institute of Electronics, Information and Communication
Engineers, Vol.J85-D-II, No.4, pp.641-648 (2002)
[3] Isao Taguchi and Yasuo Sugai :"An Efficient Learning Method for the
Layered Neural Networks Based on the Selection of Training Data and
Input Characteristics of an Output Layer Unit", The Trans. of The Institute
of Electrical engineers of Japan, Vol.129-C, No.4, pp.1208-1213, (2009)
[4] Nobuyuki Matsui and kenichi Ishimi:"A Multilayered Neural Network
Including Neurons with fluctuated Threshold", The Trans. of The Institute
of Electrical Engineers of Japan, Vol.114-C, No.11, pp.1208-1213 (1994)
[5] D.E.Falleman:"An Empirical Study of Learning Speed in Back-
Propagation Network", Technical Report CMU-CS-88-162, Carnegie-
Mellon University, Computer Sceinece Dept., (1988)
[6] M. Riedmiller and H. Braun:"A DirectbAdaptive Method for Faster
Backpropagation Learning: The RPROP Algorithm", Proc. ICNN, San
Fransisco, (1993)
[7] Isao Taguchi and Yasuo Sugai :"An Input Characteristic of Output Layer
Units in the Layered Neural Networks and Its Application to an Efficient
Learning", Proc. of the Electronics, Information and Systems Conference,
Electronics, Information and systems Society, IEE. of Japan, pp.931-934
(2004)
[8] Isao Taguchi and Yasuo Sugai :"An Efficient Learning Method for the
Layered Neural Networks Based on the Selection of Training Data and
Input Characteristics of an Output Layer Unit", The Trans. of The Institute
of Electrical engineers of Japan, Vol.129-C, No.4, pp.1208-1213, (2009)
[9] D.E.Rumelhart, J.L.McCleland, and the PDP Research Group: "Parallel
Distributed Processing Vo.1", MIT Press (1986).
[10] Takashi Kanemaru and Masatoshi Sekine:"Oscillations and Synchronizations
in Class 1 Neural Networks", TECHNICAL REPORT OF THE
INSTITUTE OF ELECTRONICS, INFORMATION AND COMMUNICATION
ENGINEERS, NC2003-138, pp.17-22 (2004)
[11] Ting Wang and Yasuo Sugai:"A Wavelet Neural Network for the
Approximation of Nonlinear multivariable Functions", The Trans. of The
Institute of Electrical engineers of Japan, Vol.120-C, No.2, pp.185-193
(2000)
[12] Ting Wang and Yasuo Sugai:"A Wavelet Neural Network for the
Approximation of Nonlinear Multivariable Functions", Proc.of IEEE
International Conference on System, Man, and Cybernetics, III, pp.378-
383 (1999)
[13] K. Funahashi:"On the Approximate Realization of Continuous Mapping
by Neural Networks", Vol.2, No.3, pp.183-192 (1989)
[14] Yasuo Sugai´╝îHiroshi Horibe, and Tarou Kawase:"Forecast of Daily
Maximum Electric Load by Neural Networks Using the Standard Electric
Load", The Trans. of The Institute of Electrical engineers of Japan,
Vol.117-B, No.6, pp.872-879 (1997)
[15] Souiti Umehara, teru Yamazaki, and Yasuo Sugai:"A Precipitation Estimation
System Based on Support Vector Machine and Neural Network",
The transactions of the Institute of Electronics, Information and Communication
Engineers, Vol.J86-D-II, No.7, pp.1090-1098 (2003)
[16] Charles K.Chui:"An Introduction to Wavelets", Academic Press (1992)
[17] L. ´╝½.Jones:"Constructive Approximations for Neural Networks by Sigmoidal
Functions", Proc. IEEE, Vol.78, No.10, pp.(1990)
[18] B.Irie and S.Miyake:"Capabilities of Three Layered Perceptrons",
Proc.ICNN, Vol.1, pp.641-648 (1988)