System Identification with General Dynamic Neural Networks and Network Pruning
This paper presents an exact pruning algorithm with
adaptive pruning interval for general dynamic neural networks
(GDNN). GDNNs are artificial neural networks with internal dynamics.
All layers have feedback connections with time delays to the
same and to all other layers. The structure of the plant is unknown, so
the identification process is started with a larger network architecture
than necessary. During parameter optimization with the Levenberg-
Marquardt (LM) algorithm irrelevant weights of the dynamic neural
network are deleted in order to find a model for the plant as
simple as possible. The weights to be pruned are found by direct
evaluation of the training data within a sliding time window. The
influence of pruning on the identification system depends on the
network architecture at pruning time and the selected weight to be
deleted. As the architecture of the model is changed drastically during
the identification and pruning process, it is suggested to adapt the
pruning interval online. Two system identification examples show
the architecture selection ability of the proposed pruning approach.
[1] Y. Le Cun, J. S. Denker, S. A. Solla, "Optimal Brain Damage," in
D. S. Touretzky, "Advances in Neural Information Processing Systems,"
Morgan Kaufmann, 1990, pp. 598-605.
[2] B. Hassibi, D. G. Stork, G. J. Wolff, "Optimal Brain Surgeon and
General Network Pruning," IEEE International Conference on Neural
Networks, vol. 1, pp. 293-299, April 1993.
[3] R. Reed, "Pruning Algorithms - A Survey," IEEE Transactions on
Neural Networks, vol. 4, no. 5, pp. 740-747, September 1993.
[4] M. Attik, L. Bougrain, F. Alexandre, "Optimal Brain Surgeon Variants
For Feature Selection," in IEEE Proceedings of the International Joint
Conference on Neural Networks, pp. 1371-1374, 2004.
[5] O. De Jes'us, M. Hagan, "Backpropagation Algorithms Through Time
for a General Class of Recurrent Network," IEEE Int. Joint Conf. Neural
Network, Washington, 2001, pp. 2638-2643.
[6] O. De Jes'us, M. Hagan, "Forward Perturbation Algorithm For a General
Class of Recurrent Network," IEEE Int. Joint Conf. Neural Network,
Washington, 2001, pp. 2626-2631.
[7] O. De Jes'us, "Training General Dynamic Neural Networks," Ph.D.
dissertation, Oklahoma State University, Stillwater, OK, 2002.
[8] O. De Jes'us, M. Hagan, "Backpropagation Algorithms for a Broad Class
of Dynamic Networks," IEEE Transactions on Neural Networks, vol. 18,
no. 1, pp. 14-27, January 2007.
[9] M. Hagan, B. M. Mohammed, "Training Feedforward Networks with
the Marquardt Algorithm," IEEE Transactions on Neural Networks, vol.
5, no. 6, pp. 989-993, November 1994.
[10] L.S.H. Ngia, J. Sj¨oberg, "Efficient Training of Neural Nets for Nonlinear
Adaptive Filtering Using a Recursive Levenberg-Marquardt Algorithm,"
IEEE Transactions on Signal Processing, vol. 48, no. 7, pp. 1915-1927,
July 2000.
[11] P.J. Werbos, "Backpropagation Through Time: What it is and how to to
it," Proc. IEEE, vol. 78, no. 10, pp. 1550-1560, 1990.
[12] R.J. Williams, D. Zipser, "A Learning Algorithm for Continually Running
Fully Recurrent Neural Networks," Neural Computing, vol. 1, pp.
270-280, 1989.
[13] O. Nelles, Nonlinear System Identification. Berlin Heidelberg New York:
Springer-Verlag, 2001.
[14] D. Schr¨oder, Elektrische Antriebe - Regelung von Antriebssystemen.
2nd edn., Berlin Heidelberg New York: Springer-Verlag, 2001.
[15] K. S. Narendra, K. Parthasarathy, "Identification and Control of Dynamical
Systems Using Neural Networks," IEEE Transactions on Neural
Networks, vol. 1, no. 1, pp. 4-27, November 1990.
[1] Y. Le Cun, J. S. Denker, S. A. Solla, "Optimal Brain Damage," in
D. S. Touretzky, "Advances in Neural Information Processing Systems,"
Morgan Kaufmann, 1990, pp. 598-605.
[2] B. Hassibi, D. G. Stork, G. J. Wolff, "Optimal Brain Surgeon and
General Network Pruning," IEEE International Conference on Neural
Networks, vol. 1, pp. 293-299, April 1993.
[3] R. Reed, "Pruning Algorithms - A Survey," IEEE Transactions on
Neural Networks, vol. 4, no. 5, pp. 740-747, September 1993.
[4] M. Attik, L. Bougrain, F. Alexandre, "Optimal Brain Surgeon Variants
For Feature Selection," in IEEE Proceedings of the International Joint
Conference on Neural Networks, pp. 1371-1374, 2004.
[5] O. De Jes'us, M. Hagan, "Backpropagation Algorithms Through Time
for a General Class of Recurrent Network," IEEE Int. Joint Conf. Neural
Network, Washington, 2001, pp. 2638-2643.
[6] O. De Jes'us, M. Hagan, "Forward Perturbation Algorithm For a General
Class of Recurrent Network," IEEE Int. Joint Conf. Neural Network,
Washington, 2001, pp. 2626-2631.
[7] O. De Jes'us, "Training General Dynamic Neural Networks," Ph.D.
dissertation, Oklahoma State University, Stillwater, OK, 2002.
[8] O. De Jes'us, M. Hagan, "Backpropagation Algorithms for a Broad Class
of Dynamic Networks," IEEE Transactions on Neural Networks, vol. 18,
no. 1, pp. 14-27, January 2007.
[9] M. Hagan, B. M. Mohammed, "Training Feedforward Networks with
the Marquardt Algorithm," IEEE Transactions on Neural Networks, vol.
5, no. 6, pp. 989-993, November 1994.
[10] L.S.H. Ngia, J. Sj¨oberg, "Efficient Training of Neural Nets for Nonlinear
Adaptive Filtering Using a Recursive Levenberg-Marquardt Algorithm,"
IEEE Transactions on Signal Processing, vol. 48, no. 7, pp. 1915-1927,
July 2000.
[11] P.J. Werbos, "Backpropagation Through Time: What it is and how to to
it," Proc. IEEE, vol. 78, no. 10, pp. 1550-1560, 1990.
[12] R.J. Williams, D. Zipser, "A Learning Algorithm for Continually Running
Fully Recurrent Neural Networks," Neural Computing, vol. 1, pp.
270-280, 1989.
[13] O. Nelles, Nonlinear System Identification. Berlin Heidelberg New York:
Springer-Verlag, 2001.
[14] D. Schr¨oder, Elektrische Antriebe - Regelung von Antriebssystemen.
2nd edn., Berlin Heidelberg New York: Springer-Verlag, 2001.
[15] K. S. Narendra, K. Parthasarathy, "Identification and Control of Dynamical
Systems Using Neural Networks," IEEE Transactions on Neural
Networks, vol. 1, no. 1, pp. 4-27, November 1990.
@article{"International Journal of Electrical, Electronic and Communication Sciences:62270", author = "Christian Endisch and Christoph Hackl and Dierk Schröder", title = "System Identification with General Dynamic Neural Networks and Network Pruning", abstract = "This paper presents an exact pruning algorithm with
adaptive pruning interval for general dynamic neural networks
(GDNN). GDNNs are artificial neural networks with internal dynamics.
All layers have feedback connections with time delays to the
same and to all other layers. The structure of the plant is unknown, so
the identification process is started with a larger network architecture
than necessary. During parameter optimization with the Levenberg-
Marquardt (LM) algorithm irrelevant weights of the dynamic neural
network are deleted in order to find a model for the plant as
simple as possible. The weights to be pruned are found by direct
evaluation of the training data within a sliding time window. The
influence of pruning on the identification system depends on the
network architecture at pruning time and the selected weight to be
deleted. As the architecture of the model is changed drastically during
the identification and pruning process, it is suggested to adapt the
pruning interval online. Two system identification examples show
the architecture selection ability of the proposed pruning approach.", keywords = "System identification, dynamic neural network, recurrentneural network, GDNN, optimization, Levenberg Marquardt, realtime recurrent learning, network pruning, quasi-online learning.", volume = "2", number = "6", pages = "1238-9", }