Multi-Context Recurrent Neural Network for Time Series Applications

this paper presents a multi-context recurrent network for time series analysis. While simple recurrent network (SRN) are very popular among recurrent neural networks, they still have some shortcomings in terms of learning speed and accuracy that need to be addressed. To solve these problems, we proposed a multi-context recurrent network (MCRN) with three different learning algorithms. The performance of this network is evaluated on some real-world application such as handwriting recognition and energy load forecasting. We study the performance of this network and we compared it to a very well established SRN. The experimental results showed that MCRN is very efficient and very well suited to time series analysis and its applications.





References:
[1] M. Boden, `A guide to recurrent neural networks and
backpropagation',The DALLAS project. Report from the NUTEKsupported
project AIS-8, SICS.Holst: Application of Data Analysis with
Learning Systems, (2001).
[2] M.A. Castano, F. Casacuberta, and A. Bonet, Training Simple recurrent
Networks Through Gradient Descent Algorithm, volume 1240 of ISBN
3-540-63047-3, chapter Biological and Arti cial Computation: From
Neuroscience to Technology, pp. 493--500, Eds. J. Mira and R.
Moreno-Diaz and J. Cabestany, Springer Verlag, 1997.
[3] W. Charytoniuk and M-S. Chen, `Very short-term load forecasting
using neural networks', IEEE Tran. On Power Systems, 15(1), 1558---
1572, (2000).
[4] B.J. Chen, M.W. Change, and C.J. Lin, `Eunite network competition:
Electricity load forecasting', Technical report, In EUNITE 2001
symposium, a forecasting competition, (2001).
[5] Y. Cheng, T.W. Karjala, and D.M. Himmelblau, `Closed loop nonliner
process identi cation using internal recurrent nets', In Neural
Networks, 10(3), pp. 573--586, (1997).
[6] A. Corradini and P. Cohen, `Multimodal speech-gesture interface for
hands-free painting on virtual paper using partial recurrent neural
networks for gesture recognition', in Proc. of the Int'l Joint Conf. on
Neural Networks (IJCNN'02), volume 3, pp. 2293--2298, (2002).
[7] B. de Vries and J.C. Principe, `A theory for neural networks with time
delays', in NIPS-3: Proceedings of the 1990 conference on Advances in
neural information processing systems 3, pp. 162--168, San Francisco,
CA, USA, (1990). Morgan Kaufmann Publishers Inc.
[8] Georg Dorffner, `Neural networks for time series processing', Neural
Network World, 6(4), pp. 447--468, (1996).
[9] W. Duch and N.Jankowski, `Transfer functions: Hidden possibilities
for better neural networks', in ESANN'2001 proceedings European
Symposium on Arti cial Neural Networks, ISBN 2-930307 01-3, pp.
25-27, Belgium, (2001). D-Facto public.
[10] J.L. Elman, `Finding structure in time', Cognitive Science, 14(2),
pp.179--211, (1999).
[11] D. Esp, `Adaptive logic networks for east slovakian electrical load
forecasting', Technical report, In EUNITE 2001 symposium, a
forecasting competition, (2001).
[12] D.V. Prokhorov E.W. Saad and D.C. Wunsch, `Comparative study of
stock trend prediction using time delay, recurrent and probabilistic
neural networks', IEEE Transactions on Neural Networks, 6(9),
PP.1456--1470, (1998).
[13] L. Fausett, Backpropagation Through Time and Derivative Adaptive
Critics: A Common Framework for Comparison, chapter Englewood
Cliffs, NJ: Prentice Hall, 1994.
[14] G. Gross and F. D. Galianan, `Short-term load forecasting', In
Proceedings of the IEEE., 75(12), 1558--1572, (1987).
[15] I. Guyon, L. Schomaker, R. Plamondon, M. Liberman, and S. janet,
`Unipen project of on-line data exchange and recognizer benchmarks',
in Proceedings of the 12th International Conference on Pattern
Recognition, ICPR'94, pp. 29--33, Jerusalem, Israel, (October 1994).
[16] S. Haykin, Neural Networks, A Comprehensive Foundation,
MacMillan Publishing Company, New York, 1994.
[17] A. Herve and E. Betty, `Neural networks, quantitative applications', in
In the Social Sciences, volume 124, London: Sage Publications, (1999).
[18] B. Q. Huang and M-T. Kechadi, `A recurrent neural network
recogniser for online recognition of handwritten symbols.', in ICEIS
(2), pp. 27--34, (2005).
[19] B. Q. Huang, T. Rashid, and T. Kechadi, `A new modi ed network
based on the elman network', in Proceedings of IASTED International
Conference on Arti cial Intelligence and Application, ed., M. H.
Hamza, volume 1 of ISBN: 088986-404-7, pp. 379--384, Innsbruck,
Austria, (2004). ACTA Press.
[20] M.I. Jordan, `Attractor dynamics and parallelism in a connectionist
sequential machine.', in Proceedings of the 8th Annual Conference of
the Cognitive Science Society, Englewood Cliffs, NJ: Erlbaum, pp.
531--546. Reprinted in IEEE Tutorials Series, New York: IEEE
Publishing Services, 1990, (1986).
[21] I. King and J. Tindle, `Storage of half hourly electric metering data and
forecasting with arti cial neural network technology', Technical
report, In EUNITE 2001 symposium, a forecasting competition, (2001).
[22] W. Kowalczyk, `Averaging and data enrichment: Two approaches to
electricity load forecasting', Technical report, In EUNITE 2001
symposium, a forecasting competition, (2001).
[23] S. Lawrence, C.L. Giles, and S. Fong, `Natural language grammatical
inference with recurrent neural networks', IEEE Trans. on Knowledge
and Data Engineering, 12(1), pp. 126--140, (2000).
[24] A. Lo, H. Mamaysky, and J. Wang, `Foundations of technical analysis:
Computational algorithms, statistical inference, and empirical
implementation', Journal of Finance 55, pp. 1705--1765, (2000).
[25] Yee-Ling LU, Man-Wai MAK, and Wan-Chi SIU, `Application of a
fast real time recurrent learning algorithm to text-to-phone conversion',
in Proceedings of the International Conference of Neural Networks,
volume 5, pp. 2853--2857, (1995).
[26] Simone Marinai, Marco Gori, and Giovanni Soda, `Arti cial neural
networks for document analysis and recognition.', IEEE Trans. Pattern
Anal. Mach. Intell., 27(1), pp. 23--35, (2005).
[27] A. D. Papalxopoulos and T. C. Hiterbeg, `A regression-based approach
to short-term load forecasting', In IEEE Tran. On Power Systems, 4(1),
pp. 1535--1547, (1990).
[28] D. Park, M. El-Sharkawia, R. Marks, A. Atlas, and M. Damborg,
`Electic load forecasting using arti cial neural networks', IEEE Trans.
on Power Systems, 6(2), 442--449, (1991).
[29] D. C. Plaut, `Semantic and associative priming in a distributed attractor
network', in Proceedings of the 17th Annual Conference of the
Cognitive Science Society, pp. 37--42, Hillsdale, (1995). NJ:Erlbaum.
[30] D. Prokhorov, Backpropagation Through Time and Derivative
Adaptive Critics: A Common Framework for Comparison, chapter
Learning and Approximate Dynamic Programming, Wiley, 2004.
[31] T. Rashid, B. Q. Huang, and T. Kechadi, `A new simple recurrent
network with real-time recurrent learning process', in The 14th Irish
Artifcial Intelligence and Cognitive Science (AICS'03), ed., Padraig
Cunningham, volume 1, pp. 169--174, Dublin, Ireland, (2003).
[32] T. Rashid and T. Kechadi, `A practical approach for electricity load
forecasting', in The proceeding WEC'05, The ThirdWorld Enformatika,
ed., C. Ardal, volume 5 of ISBN 975-98458-4-9, pp. 201--205, Isanbul,
Turky, (2005). ACTA Press.
[33] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, Learning Internal
Representations by Error Propagation In D. E. Rumelhart, et al. (Eds.),
Parallel Distributed Processing: Explorations in the Microstructures
of Cognition, 1: Foundations, MA: MIT Press, Cambridge, 1962.
[34] M. Schnekel, I. Guyon, and D. Henderson, `On-line cursive script
recognition using time delay networks and hidden markove models', in
Proc. ICASSP'94, volume 2, pp. 637--640, Adelaide, Australia, (April
1994).
[35] P. Stagge and B. Sendho, `Organization of past states in recurrent
neural networks: Implicit embedding', in Proc. The Internation
conference Computational Intelligence for Modelling, Control &
Automation, pp. 21--27, Amsterdam, (1999). IOS Press.
[36] J.C. Tomasz and M.Z. Jacek, `Neural network tools for stellar light
prediction', in Proc. of the IEEE Aerospace Conference, volume 3, pp.
415--422, Snowmass, Colorado, USA, (February 1997).
[37] P. J. Werbos, `Backpropagation through time: What it does and how to
do it', in Proceedings of the IEEE, volume 78, pp. 1550--1560, (1990).
[38] William H. Wilson, `Learning performance of networks like elman's
simple recurrent netwroks but having multiple state vectors', Workshop
of the 7th Australian Conference on Neural Networks, Australian
National University Canberra, (1996).
[39] Shi XH, YC. Liang, and X. Xu, `An improved elman model and
recurrent bck-propagation control neural networks', Journal of
Software, 6(14), 1110--1119, (2003).