Enhancing Multi-Frame Images Using Self-Delaying Dynamic Networks
This paper presents the use of a newly created network
structure known as a Self-Delaying Dynamic Network (SDN) to
create a high resolution image from a set of time stepped input
frames. These SDNs are non-recurrent temporal neural networks
which can process time sampled data. SDNs can store input data
for a lifecycle and feature dynamic logic based connections between
layers. Several low resolution images and one high resolution image
of a scene were presented to the SDN during training by a Genetic
Algorithm. The SDN was trained to process the input frames in order
to recreate the high resolution image. The trained SDN was then used
to enhance a number of unseen noisy image sets. The quality of high
resolution images produced by the SDN is compared to that of high
resolution images generated using Bi-Cubic interpolation. The SDN
produced images are superior in several ways to the images produced
using Bi-Cubic interpolation.
[1] E. H. W. Meijering, "A chronology of interpolation," in Proceedings of
the IEEE, vol. 90, no. 3, March 2002, pp. 319-342.
[2] T. M. Lehmann, C. Gonner, and K. Spitzer, "Survey: Interpolation
methods in medical image processing," IEEE Trans. Med. Imag., vol. 18,
no. 11, pp. 1049-1075, November 1999.
[3] S. Battiato, G. Gallo, M. Mancuso, G. Messina, and F. Stanco, "Analysis
and characterization of super-resolution reconstruction methods," in
SPIE Electronic Imaging 2003, Sensors, Cameras and Applications for
Digital Photography, vol. 5017B-37, 2003.
[4] E. H. W. Meijering, W. J. Niessen, and M. A. Viergever, "Quantitative
evaluation of convolution-based methods for medical image interpolation,"
Medical Image Analysis, vol. 5, no. 2, pp. 111-126, June 2001.
[5] E. H. W. Meijering, "Spline interpolation in medical imaging: Comparison
with other convolution based approaches," in Signal Processing X:
Theories and Applications, EUSIPCO 2000, M. Gabbouj and P. Kuosmanen,
Eds., vol. 4. The European Association for Signal Processing,
Tampere, 2000, pp. 1989-1996.
[6] M. Unser, A. Aldroubi, and M. Eden, "Enlargement or reduction of
digital images with minimum loss of information," IEEE Trans. Image
Processing, vol. 4, no. 3, pp. 247-258, 1995.
[7] N. A. Dodgson, "Quadratic interpolation for image resampling," IEEE
Trans. Image Processing, vol. 6, no. 9, pp. 1322-1326, September 1997.
[8] F. Anton, D. Mioc, and A. Fournier, "2d image reconstruction using
natural neighbour interpolation," in The Eighth International Conference
on Computer Graphics and Visualization in Central Europe (WSCG-
2000), Febuary 2000, pp. 263-269.
[9] X. Li, "New edge-directed interpolation," IEEE Trans. Image Processing,
vol. 10, pp. 1521-1527, October 2001.
[10] K. P. Hong, J. K. Paik, H. J. Kim, and C. H. Lee, "An edgepreserving
image interpolation system for a digital camcorder," IEEE
Trans. Consumer Electron., vol. 42, no. 3, pp. 279-284, 1996.
[11] L. W. Leung, B. King, and V. Vohora, "Comparison of image data fusion
techniques using entropy and ini," in 22nd Asian Converence of Remove
Sensing. Centre for Remote Imaging, Sensing and Processing CRISP,
November 2001.
[12] G. Piella, "A general framework for multiresolution image fusion: from
pixels to regions," Information Fusion, pp. 259-280, 2003.
[13] A. Waxman, A. Gove, D. Fay, J. Racamato, J. Carrick, M. Seibert, and
D. Savoye, "Color night vision: Opponent processing in the fusion of
visible and ir imagery," in Neural Networks, vol. 10, no. 1, 1997.
[14] H. Li, B. S. Manjunath, and S. K. Mitra, "Multi-sensor image fusion
using the wavelet transform." in ICIP (1), 1994, pp. 51-55.
[15] J. Nunez, X. Otazu, O. Fors, A. Prades, V. Pala, and R. Arbiol,
"Multiresolution-based image fusion with additive wavelet decomposition,"
IEEE Trans. Geosci. Remote Sensing, vol. 37, pp. 1204-1211,
1999.
[16] A. Garzelli, "Wavelet-based fusion of optical and sar image data over
urban area," in PCV02, 2002, p. B: 59.
[17] Y. Zhang and R. Wang, "Multi-resolution and multi-spectral image
fusion for urban object extraction," in Proceedings of XXth ISPRS
Congress, Commission III, 2004, pp. 960-966.
[18] P. S. Chavez, C. S. Stuart, and J. A. Anderson, "Comparison of
three different methods to merge multiresolution and multispectral data:
Landsat tm and spot panchromatic," Photogrammetric Engineering and
Remote Sensing, vol. 57, pp. 295-303, 1991.
[19] C. K. Munechika, J. S. Warnick, and C. Salvaggo, "Resolution enhancement
of multispectral image data to improve calssification accuracy,"
Photographic Engineering and Remote Sensing, vol. 59, no. 1, pp. 67-
72, Januray 1993.
[20] F. Sroubek and J. Flusser, "Image fusion via probabilistic deconvolution,"
in Third International Workshop on Pattern Recognition in Remote
Sensing, 2004, pp. 1-5.
[21] H. Kiiveri, "Image fusion with conditional probability networks for
monitoring salinisation of farmland," 1998.
[22] G. Pajares and J. Manuel de la Cruz, "A wavelet based image fusion
tutorial," Pattern Recognition, vol. 37, pp. 1855-1872, 2004.
[23] R. Tsai and T. Huang, Multiframe Image Restoration and Registration,
R. Tsai and T. Huang, Eds. JAI Press, 1984, vol. 1.
[24] M. Irani and S. Peleg, "Super resolution from image sequences," 10th
ICPR, vol. 2, pp. 115-120, 1990.
[25] R. Schultz and R. Stevenson, "Extraction of high resolution frames from
video sequences," IEEE Trans. Image Processing, vol. 5, no. 6, pp. 996-
1011, 1996.
[26] S. Kim, N. Bose, and H. Valenzuela, "Recursive reconstruction of high
resolution image from noisy undersampled multiframes," IEEE Trans.
Acoust., Speech, Signal Processing, vol. 38, no. 6, 1990.
[27] E. Salari and S. Zhang, "Integrated recurrent neural network for image
resolution enhancement from multiple image frames," in IEE Proceedings
on Vision, Image and Signal Processing, vol. 150, no. 5, 2003.
[28] M. Van Veelen, J. Nijhuis, and B. Spaanenburg, "Neural network
approaches to capture temporal information," in CASYS 1999, 2000, pp.
361-371.
[29] R. Ivry and R. Spencer, "The neural representation of time," Current
Opinion in Neurobiology, vol. 14, no. 2, pp. 225-232, 2004.
[30] A. Herz, "How is time represented in the brain," to Appear: Problems
in Systems Neuroscience.
[31] J. Elman, "Finding structure in time," Cognitive Science, vol. 14, pp.
179-211, 1990.
[32] S. Hochreiter and J. Schmidhuber, "Long short term memory," Neural
Computation, vol. 9, no. 8, pp. 1735-1780, 1997.
[33] D. Wang, X. Liu, and S. Ahalt, "On temporal generalization of simple
recurrent networks," Neural Networks, vol. 9, pp. 1099-1118, 1996.
[1] E. H. W. Meijering, "A chronology of interpolation," in Proceedings of
the IEEE, vol. 90, no. 3, March 2002, pp. 319-342.
[2] T. M. Lehmann, C. Gonner, and K. Spitzer, "Survey: Interpolation
methods in medical image processing," IEEE Trans. Med. Imag., vol. 18,
no. 11, pp. 1049-1075, November 1999.
[3] S. Battiato, G. Gallo, M. Mancuso, G. Messina, and F. Stanco, "Analysis
and characterization of super-resolution reconstruction methods," in
SPIE Electronic Imaging 2003, Sensors, Cameras and Applications for
Digital Photography, vol. 5017B-37, 2003.
[4] E. H. W. Meijering, W. J. Niessen, and M. A. Viergever, "Quantitative
evaluation of convolution-based methods for medical image interpolation,"
Medical Image Analysis, vol. 5, no. 2, pp. 111-126, June 2001.
[5] E. H. W. Meijering, "Spline interpolation in medical imaging: Comparison
with other convolution based approaches," in Signal Processing X:
Theories and Applications, EUSIPCO 2000, M. Gabbouj and P. Kuosmanen,
Eds., vol. 4. The European Association for Signal Processing,
Tampere, 2000, pp. 1989-1996.
[6] M. Unser, A. Aldroubi, and M. Eden, "Enlargement or reduction of
digital images with minimum loss of information," IEEE Trans. Image
Processing, vol. 4, no. 3, pp. 247-258, 1995.
[7] N. A. Dodgson, "Quadratic interpolation for image resampling," IEEE
Trans. Image Processing, vol. 6, no. 9, pp. 1322-1326, September 1997.
[8] F. Anton, D. Mioc, and A. Fournier, "2d image reconstruction using
natural neighbour interpolation," in The Eighth International Conference
on Computer Graphics and Visualization in Central Europe (WSCG-
2000), Febuary 2000, pp. 263-269.
[9] X. Li, "New edge-directed interpolation," IEEE Trans. Image Processing,
vol. 10, pp. 1521-1527, October 2001.
[10] K. P. Hong, J. K. Paik, H. J. Kim, and C. H. Lee, "An edgepreserving
image interpolation system for a digital camcorder," IEEE
Trans. Consumer Electron., vol. 42, no. 3, pp. 279-284, 1996.
[11] L. W. Leung, B. King, and V. Vohora, "Comparison of image data fusion
techniques using entropy and ini," in 22nd Asian Converence of Remove
Sensing. Centre for Remote Imaging, Sensing and Processing CRISP,
November 2001.
[12] G. Piella, "A general framework for multiresolution image fusion: from
pixels to regions," Information Fusion, pp. 259-280, 2003.
[13] A. Waxman, A. Gove, D. Fay, J. Racamato, J. Carrick, M. Seibert, and
D. Savoye, "Color night vision: Opponent processing in the fusion of
visible and ir imagery," in Neural Networks, vol. 10, no. 1, 1997.
[14] H. Li, B. S. Manjunath, and S. K. Mitra, "Multi-sensor image fusion
using the wavelet transform." in ICIP (1), 1994, pp. 51-55.
[15] J. Nunez, X. Otazu, O. Fors, A. Prades, V. Pala, and R. Arbiol,
"Multiresolution-based image fusion with additive wavelet decomposition,"
IEEE Trans. Geosci. Remote Sensing, vol. 37, pp. 1204-1211,
1999.
[16] A. Garzelli, "Wavelet-based fusion of optical and sar image data over
urban area," in PCV02, 2002, p. B: 59.
[17] Y. Zhang and R. Wang, "Multi-resolution and multi-spectral image
fusion for urban object extraction," in Proceedings of XXth ISPRS
Congress, Commission III, 2004, pp. 960-966.
[18] P. S. Chavez, C. S. Stuart, and J. A. Anderson, "Comparison of
three different methods to merge multiresolution and multispectral data:
Landsat tm and spot panchromatic," Photogrammetric Engineering and
Remote Sensing, vol. 57, pp. 295-303, 1991.
[19] C. K. Munechika, J. S. Warnick, and C. Salvaggo, "Resolution enhancement
of multispectral image data to improve calssification accuracy,"
Photographic Engineering and Remote Sensing, vol. 59, no. 1, pp. 67-
72, Januray 1993.
[20] F. Sroubek and J. Flusser, "Image fusion via probabilistic deconvolution,"
in Third International Workshop on Pattern Recognition in Remote
Sensing, 2004, pp. 1-5.
[21] H. Kiiveri, "Image fusion with conditional probability networks for
monitoring salinisation of farmland," 1998.
[22] G. Pajares and J. Manuel de la Cruz, "A wavelet based image fusion
tutorial," Pattern Recognition, vol. 37, pp. 1855-1872, 2004.
[23] R. Tsai and T. Huang, Multiframe Image Restoration and Registration,
R. Tsai and T. Huang, Eds. JAI Press, 1984, vol. 1.
[24] M. Irani and S. Peleg, "Super resolution from image sequences," 10th
ICPR, vol. 2, pp. 115-120, 1990.
[25] R. Schultz and R. Stevenson, "Extraction of high resolution frames from
video sequences," IEEE Trans. Image Processing, vol. 5, no. 6, pp. 996-
1011, 1996.
[26] S. Kim, N. Bose, and H. Valenzuela, "Recursive reconstruction of high
resolution image from noisy undersampled multiframes," IEEE Trans.
Acoust., Speech, Signal Processing, vol. 38, no. 6, 1990.
[27] E. Salari and S. Zhang, "Integrated recurrent neural network for image
resolution enhancement from multiple image frames," in IEE Proceedings
on Vision, Image and Signal Processing, vol. 150, no. 5, 2003.
[28] M. Van Veelen, J. Nijhuis, and B. Spaanenburg, "Neural network
approaches to capture temporal information," in CASYS 1999, 2000, pp.
361-371.
[29] R. Ivry and R. Spencer, "The neural representation of time," Current
Opinion in Neurobiology, vol. 14, no. 2, pp. 225-232, 2004.
[30] A. Herz, "How is time represented in the brain," to Appear: Problems
in Systems Neuroscience.
[31] J. Elman, "Finding structure in time," Cognitive Science, vol. 14, pp.
179-211, 1990.
[32] S. Hochreiter and J. Schmidhuber, "Long short term memory," Neural
Computation, vol. 9, no. 8, pp. 1735-1780, 1997.
[33] D. Wang, X. Liu, and S. Ahalt, "On temporal generalization of simple
recurrent networks," Neural Networks, vol. 9, pp. 1099-1118, 1996.
@article{"International Journal of Information, Control and Computer Sciences:53167", author = "Lewis E. Hibell and Honghai Liu and David J. Brown", title = "Enhancing Multi-Frame Images Using Self-Delaying Dynamic Networks", abstract = "This paper presents the use of a newly created network
structure known as a Self-Delaying Dynamic Network (SDN) to
create a high resolution image from a set of time stepped input
frames. These SDNs are non-recurrent temporal neural networks
which can process time sampled data. SDNs can store input data
for a lifecycle and feature dynamic logic based connections between
layers. Several low resolution images and one high resolution image
of a scene were presented to the SDN during training by a Genetic
Algorithm. The SDN was trained to process the input frames in order
to recreate the high resolution image. The trained SDN was then used
to enhance a number of unseen noisy image sets. The quality of high
resolution images produced by the SDN is compared to that of high
resolution images generated using Bi-Cubic interpolation. The SDN
produced images are superior in several ways to the images produced
using Bi-Cubic interpolation.", keywords = "Image Enhancement, Neural Networks, Multi-Frame.", volume = "1", number = "12", pages = "3819-6", }
