Learning an Overcomplete Dictionary using a Cauchy Mixture Model for Sparse Decay
An algorithm for learning an overcomplete dictionary
using a Cauchy mixture model for sparse decomposition of an underdetermined
mixing system is introduced. The mixture density
function is derived from a ratio sample of the observed mixture
signals where 1) there are at least two but not necessarily more
mixture signals observed, 2) the source signals are statistically
independent and 3) the sources are sparse. The basis vectors of the
dictionary are learned via the optimization of the location parameters
of the Cauchy mixture components, which is shown to be more
accurate and robust than the conventional data mining methods
usually employed for this task. Using a well known sparse
decomposition algorithm, we extract three speech signals from two
mixtures based on the estimated dictionary. Further tests with
additive Gaussian noise are used to demonstrate the proposed
algorithm-s robustness to outliers.
[1] M. S. Lewicki and T. J. Sejnowski, "Learning Overcomplete
Representations, " Neural Computations, vol. 12, No. 2, pp. 337-365,
February 2000.
[2] M. Zhong, H. Tang, H. Cheng and Y. Tang, "An EM Algorithm for
Learning Sparse and Overcomplete Representations, " Neurocomputing,
vol. 57, pp. 467-476, 2004.
[3] M. Aharon, M. Elad and A. Bruckstein, "K-SVD: An algorithm for
Designing Overcomplete Dictionaries for Sparse Representations," IEEE
Transactions on Signal Processing, vol. 54, No. 11, November 2006.
[4] K. Kreutz-Delgado, J. F. Murray, B. D. Rao, K. Egan, T. Lee and T. J.
Sejnowski, "Dictionary Learning Algorithms for Sparse Representations,
" eural Computations, vol. 15, No. 2, pp. 349-396, 2003.
[5] K. Egan, S. O. Aase and J. H. Hakon-Husoy, "Method of Optimal
Directions for Frame Design, "in IEEE International Conference of
Acoustic, Speech and Signal Processing, Vol. 5, pp. 2443-2446, 1999.
[6] I. T. Jollife, Principal Component Analysis, Series: Springer Series in
Statistics, 2nd Edition, .
[7] P. Common, "Independent Component Analysis: A new Concept?, "
Signal Processing, vol. 36, pp. 287-314, 1994.
[8] G. Marsaglia, "Ratios of Normal Variables, " Journal of Statistical
Software, vol. 16, No. 4, May 2006.
[9] W. Feller, An Introduction to Probability Theory and Applications, 2nd
Edition, New York: Wiley, pp. 229-235, 1968.
[10] P. G. Hoel, Introduction to Mathematical Statistics, 3rd Edition, New
York: Wiley, pp. 57-62, 1962.
[11] J. W. Harris and H. Stoker, "Maximum Likelihood Method, " in
Handbook of Mathematics and Computational Science, New York:
Springer-Verlag, pp. 824-835, 1998.
[12] A. Koutrouvelis, "Estimation of the Location and Scale of the Cauchy
Distribution using the Empirical Characteristic Function, " Biometrica,
Vol. 69, pp. 205-213, 1982.
[13] F. Nagy, "Parameter Estimation of the Cauchy Distribution in
Information Theory Approach, "Journal of Universal Computer Science,
Vol. 12, No. 9, pp. 1332-1344, May 2006.
[14] G. B. Freue, "The Pitman Estimator of the Cauchy Location Parameter, "
Journal of Statistical Planning and Inference, vol. 137, pp. 1900-1913,
May 2006.
[15] K. M. Hanson and D. R. Wolf, "Estimators for the Cauchy Distribution,
" In Maximum Entropy and Bayesian Methods, pp. 255-263, 1996.
[16] C. Jutten, G. H. Mohimani and M. B. Zadeh, "Fast Sparse
Representations based on the l0 Norm, " In Pro. ICA-07, London, UK,
2007.
[17] G. H. Mohimani, M. B. Zaden and C. Jutten, "Complex Valued Sparse
Representations based on Smoothed l0 Norm, " In ICASSP-08, 2008.
[18] S. S. Chen, D. L. Donoho and M. A. Saunders, "Atomic Decomposition
by Basis Pursuit, " SIAM Journal of Computing, Vol. 20, No. 1, pp. 33-
61, 1999.
[19] S. G. Mallat and Z. Zhang, "Matching Pursuit with Time-Frequency
Dictionaries, " IEEE Transactions on Signal Processing, pp. 3397-3415,
December 1993.
[20] I. F. Gorodnitsky and B. D. Rao, "Sparse Signal Reconstruction from
Limited Data using FOCUSS: A Re-weighted Minimum Norm
Algorithm, " IEEE Transactions on Signal Processing, Vol. 45, No. 3,
March 1997.
[21] J. E. Marsden and J. M. Hoffman, Basic Complex Analysis, 3rd Edition,
W. H. Freeman, 1998, ISBN: 978-0716728771.
[1] M. S. Lewicki and T. J. Sejnowski, "Learning Overcomplete
Representations, " Neural Computations, vol. 12, No. 2, pp. 337-365,
February 2000.
[2] M. Zhong, H. Tang, H. Cheng and Y. Tang, "An EM Algorithm for
Learning Sparse and Overcomplete Representations, " Neurocomputing,
vol. 57, pp. 467-476, 2004.
[3] M. Aharon, M. Elad and A. Bruckstein, "K-SVD: An algorithm for
Designing Overcomplete Dictionaries for Sparse Representations," IEEE
Transactions on Signal Processing, vol. 54, No. 11, November 2006.
[4] K. Kreutz-Delgado, J. F. Murray, B. D. Rao, K. Egan, T. Lee and T. J.
Sejnowski, "Dictionary Learning Algorithms for Sparse Representations,
" eural Computations, vol. 15, No. 2, pp. 349-396, 2003.
[5] K. Egan, S. O. Aase and J. H. Hakon-Husoy, "Method of Optimal
Directions for Frame Design, "in IEEE International Conference of
Acoustic, Speech and Signal Processing, Vol. 5, pp. 2443-2446, 1999.
[6] I. T. Jollife, Principal Component Analysis, Series: Springer Series in
Statistics, 2nd Edition, .
[7] P. Common, "Independent Component Analysis: A new Concept?, "
Signal Processing, vol. 36, pp. 287-314, 1994.
[8] G. Marsaglia, "Ratios of Normal Variables, " Journal of Statistical
Software, vol. 16, No. 4, May 2006.
[9] W. Feller, An Introduction to Probability Theory and Applications, 2nd
Edition, New York: Wiley, pp. 229-235, 1968.
[10] P. G. Hoel, Introduction to Mathematical Statistics, 3rd Edition, New
York: Wiley, pp. 57-62, 1962.
[11] J. W. Harris and H. Stoker, "Maximum Likelihood Method, " in
Handbook of Mathematics and Computational Science, New York:
Springer-Verlag, pp. 824-835, 1998.
[12] A. Koutrouvelis, "Estimation of the Location and Scale of the Cauchy
Distribution using the Empirical Characteristic Function, " Biometrica,
Vol. 69, pp. 205-213, 1982.
[13] F. Nagy, "Parameter Estimation of the Cauchy Distribution in
Information Theory Approach, "Journal of Universal Computer Science,
Vol. 12, No. 9, pp. 1332-1344, May 2006.
[14] G. B. Freue, "The Pitman Estimator of the Cauchy Location Parameter, "
Journal of Statistical Planning and Inference, vol. 137, pp. 1900-1913,
May 2006.
[15] K. M. Hanson and D. R. Wolf, "Estimators for the Cauchy Distribution,
" In Maximum Entropy and Bayesian Methods, pp. 255-263, 1996.
[16] C. Jutten, G. H. Mohimani and M. B. Zadeh, "Fast Sparse
Representations based on the l0 Norm, " In Pro. ICA-07, London, UK,
2007.
[17] G. H. Mohimani, M. B. Zaden and C. Jutten, "Complex Valued Sparse
Representations based on Smoothed l0 Norm, " In ICASSP-08, 2008.
[18] S. S. Chen, D. L. Donoho and M. A. Saunders, "Atomic Decomposition
by Basis Pursuit, " SIAM Journal of Computing, Vol. 20, No. 1, pp. 33-
61, 1999.
[19] S. G. Mallat and Z. Zhang, "Matching Pursuit with Time-Frequency
Dictionaries, " IEEE Transactions on Signal Processing, pp. 3397-3415,
December 1993.
[20] I. F. Gorodnitsky and B. D. Rao, "Sparse Signal Reconstruction from
Limited Data using FOCUSS: A Re-weighted Minimum Norm
Algorithm, " IEEE Transactions on Signal Processing, Vol. 45, No. 3,
March 1997.
[21] J. E. Marsden and J. M. Hoffman, Basic Complex Analysis, 3rd Edition,
W. H. Freeman, 1998, ISBN: 978-0716728771.
@article{"International Journal of Engineering, Mathematical and Physical Sciences:59387", author = "E. S. Gower and M. O. J. Hawksford", title = "Learning an Overcomplete Dictionary using a Cauchy Mixture Model for Sparse Decay", abstract = "An algorithm for learning an overcomplete dictionary
using a Cauchy mixture model for sparse decomposition of an underdetermined
mixing system is introduced. The mixture density
function is derived from a ratio sample of the observed mixture
signals where 1) there are at least two but not necessarily more
mixture signals observed, 2) the source signals are statistically
independent and 3) the sources are sparse. The basis vectors of the
dictionary are learned via the optimization of the location parameters
of the Cauchy mixture components, which is shown to be more
accurate and robust than the conventional data mining methods
usually employed for this task. Using a well known sparse
decomposition algorithm, we extract three speech signals from two
mixtures based on the estimated dictionary. Further tests with
additive Gaussian noise are used to demonstrate the proposed
algorithm-s robustness to outliers.", keywords = "expectation-maximization, Pitman estimator, sparsedecomposition", volume = "5", number = "3", pages = "433-8", }
