Hybrid Machine Learning Approach for Text Categorization

Text categorization - the assignment of natural language documents to one or more predefined categories based on their semantic content - is an important component in many information organization and management tasks. Performance of neural networks learning is known to be sensitive to the initial weights and architecture. This paper discusses the use multilayer neural network initialization with decision tree classifier for improving text categorization accuracy. An adaptation of the algorithm is proposed in which a decision tree from root node until a final leave is used for initialization of multilayer neural network. The experimental evaluation demonstrates this approach provides better classification accuracy with Reuters-21578 corpus, one of the standard benchmarks for text categorization tasks. We present results comparing the accuracy of this approach with multilayer neural network initialized with traditional random method and decision tree classifiers.

• Nerijus Remeikis
• Ignas Skucas
• Vida Melninkaite

• Text categorization
• decision trees
• neural networks
• machine learning.

[1] Banerji, A. (1997). Initializing neural networks using decision trees.
Computational Learning Theory and Natural Learning Systems, MIT
Press, IV, 3-15.
[2] Frakes, W., and R. Baeza-Yates (1992). Information Retrieval: Data
Structures & Algorithms, Prentice Hall.
[3] Haykin, S. (1994). Neural Networks: A comprehensive foundation.
Macmillan College Publishing Comp., New York.
[4] Yang, Y., and J. Pedersen (1997). A comparative study on feature
selection in text categorization. In Proceedings of ICML-97, 14th
International Conference on Machine Learning, Nashville, US, 412-420.
[5] Yang,Y., and X. Liu (1999). A re-examination of text categorization
methods. In Proceedings of SIGIR-99, 22nd ACM International
Conference on Research and Development in Information Retrieval,
Berkeley, US, 42-49.
[6] Lewis, D.D., and M. Ringuette (1994). A comparison of two learning
algorithms for text categorization. In Proceedings of SDAIR-94, 3rd
Annual Symposium on Document Analysis and Information Retrieval,
Las Vegas, 81-93.
[7] Quinlan, J.R. (1993). C4-5: Programs for machine learning, Morgan
Kaufmann, San Mateo, CA.
[8] Rumelhart, D.E., and J.L. Mcclelland (1986). Parallel distributed
processing 1. MIT Press, Cambridge, MA.
[9] Sebastiani, F. (2002). Machine learning in automated text categorization.
ACM Computing Surveys, 34(1), 1-47.
[10] van Rijsbergen, C.J. (1979). Information Retrieval. Butterworths,
London.
[11] Wiener, E.D., J.O. Pedersen, and A. S. Weigend (1995). A neural
network approach to topic spotting. In Proceedings of SDAIR-95, 4th
Annual Symposium on Document Analysis and Information Retrieval,
Las Vegas, 317-332.
[12] Joachims, T. (1998). Text categorization with support vector machines:
learning with many relevant features. In Proceedings of ECML-98,10th
European Conference on Machine Learning, 137-142.
[13] Dumais, S. T. (1991). Improving the retrieval information from external
soures. Behaviour Research Methods, Instruments and Computers,
23(2), 229-236.