Extraction of Symbolic Rules from Artificial Neural Networks

Although backpropagation ANNs generally predict better than decision trees do for pattern classification problems, they are often regarded as black boxes, i.e., their predictions cannot be explained as those of decision trees. In many applications, it is desirable to extract knowledge from trained ANNs for the users to gain a better understanding of how the networks solve the problems. A new rule extraction algorithm, called rule extraction from artificial neural networks (REANN) is proposed and implemented to extract symbolic rules from ANNs. A standard three-layer feedforward ANN is the basis of the algorithm. A four-phase training algorithm is proposed for backpropagation learning. Explicitness of the extracted rules is supported by comparing them to the symbolic rules generated by other methods. Extracted rules are comparable with other methods in terms of number of rules, average number of conditions for a rule, and predictive accuracy. Extensive experimental studies on several benchmarks classification problems, such as breast cancer, iris, diabetes, and season classification problems, demonstrate the effectiveness of the proposed approach with good generalization ability.

Authors:

• S. M. Kamruzzaman
• Md. Monirul Islam

Keywords:

• Backpropagation
• clustering algorithm
• constructivealgorithm
• continuous activation function
• pruning algorithm
• ruleextraction algorithm
• symbolic rules.

References:

[1] R. Andrews, J. Diederich and A. B., Tickle, "Survey and critique of
techniques for extracting rules from trained artificial neural
networks," Knowledge Based System, vol. 8, 1995, pp. 373-389.
[2] Ashish Darbari, "Rule Extraction from Trained ANN: A Survey,"
Technical Report, Department of Computer Science, Dresden
University of Technology, Dresden, Germany, 2000.
[3] K. Saito and R. Nakano, "Medical diagnosis expert system based on
PDP model," Proceedings of IEEE International Conference on
Neutal Networks, IEEE Press, 1988, pp. 1255-1262.
[4] H. Liu and S. T. Tan, "X2R: A fast rule generator," Proceedings of
IEEE International Conference on Systems, Man and Cybernetics,
Vancouver, CA, 1995.
[5] R. Setiono and Huan Liu, "Understanding neural networks via rule
extraction," Proceedings of the 14th International Joint Conference on
Artificial Intelligence, 1995, pp. 480-485.
[6] Olcay Boz, "Knowledge integration and rule extraction in neural
networks," EECS Department, Lehigh University, 1995.
[7] R. Setiono, "Extracting rules from pruned neural networks for breast
cancer diagnosis," Artificial Intelligence in Medicine, vol. 8,
February 1996, pp. 37-51.
[8] R. Setiono and H. Liu, "Symbolic presentation of neural networks,"
IEEE Computer, March 1996, pp. 71-77.
[9] I. Taha and J. Ghosh, "Three techniques for extracting rules from
feedforward networks," Intelligent Engineering Systems Through
Artificial Neural Networks, vol. 6, pp. 23-28, ASME Press, St. Louis,
1996.
[10] R. Setiono, "Extracting rules from neural networks by pruning and
hidden-unit node splitting," Neural Computation, vol. 9, 1997, pp.
205-225.
[11] R. Setiono, "Extracting M-of-N rules from trained neural networks,"
IEEE Transactions of Neural Networks, vol. 11, pp. 512-519, 2000.
[12] R. Setiono, W. K. Leow and Jack M. Zurada, "Extraction of Rules
from Artificial Neural Networks for Nonlinear regression," IEEE
Trans. of Neural Networks, vol. 13, 2002, pp. 564-577.
[13] R. Setiono, "Techniques for extracting rules from artificial neural
networks," Plenary lecture presented at the 5th International
Conference on Soft Computing and Information Systems, Iizuka,
Japan, October 1998.
[14] R. Reed, "Pruning algorithms-A survey," IEEE Transactions on
Neural Networks, vol. 4, pp. 740-747, 1993.
[15] Han Jiawei, Micheline Kamber, "Data Mining: Concepts and
Techniques," Morgan Kaufmann Publisher: CA, 2001.
[16] L. Kaufman, P. J. Rousseeuw, "Finding Groups in Data: An
Introduction to Cluster Analysis," John Wiley & Sons, 1990.
[17] T. Ng. Raymond, Jiawei Han, "Efficient and effective clustering
methods for spatial data mining," VLDB Conference, Santiago, Chile,
1994.
[18] M. Monirul Islam and K. Murase, "A new algorithm to design
compact two hidden-layer artificial neural networks", Neural
Networks, vol. 4, 2001, pp. 1265-1278.
[19] J. R. Quinlan, "C4.5: Programs for Machine Learning," Morgan
Kaufmann, San Mateo, CA, 1993.
[20] S. Russel and P. Norvig, "Artificial Intelligence: A Modern
Approach," Prentice Hall, 1995.
[21] R. Agrawal, T. Imielinski, and A. Swami, "Database mining: A
performance perspective," IEEE Transactions on Knowledge and
Data Engineering, vol. 5, pp. 914-925, 1993.
[22] S-J Yen and A. L. P. Chen, "An efficient algorithm for deriving
compact rules from databases," Proceedings of the Fourth
International Conference on Database Systems for Advanced
Applications, 1995.
[23] Prechelt, "Proben1-A Set of Neural Network Benchmark Problems
and Benchmarking Rules", University of Karlsruhe, Germany, 1994.
[24] C. Blake, E. Keogh, and C. J. Merz, "UCI repository of of machine
learning databases
[http://www.ics.uci.edu/~mlearn/MLRepository.htm]," Department of
Information and Computer Science, University of California, Irvine,
CA, 1998.
[25] L. Breiman, J. Friedman, R. Olshen, and C. Stone, "Classification and
Regression Trees," Wadsworth and Brooks, Monterey, CA, 1984.
[26] D. T. Pham and M. S. Aksoy, "Rules: A simple rule extraction
system," Expert Systems with Applications, vol. 8, 1995.