Eclectic Rule-Extraction from Support Vector Machines
Support vector machines (SVMs) have shown
superior performance compared to other machine learning techniques,
especially in classification problems. Yet one limitation of SVMs is
the lack of an explanation capability which is crucial in some
applications, e.g. in the medical and security domains. In this paper, a
novel approach for eclectic rule-extraction from support vector
machines is presented. This approach utilizes the knowledge acquired
by the SVM and represented in its support vectors as well as the
parameters associated with them. The approach includes three stages;
training, propositional rule-extraction and rule quality evaluation.
Results from four different experiments have demonstrated the value
of the approach for extracting comprehensible rules of high accuracy
and fidelity.
[1] A.B. Tickle, R.Andrews, M.Golea, and J.Diederich, "The truth will come
to light: directions and challenges in extracting the knowledge embedded
within trained artificial neural network", IEEE Trans. Neural Networks,
vol. 9(6), pp. 1057-1068, 1998.
[2] R. Andrews, J. Diederich, and A.B. Tickle, "A Survey and Critique of
Techniques For Extracting Rules From Trained Artificial Neural Networks",
Knowledge Based Systems, vol. 8, pp. 373-389, 1995.
[3] R. Davis, B.G. Buchanan, and E. Shortcliff, "Production Rules as a
Representation for a Knowledge Based Consultation Progra", J. Artificial
Intelligence, vol. 8(1), pp.15-45, 1977.
[4] S. Gallant, "Connectionist Expert System", Communications of the ACM,
vol. 31 (2), pp. 152-169, 1988.
[5] S. Sestito and T. Dillon, "Automated Knowledge Acquisition of Rules
With Continuously Valued Attributes", in Proc.12th International
Conference on Expert Systems and their Applications (AVIGNON'92),
Avignon -France, 1992, pp. 645-656.
[6] M.W. Craven, and J.W. Shavlik, "Using Sampling and Queries to Extract
Rules From Trained Neural Networks", in Proc. of the 11th International
Conference on Machine learning, NJ, 1994, pp.37-45.
[7] G. Towell, and J. Shavlik. "The Extraction of Refined Rules From
Knowledge Based Neural Networks", J. Machine Learning, vol. 131,
pp.71-101, 1993.
[8] M.W. Craven, and J.W. Shavlik, "Extracting Tree-Structured
Representation of Trained Networks", Advances in Neural Information
Processing Systems, vol. 8, pp.24-30, 1996.
[9] A. Tickle, A, M. Orlowski, M, J. Diederich, "DEDEC: A Methodology
for Extracting Rules from Trained Artificial Neural Networks. "In:
Andrews, R.; Diederich, J. (Eds.): Rules and Networks. Brisbane, Qld.:
QUT Publication 1996, 90-102.
[10] R. Mitsdorffer, J. Diederich, and C. Tan, "Rule-extraction from
Technology IPOs in the US Stock Market", presented at ICONIP02,
Singapore, 2002.
[11] H. Khuu, H.K. Lee, J-L, Tsai. " Machine learning with Neural Networks
and support vector machines", University of Wisconsin, unpublished,
2004
[12] C. Burges, A tutorial on support vector machines for pattern recognition.
data mining and knowledge discovery, Boston, Kluwer Academic
publishers, 1998.
[13] V. Kecman, Learning and Soft Computing. Cambridge, MA: MIT Press,
2001
[14] V. Kecman, "Learning by Support Vector Machines from Huge Data
Sets", presented at KES 2004, Eighth international conference on
knowledge-based intelligent information & engineering systems, 20-24
September, 2004, Wellington, New Zeland.
[15] H. N├║├▒ez, C. Angulo, and A.Catala, "Rule-extraction from Support
Vector Machines", in Proc. of European Symposium on Artificial Neural
Networks, Burges, 2002, pp.107-112.
[16] N. Barakat , and J. Diederich, "Learning-based rule-extraction from
support vector machines: Performance on benchmark data sets":
Kasabov, N., Chan, Z.S.H. (Eds.), in Proc. of the conference on Neuro-
Computing and Evolving Intelligence, Auckland, New Zealand,
Auckland. Knowledge Engineering and Discovery Research Institute
(KEDRI) (2004).
[17] J. Diederich , and N. Barakat, "Hybrid rule-extraction from support
vector machines" in Proc. of IEEE conference on cybernetics and
intelligent systems, Singapore, 2004, pp. 1270-1275.
[18] http://www.rulequest.com
[19] http://www.ics.uci.edu/~mlearn/MLRepository.html
[20] http://svmlight.joachims.org/
[21] M. Craven and J. Shavlik, "Rule Extraction: Where Do We Go from
Here?", Department of Computer Sciences, Machine Learning Research
Group Working Paper 99-1, 1999.
[22] A.Tickel, F. Maire, G. Bologna, J. Diederich." Lessons from past, current
issues and future research directions in extracting the knowledge
embedded in Artificial Neural Networks". Lecture notes in computer
science, Hybrid Neural Systems, vol. 1778, revised papers from a
workshop 1998, pp. 226 - 239
[1] A.B. Tickle, R.Andrews, M.Golea, and J.Diederich, "The truth will come
to light: directions and challenges in extracting the knowledge embedded
within trained artificial neural network", IEEE Trans. Neural Networks,
vol. 9(6), pp. 1057-1068, 1998.
[2] R. Andrews, J. Diederich, and A.B. Tickle, "A Survey and Critique of
Techniques For Extracting Rules From Trained Artificial Neural Networks",
Knowledge Based Systems, vol. 8, pp. 373-389, 1995.
[3] R. Davis, B.G. Buchanan, and E. Shortcliff, "Production Rules as a
Representation for a Knowledge Based Consultation Progra", J. Artificial
Intelligence, vol. 8(1), pp.15-45, 1977.
[4] S. Gallant, "Connectionist Expert System", Communications of the ACM,
vol. 31 (2), pp. 152-169, 1988.
[5] S. Sestito and T. Dillon, "Automated Knowledge Acquisition of Rules
With Continuously Valued Attributes", in Proc.12th International
Conference on Expert Systems and their Applications (AVIGNON'92),
Avignon -France, 1992, pp. 645-656.
[6] M.W. Craven, and J.W. Shavlik, "Using Sampling and Queries to Extract
Rules From Trained Neural Networks", in Proc. of the 11th International
Conference on Machine learning, NJ, 1994, pp.37-45.
[7] G. Towell, and J. Shavlik. "The Extraction of Refined Rules From
Knowledge Based Neural Networks", J. Machine Learning, vol. 131,
pp.71-101, 1993.
[8] M.W. Craven, and J.W. Shavlik, "Extracting Tree-Structured
Representation of Trained Networks", Advances in Neural Information
Processing Systems, vol. 8, pp.24-30, 1996.
[9] A. Tickle, A, M. Orlowski, M, J. Diederich, "DEDEC: A Methodology
for Extracting Rules from Trained Artificial Neural Networks. "In:
Andrews, R.; Diederich, J. (Eds.): Rules and Networks. Brisbane, Qld.:
QUT Publication 1996, 90-102.
[10] R. Mitsdorffer, J. Diederich, and C. Tan, "Rule-extraction from
Technology IPOs in the US Stock Market", presented at ICONIP02,
Singapore, 2002.
[11] H. Khuu, H.K. Lee, J-L, Tsai. " Machine learning with Neural Networks
and support vector machines", University of Wisconsin, unpublished,
2004
[12] C. Burges, A tutorial on support vector machines for pattern recognition.
data mining and knowledge discovery, Boston, Kluwer Academic
publishers, 1998.
[13] V. Kecman, Learning and Soft Computing. Cambridge, MA: MIT Press,
2001
[14] V. Kecman, "Learning by Support Vector Machines from Huge Data
Sets", presented at KES 2004, Eighth international conference on
knowledge-based intelligent information & engineering systems, 20-24
September, 2004, Wellington, New Zeland.
[15] H. N├║├▒ez, C. Angulo, and A.Catala, "Rule-extraction from Support
Vector Machines", in Proc. of European Symposium on Artificial Neural
Networks, Burges, 2002, pp.107-112.
[16] N. Barakat , and J. Diederich, "Learning-based rule-extraction from
support vector machines: Performance on benchmark data sets":
Kasabov, N., Chan, Z.S.H. (Eds.), in Proc. of the conference on Neuro-
Computing and Evolving Intelligence, Auckland, New Zealand,
Auckland. Knowledge Engineering and Discovery Research Institute
(KEDRI) (2004).
[17] J. Diederich , and N. Barakat, "Hybrid rule-extraction from support
vector machines" in Proc. of IEEE conference on cybernetics and
intelligent systems, Singapore, 2004, pp. 1270-1275.
[18] http://www.rulequest.com
[19] http://www.ics.uci.edu/~mlearn/MLRepository.html
[20] http://svmlight.joachims.org/
[21] M. Craven and J. Shavlik, "Rule Extraction: Where Do We Go from
Here?", Department of Computer Sciences, Machine Learning Research
Group Working Paper 99-1, 1999.
[22] A.Tickel, F. Maire, G. Bologna, J. Diederich." Lessons from past, current
issues and future research directions in extracting the knowledge
embedded in Artificial Neural Networks". Lecture notes in computer
science, Hybrid Neural Systems, vol. 1778, revised papers from a
workshop 1998, pp. 226 - 239
@article{"International Journal of Information, Control and Computer Sciences:50742", author = "Nahla Barakat and Joachim Diederich", title = "Eclectic Rule-Extraction from Support Vector Machines", abstract = "Support vector machines (SVMs) have shown
superior performance compared to other machine learning techniques,
especially in classification problems. Yet one limitation of SVMs is
the lack of an explanation capability which is crucial in some
applications, e.g. in the medical and security domains. In this paper, a
novel approach for eclectic rule-extraction from support vector
machines is presented. This approach utilizes the knowledge acquired
by the SVM and represented in its support vectors as well as the
parameters associated with them. The approach includes three stages;
training, propositional rule-extraction and rule quality evaluation.
Results from four different experiments have demonstrated the value
of the approach for extracting comprehensible rules of high accuracy
and fidelity.", keywords = "Data mining, hybrid rule-extraction algorithms,
medical diagnosis, SVMs", volume = "2", number = "5", pages = "1356-4", }
