An Experiment for Assessment of a “Functional Scenario-based“ Test Case Generation Method

Specification-based testing enables us to detect errors in the implementation of functions defined in given specifications. Its effectiveness in achieving high path coverage and efficiency in generating test cases are always major concerns of testers. The automatic test cases generation approach based on formal specifications proposed by Liu and Nakajima is aimed at ensuring high effectiveness and efficiency, but this approach has not been empirically assessed. In this paper, we present an experiment for assessing Liu-s testing approach. The result indicates that this testing approach may not be effective in some circumstances. We discuss the result, analyse the specific causes for the ineffectiveness, and describe some suggestions for improvement.

Authors:

• Cencen Li
• Shaoying Liu
• Shin Nakajima

Keywords:

• experiment
• functional scenario
• specification-based
• testing.

References:

[1] Shaoying Liu, and Shin Nakajima. A Decompositional Approach to
Automatic Test Case Generation Based on Formal Specification. Fourth
IEEE International Conference on Secure Software Integration and
Reliability Improvement, pages 147-155, 2010.
[2] J. Dick, and A. Faivre. Automating the Generation and Sequencing
of Test Cases from Model-based Specifications. In Proceedings of
FME -93: Industrial-Strength Formal Methods, pages 268-284, Odense,
Denmark, 1993. Springer-Verlag Lecture Notes in Computer Science
Volume 670.
[3] Shaoying Liu. Formal Engineering for Industrial Software Development
Using the SOFL Method. Springer-Verlag, ISBN 3-540-20602-7, 2004.
[4] M. C. Gaudel and P. Le Gall. Testing Data Types Implementation from
Algebraic Specifications. In R. Hierons, J. Bowen, and M. Harman,
editors, Formal Methods and Testing, pages 209-239. LNCS 4949,
Springer-Verlag, 2008.
[5] M. Broy, B. Jonsson, J. -P. Katoen, M. Leucker, and A. Pretschner (eds.).
Model-based Testing of Reactive Systems. LNCS 3472, Springer-Verlag,
2005.
[6] Yoonsik Cheon, and Carmen Avila. Automating Java Program Testing
Using OCL and AspectJ. 7th International Conference on Information
Technology, pages 1020-1025, 2010.
[7] Anton Michlmayr, Pascal Fenkam, and Schahram Dustdar. Specification-
Based Unit Testing of Publich/Subscribe Applications. Proceedings
of the 26th IEEE International Conference on Distributed Computing
Systems Workshops, pages 34-34, 2006.
[8] S. Khurshid, and D. Marinov. TestEra: Specification-based Testing of
Java Program using SAT. Automated Software Engineering, 11(4), pages
403-434, 2004.
[9] S. Khurshid, D. Marinov, I. Shlyakhter, and D. Jackson. A Case for
Efficient Solution Enumeration. Proceedings of SAT2003, pages 297-
298. LNCS 2919, Springer-Verlag, 2003.
[10] Aritra Bandyopadhyay, and Sudipto Ghosh. Test Input Generation using
UML Sequence and State Machines Models. International Conference
on Software Testing Verification and Validation, pages 121-130, 2009.
[11] Gordon Fraser, and Angelo Gargantini. Experiments on the Test Case
Length in Specification Based Test Case Generation. ICSE Workshop
on Automation of Software Test, pages 18-26, 2009.
[12] Shaoying Liu, and Shin Nakajima. A "Vibration" Method for Automatically
Generating Test Cases Based on Formal Specifications. 18th
Asia-Pacific Software Engineering Conference, pages 5-8, 2011.