There are some existing Java benchmarks, application benchmarks as well as micro benchmarks or mixture both of them,such as: Java Grande, Spec98, CaffeMark, HBech, etc. But none of them deal with behaviors of multi tasks operating systems. As a result, the achieved outputs are not satisfied for performance evaluation engineers. Behaviors of multi tasks operating systems are based on a schedule management which is employed in these systems. Different processes can have different priority to share the same resources. The time is measured by estimating from applications started to it is finished does not reflect the real time value which the system need for running those programs. New approach to this problem should be done. Having said that, in this paper we present a new Java benchmark, named FHOJ benchmark, which directly deals with multi tasks behaviors of a system. Our study shows that in some cases, results from FHOJ benchmark are far more reliable in comparison with some existing Java benchmarks.
[1] Q. Li and C. Yao, Real-Time Concepts for Embedded Systems. CMP Books, 2003.
[2] H. resolution timers, "http://high-res-dimers.sourceforge.net/." [Online].
Available: http://high-res-timers.sourceforge.net/
[3] J. P. Charles Daly, Jane Horgan and J. Waldron, "Platform Independent
Dynamic Java Virtual Machine Analysis: the Java Grande Forum Benchmark
Suite," In Proceedings of the 2001 joint ACM-ISCOPE conference on Java Grande, pp. 106-115, 2001.
[4] M. D. W. D. S. H. J. M. Bull, L. A. Smith and R. A. Davey, "A Methodology
for Benchmarking Java Grande Applications," In Proceedings of the ACM 1999 conference on Java Grande, pp. 81-88, 1999.
[5] P. S. Corporation, "CaffeineMark," http://www.benchmarkhq.ru/cm30/,2002.
[6] S. Org, "Spec JVM 98," http://www.spec.org/osg/jvm98, 1998.
[7] X. Zhang and M. Seltzer, "HBench: Java: An Application-Specific
Benchmarking Framework for Java Virtual Machine," In Proceedings of the ACM 2000 conference on Java Grande, 2000.
[8] R. W. Jack Dongarra and P. McMahan, "LINPACK Benchmarks,"
http://www.netlib.org/benchmark/linpackjava/.
[9] N. V. G. Chen, M. Kandemir and M. J. Irwin, "PennBench: A
Benchmark Suite for Embedded Java," IEEE International Workshop
on Workload characterization, pp. 71-80, 2002.
[10] S. M. Blackburn, R. Garner, C. Hoffmann, A. M. Khang, K. S.
McKinley, R. Bentzur, A. Diwan, D. Feinberg, D. Frampton, S. Z.
Guyer, M. Hirzel, A. Hosking, M. Jump, H. Lee, J. E. B. Moss,B. Moss, A. Phansalkar, D. Stefanovi'c, T. VanDrunen, D. von Dincklage, and B. Wiedermann, "The DaCapo benchmarks: Java benchmarking development and analysis," in OOPSLA -06: Proceedings of the 21st
annual ACM SIGPLAN conference on Object-oriented programming
systems, languages, and applications. New York, NY, USA: ACM, 2006, pp. 169-190.
[11] S. M. Blackburn, K. S. McKinley, R. Garner, C. Hoffmann, A. M. Khan, R. Bentzur, A. Diwan, D. Feinberg, D. Frampton, S. Z. Guyer, M. Hirzel, A. Hosking, M. Jump, H. Lee, J. E. B. Moss, A. Phansalkar,
D. Stefanovik, T. VanDrunen, D. von Dincklage, and B. Wiedermann, "Wake up and smell the coffee: evaluation methodology for the 21st
century," Commun. ACM, vol. 51, no. 8, pp. 83-89, 2008.
[12] M. S. Shuichi Oikawa and T. Nakajima, "Accounting system: a finegrained
CPU resource protection mechanism for embedded system," In Proceedings of the Ninth IEEE International Symposium on Object and
Component-Oriented Real-Time Distributed Computing, 2006.
[13] G. M. I. Bate, G.Bernat and P. Puschner, "Low-Level Analysis of a Portable Java Byte Code WCET Analysis Framework," In Proceedings
of the Seventh International Conference on Real-Time Systems and Applications, 2000.
[14] Jack Shirazi and Kirk Pepperdine, "Eye on performance: Micro performance benchmarking," IBM journal, 2003.
[15] Jack Shizazi and Kirk Pepperdine, "Eye on performance: When good
benchmark go bad," IBM journal, 2005.
[16] A. G. Lieve Eeckhout and K. D. Bosschere, "How Java Programs Interact with Virtual Machines at the Microarchitectural Level," In
Proceedings of the OOPSLA 03 conference, 2003.
[17] R. Pozo and B. Miller, "SciMark 2.0 Benchmark,"
http://math.nist.gov/scimark2, 2004.
[18] M. G. Yefim Shuf, Maurico J. Serrano and J. P. Singh, "Characterizing
the Memory Behavior of Java Workloads: A Structured View and Opportunities for Optimizations," ACM Sigmetrics, 2001.
[19] S. M. Y. T. T. Y. Morgan Hirosuke Miki, Mamoru Sakamoto and
I. Shirakawa, "Evaluation of Processor Code Efficiency for Embedded Systems," In Proceedings of the 15th international conference on
Supercomputing, 2001.
[20] V. Q. La, "Design Virtual Machine for Java Processing for a Small Embedded Microprocessor Core," Master-s thesis, University Of Applied Sciences Offenburg, 2007.
[21] V.-Q. La, "A study on Java Virtual Machine for Real-time embedded systems," IEEE International Conference on Computer Science and Software Engineering (CSSE 2008), Accepted 2008.
[1] Q. Li and C. Yao, Real-Time Concepts for Embedded Systems. CMP Books, 2003.
[2] H. resolution timers, "http://high-res-dimers.sourceforge.net/." [Online].
Available: http://high-res-timers.sourceforge.net/
[3] J. P. Charles Daly, Jane Horgan and J. Waldron, "Platform Independent
Dynamic Java Virtual Machine Analysis: the Java Grande Forum Benchmark
Suite," In Proceedings of the 2001 joint ACM-ISCOPE conference on Java Grande, pp. 106-115, 2001.
[4] M. D. W. D. S. H. J. M. Bull, L. A. Smith and R. A. Davey, "A Methodology
for Benchmarking Java Grande Applications," In Proceedings of the ACM 1999 conference on Java Grande, pp. 81-88, 1999.
[5] P. S. Corporation, "CaffeineMark," http://www.benchmarkhq.ru/cm30/,2002.
[6] S. Org, "Spec JVM 98," http://www.spec.org/osg/jvm98, 1998.
[7] X. Zhang and M. Seltzer, "HBench: Java: An Application-Specific
Benchmarking Framework for Java Virtual Machine," In Proceedings of the ACM 2000 conference on Java Grande, 2000.
[8] R. W. Jack Dongarra and P. McMahan, "LINPACK Benchmarks,"
http://www.netlib.org/benchmark/linpackjava/.
[9] N. V. G. Chen, M. Kandemir and M. J. Irwin, "PennBench: A
Benchmark Suite for Embedded Java," IEEE International Workshop
on Workload characterization, pp. 71-80, 2002.
[10] S. M. Blackburn, R. Garner, C. Hoffmann, A. M. Khang, K. S.
McKinley, R. Bentzur, A. Diwan, D. Feinberg, D. Frampton, S. Z.
Guyer, M. Hirzel, A. Hosking, M. Jump, H. Lee, J. E. B. Moss,B. Moss, A. Phansalkar, D. Stefanovi'c, T. VanDrunen, D. von Dincklage, and B. Wiedermann, "The DaCapo benchmarks: Java benchmarking development and analysis," in OOPSLA -06: Proceedings of the 21st
annual ACM SIGPLAN conference on Object-oriented programming
systems, languages, and applications. New York, NY, USA: ACM, 2006, pp. 169-190.
[11] S. M. Blackburn, K. S. McKinley, R. Garner, C. Hoffmann, A. M. Khan, R. Bentzur, A. Diwan, D. Feinberg, D. Frampton, S. Z. Guyer, M. Hirzel, A. Hosking, M. Jump, H. Lee, J. E. B. Moss, A. Phansalkar,
D. Stefanovik, T. VanDrunen, D. von Dincklage, and B. Wiedermann, "Wake up and smell the coffee: evaluation methodology for the 21st
century," Commun. ACM, vol. 51, no. 8, pp. 83-89, 2008.
[12] M. S. Shuichi Oikawa and T. Nakajima, "Accounting system: a finegrained
CPU resource protection mechanism for embedded system," In Proceedings of the Ninth IEEE International Symposium on Object and
Component-Oriented Real-Time Distributed Computing, 2006.
[13] G. M. I. Bate, G.Bernat and P. Puschner, "Low-Level Analysis of a Portable Java Byte Code WCET Analysis Framework," In Proceedings
of the Seventh International Conference on Real-Time Systems and Applications, 2000.
[14] Jack Shirazi and Kirk Pepperdine, "Eye on performance: Micro performance benchmarking," IBM journal, 2003.
[15] Jack Shizazi and Kirk Pepperdine, "Eye on performance: When good
benchmark go bad," IBM journal, 2005.
[16] A. G. Lieve Eeckhout and K. D. Bosschere, "How Java Programs Interact with Virtual Machines at the Microarchitectural Level," In
Proceedings of the OOPSLA 03 conference, 2003.
[17] R. Pozo and B. Miller, "SciMark 2.0 Benchmark,"
http://math.nist.gov/scimark2, 2004.
[18] M. G. Yefim Shuf, Maurico J. Serrano and J. P. Singh, "Characterizing
the Memory Behavior of Java Workloads: A Structured View and Opportunities for Optimizations," ACM Sigmetrics, 2001.
[19] S. M. Y. T. T. Y. Morgan Hirosuke Miki, Mamoru Sakamoto and
I. Shirakawa, "Evaluation of Processor Code Efficiency for Embedded Systems," In Proceedings of the 15th international conference on
Supercomputing, 2001.
[20] V. Q. La, "Design Virtual Machine for Java Processing for a Small Embedded Microprocessor Core," Master-s thesis, University Of Applied Sciences Offenburg, 2007.
[21] V.-Q. La, "A study on Java Virtual Machine for Real-time embedded systems," IEEE International Conference on Computer Science and Software Engineering (CSSE 2008), Accepted 2008.
@article{"International Journal of Information, Control and Computer Sciences:54721", author = "Vinh Quang La and Dirk Jansen", title = "FHOJ: A New Java Benchmark Framework", abstract = "There are some existing Java benchmarks, application benchmarks as well as micro benchmarks or mixture both of them,such as: Java Grande, Spec98, CaffeMark, HBech, etc. But none of them deal with behaviors of multi tasks operating systems. As a result, the achieved outputs are not satisfied for performance evaluation engineers. Behaviors of multi tasks operating systems are based on a schedule management which is employed in these systems. Different processes can have different priority to share the same resources. The time is measured by estimating from applications started to it is finished does not reflect the real time value which the system need for running those programs. New approach to this problem should be done. Having said that, in this paper we present a new Java benchmark, named FHOJ benchmark, which directly deals with multi tasks behaviors of a system. Our study shows that in some cases, results from FHOJ benchmark are far more reliable in comparison with some existing Java benchmarks.
", keywords = "Java Virtual Machine, Java benchmark, FHOJ framework.", volume = "3", number = "1", pages = "80-7", }
