Abstract: Many measures have been proposed for machine
translation evaluation (MTE) while little research has been done on
the performance of MTE methods. This paper is an effort for MTE
performance analysis. A general frame is proposed for the description
of the MTE measure and the test suite, including whether the
automatic measure is consistent with human evaluation, whether
different results from various measures or test suites are consistent,
whether the content of the test suite is suitable for performance
evaluation, the degree of difficulty of the test suite and its influence
on the MTE, the relationship of MTE result significance and the size
of the test suite, etc. For a better clarification of the frame, several
experiment results are analyzed relating human evaluation, BLEU
evaluation, and typological MTE. A visualization method is
introduced for better presentation of the results. The study aims for
aid in construction of test suite and method selection in MTE
practice.