The Reproducibility and Repeatability of Modified Likelihood Ratio for Forensics Handwriting Examination
The forensic use of handwriting depends on the analysis, comparison, and evaluation decisions made by forensic document examiners. When using biometric technology in forensic applications, it is necessary to compute Likelihood Ratio (LR) for quantifying strength of evidence under two competing hypotheses, namely the prosecution and the defense hypotheses wherein a set of assumptions and methods for a given data set will be made. It is therefore important to know how repeatable and reproducible our estimated LR is. This paper evaluated the accuracy and reproducibility of examiners' decisions. Confidence interval for the estimated LR were presented so as not get an incorrect estimate that will be used to deliver wrong judgment in the court of Law. The estimate of LR is fundamentally a Bayesian concept and we used two LR estimators, namely Logistic Regression (LoR) and Kernel Density Estimator (KDE) for this paper. The repeatability evaluation was carried out by retesting the initial experiment after an interval of six months to observe whether examiners would repeat their decisions for the estimated LR. The experimental results, which are based on handwriting dataset, show that LR has different confidence intervals which therefore implies that LR cannot be estimated with the same certainty everywhere. Though the LoR performed better than the KDE when tested using the same dataset, the two LR estimators investigated showed a consistent region in which LR value can be estimated confidently. These two findings advance our understanding of LR when used in computing the strength of evidence in handwriting using forensics.
[1] R. Hasting. From grainy cctv to a positive id: Recognizing the benefits" of surveillance. The Independent, 2013.
[2] I. Bouchrika, M. Goffredo, J. Carter, and M. Nixon. On using gait in forensic biometrics. Journal of Forensic Sciences, 56(4):882–889, 2011.
[3] P. K. Larsen, E. B. Simonsen, and N. Lynnerup. Gait analysis in forensic medicine*. Journal of Forensic Sciences, 53(5):1149–1153, 2008.
[4] I.E. Evett, R.L. Williams. A review of the sixteen points fingerprint standard in England and Wales. J Forensic Identification.1996;46:49–73. Available:http://www.thefingerprintinquiryscotland.org.uk/inquiry/files/DB_0769-02.pdf.
[5] S. Gutowski. Error rates in fingerprint examination: the view in 2006. The Forensic Bulletin Autumn. 2006;2006:18–19.
[6] G. Langenburg. A Performance study of the ACE-V process: a pilot study to measure the accuracy, precision, reproducibility, and the biasability of conclusions resulting from the ACE-V process. J Forensic Identification. 2009;59(2):219–257.
[7] B.T. Ulery, R.A. Hicklin, J. Buscaglia, M.A. Roberts. Accuracy and reliability of forensic latent fingerprint decisions. Proc Natl Acad Sci USA. 2011;108(19):7733–7738. Available: http://www.pnas.org/content/108/19/7733.full.pdf. (PMC free article) (PubMed)
[8] S. M. Stigler. In The History of Statistics: The Measurement of Uncertainty before 1900. Cambridge, MA: Belknap of Harvard UP, 1986.
[9] A.B. Hepler, C.P. Saunders, L.J. Davis, J. Buscaglia. Score-based likelihood ratios for handwriting evidence. Forensic Science International, 219: 129140. doi:10.1016/j.forsciint.2011.12.009, 2012.
[10] A. Mishra. (2017). Forensic Graphology : Assessment of Personality, 4(1), 1–4. https://doi.org/10.15406/frcij.2017.04.00097
[11] C.G.G. Aitken, D. Lucy, Evaluation of trace evidence in the form of multivariate data, J. R. Stat. Soc. Ser. C: Appl. Stat. 53 (2004) 109–122.
[12] S. Bozza, F. Taroni, R. Marquis, M. Schmittbuhl, Probabilistic evaluation of handwriting evidence: likelihood ratio for authorship, J. R. Stat. Soc. Ser. C: Appl. Stat. 57 (2008) 329–341.
[13] R. Marquis, S. Bozza, M. Schmittbuhl, F. Taroni, Handwriting evidence evaluation based on the shape of characters: application of multivariate likelihood ratios, J. Forensic Sci. 56 (2011) S238–S242.
[14] S.N. Srihari, S.H. Cha, H. Arora, S. Lee, Individuality of handwriting, J. Forensic Sci. 47 (2002) 856–872.
[15] T. Ali, L. J. Spreeuwers, and R. N. J. Veldhuis. A review of calibration methods for biometric systems in forensic applications. In 33rd WIC Symposium on Information Theory in the Benelux, Boekelo, Netherlands, pages 126–133, Enschede, May 2012. WIC.
[16] N. Suki, N. Poh, F. M. Senan, N. A. Zamani, M. Z. A. Darus, On the reproducibility and repeatability of likelihood ratio in forensics: A case study using face biometrics, 2016.
[17] B.T. Ulery, R.A. Hicklin, J. Buscaglia, & M.A. Roberts. (2012). Repeatability and reproducibility of decisions by latent fingerprint examiners. PLoS ONE, 7(3). Article ID e32800. http://dx.doi.org/10.1371/journal.pone.0032800
[18] F. Taroni, S. Bozza, A. Biedermann, & C. Aitken. Dismissal of the illusion of uncertainty in the assessment of a likelihood ratio. Law, Probability and Risk, 2015.
[19] J. M. Curran. Statistics in forensic science. Wiley Interdisciplinary Reviews: Computational Statistics, 1(2):141–156, 2009.
[20] J. Buckleton, C. Triggs, and C. Champod. An extended likelihood ratio framework for interpreting evidence. Science & Justice, 46(2):69 – 78, 2006.
[21] C. Champod and D. Meuwly. The inference of identity in forensic speaker recognition. Speech Communication, 31(23):193 –203, 2000.
[22] J. Gonzalez-Rodriguez, J. Fierrez-Aguilar, D. Ramos-Castro, and J. Ortega-Garcia. Bayesian analysis of fingerprint, face and signature evidences with automatic biometric systems. Forensic Science International, 155(23):126 – 140, 2005.
[23] N. Brummer, L. Burget, J. Cernocky, O. Glembek, F. Grezl, M. Karafiat, D. van Leeuwen, P. Matejka, P. Schwarz, and A. Strasheim. Fusion of heterogeneous speaker recognition systems in the stbu submission for the nist speaker recognition evaluation 2006. Audio, Speech, and Language Processing, IEEE Transactions on, 15(7):2072–2084, Sept 2007.
[24] S. Pigeon, P. Druyts, and P. Verlinde. Applying logistic regression to the fusion of the nist’99 1-speaker submissions. Digital Signal Processing, 10(13):237 – 248, 2000.
[25] J. Gonzalez-Rodriguez, P. Rose, D. Ramos, D. Toledano, and J. Ortega-Garcia. Emulating dna: Rigorous quantification of evidential weight in transparent and testable forensic speaker recognition. Audio, Speech, and Language Processing, IEEE Transactions on, 15(7):2104–2115, Sept 2007.
[26] N. Brmmer and J. du Preez. Application-independent evaluation of speaker detection. Computer Speech & Language, 20(23):230– 275, 2006. Odyssey 2004: The speaker and Language Recognition Workshop Odyssey-04 Odyssey 2004: The speaker and Language Recognition Workshop.
[27] A.O. Abiodun, A.B. Adeyemo. (2019) An Exhaustive Mapping Model for Modified Likelihood Ratio for Handwriting Recognition in Forensic Science. J Forensic Sci Criminol 7(3): 301 ISSN: 2348-9804
[1] R. Hasting. From grainy cctv to a positive id: Recognizing the benefits" of surveillance. The Independent, 2013.
[2] I. Bouchrika, M. Goffredo, J. Carter, and M. Nixon. On using gait in forensic biometrics. Journal of Forensic Sciences, 56(4):882–889, 2011.
[3] P. K. Larsen, E. B. Simonsen, and N. Lynnerup. Gait analysis in forensic medicine*. Journal of Forensic Sciences, 53(5):1149–1153, 2008.
[4] I.E. Evett, R.L. Williams. A review of the sixteen points fingerprint standard in England and Wales. J Forensic Identification.1996;46:49–73. Available:http://www.thefingerprintinquiryscotland.org.uk/inquiry/files/DB_0769-02.pdf.
[5] S. Gutowski. Error rates in fingerprint examination: the view in 2006. The Forensic Bulletin Autumn. 2006;2006:18–19.
[6] G. Langenburg. A Performance study of the ACE-V process: a pilot study to measure the accuracy, precision, reproducibility, and the biasability of conclusions resulting from the ACE-V process. J Forensic Identification. 2009;59(2):219–257.
[7] B.T. Ulery, R.A. Hicklin, J. Buscaglia, M.A. Roberts. Accuracy and reliability of forensic latent fingerprint decisions. Proc Natl Acad Sci USA. 2011;108(19):7733–7738. Available: http://www.pnas.org/content/108/19/7733.full.pdf. (PMC free article) (PubMed)
[8] S. M. Stigler. In The History of Statistics: The Measurement of Uncertainty before 1900. Cambridge, MA: Belknap of Harvard UP, 1986.
[9] A.B. Hepler, C.P. Saunders, L.J. Davis, J. Buscaglia. Score-based likelihood ratios for handwriting evidence. Forensic Science International, 219: 129140. doi:10.1016/j.forsciint.2011.12.009, 2012.
[10] A. Mishra. (2017). Forensic Graphology : Assessment of Personality, 4(1), 1–4. https://doi.org/10.15406/frcij.2017.04.00097
[11] C.G.G. Aitken, D. Lucy, Evaluation of trace evidence in the form of multivariate data, J. R. Stat. Soc. Ser. C: Appl. Stat. 53 (2004) 109–122.
[12] S. Bozza, F. Taroni, R. Marquis, M. Schmittbuhl, Probabilistic evaluation of handwriting evidence: likelihood ratio for authorship, J. R. Stat. Soc. Ser. C: Appl. Stat. 57 (2008) 329–341.
[13] R. Marquis, S. Bozza, M. Schmittbuhl, F. Taroni, Handwriting evidence evaluation based on the shape of characters: application of multivariate likelihood ratios, J. Forensic Sci. 56 (2011) S238–S242.
[14] S.N. Srihari, S.H. Cha, H. Arora, S. Lee, Individuality of handwriting, J. Forensic Sci. 47 (2002) 856–872.
[15] T. Ali, L. J. Spreeuwers, and R. N. J. Veldhuis. A review of calibration methods for biometric systems in forensic applications. In 33rd WIC Symposium on Information Theory in the Benelux, Boekelo, Netherlands, pages 126–133, Enschede, May 2012. WIC.
[16] N. Suki, N. Poh, F. M. Senan, N. A. Zamani, M. Z. A. Darus, On the reproducibility and repeatability of likelihood ratio in forensics: A case study using face biometrics, 2016.
[17] B.T. Ulery, R.A. Hicklin, J. Buscaglia, & M.A. Roberts. (2012). Repeatability and reproducibility of decisions by latent fingerprint examiners. PLoS ONE, 7(3). Article ID e32800. http://dx.doi.org/10.1371/journal.pone.0032800
[18] F. Taroni, S. Bozza, A. Biedermann, & C. Aitken. Dismissal of the illusion of uncertainty in the assessment of a likelihood ratio. Law, Probability and Risk, 2015.
[19] J. M. Curran. Statistics in forensic science. Wiley Interdisciplinary Reviews: Computational Statistics, 1(2):141–156, 2009.
[20] J. Buckleton, C. Triggs, and C. Champod. An extended likelihood ratio framework for interpreting evidence. Science & Justice, 46(2):69 – 78, 2006.
[21] C. Champod and D. Meuwly. The inference of identity in forensic speaker recognition. Speech Communication, 31(23):193 –203, 2000.
[22] J. Gonzalez-Rodriguez, J. Fierrez-Aguilar, D. Ramos-Castro, and J. Ortega-Garcia. Bayesian analysis of fingerprint, face and signature evidences with automatic biometric systems. Forensic Science International, 155(23):126 – 140, 2005.
[23] N. Brummer, L. Burget, J. Cernocky, O. Glembek, F. Grezl, M. Karafiat, D. van Leeuwen, P. Matejka, P. Schwarz, and A. Strasheim. Fusion of heterogeneous speaker recognition systems in the stbu submission for the nist speaker recognition evaluation 2006. Audio, Speech, and Language Processing, IEEE Transactions on, 15(7):2072–2084, Sept 2007.
[24] S. Pigeon, P. Druyts, and P. Verlinde. Applying logistic regression to the fusion of the nist’99 1-speaker submissions. Digital Signal Processing, 10(13):237 – 248, 2000.
[25] J. Gonzalez-Rodriguez, P. Rose, D. Ramos, D. Toledano, and J. Ortega-Garcia. Emulating dna: Rigorous quantification of evidential weight in transparent and testable forensic speaker recognition. Audio, Speech, and Language Processing, IEEE Transactions on, 15(7):2104–2115, Sept 2007.
[26] N. Brmmer and J. du Preez. Application-independent evaluation of speaker detection. Computer Speech & Language, 20(23):230– 275, 2006. Odyssey 2004: The speaker and Language Recognition Workshop Odyssey-04 Odyssey 2004: The speaker and Language Recognition Workshop.
[27] A.O. Abiodun, A.B. Adeyemo. (2019) An Exhaustive Mapping Model for Modified Likelihood Ratio for Handwriting Recognition in Forensic Science. J Forensic Sci Criminol 7(3): 301 ISSN: 2348-9804
@article{"International Journal of Information, Control and Computer Sciences:80319", author = "O. Abiodun Adeyinka and B. Adeyemo Adesesan", title = "The Reproducibility and Repeatability of Modified Likelihood Ratio for Forensics Handwriting Examination", abstract = "The forensic use of handwriting depends on the analysis, comparison, and evaluation decisions made by forensic document examiners. When using biometric technology in forensic applications, it is necessary to compute Likelihood Ratio (LR) for quantifying strength of evidence under two competing hypotheses, namely the prosecution and the defense hypotheses wherein a set of assumptions and methods for a given data set will be made. It is therefore important to know how repeatable and reproducible our estimated LR is. This paper evaluated the accuracy and reproducibility of examiners' decisions. Confidence interval for the estimated LR were presented so as not get an incorrect estimate that will be used to deliver wrong judgment in the court of Law. The estimate of LR is fundamentally a Bayesian concept and we used two LR estimators, namely Logistic Regression (LoR) and Kernel Density Estimator (KDE) for this paper. The repeatability evaluation was carried out by retesting the initial experiment after an interval of six months to observe whether examiners would repeat their decisions for the estimated LR. The experimental results, which are based on handwriting dataset, show that LR has different confidence intervals which therefore implies that LR cannot be estimated with the same certainty everywhere. Though the LoR performed better than the KDE when tested using the same dataset, the two LR estimators investigated showed a consistent region in which LR value can be estimated confidently. These two findings advance our understanding of LR when used in computing the strength of evidence in handwriting using forensics.", keywords = "Logistic Regression LoR, Kernel Density Estimator KDE, Handwriting, Confidence Interval, Repeatability, Reproducibility.", volume = "15", number = "5", pages = "182-7", }
