Evaluating Psychologist Practice Competencies through Multisource Feedback: An International Research Design
Effective practicing psychologists require ongoing skill development that is constructivist and recursive in nature, with mentor, colleague, co-worker, and patient feedback critical to successful acquisition and maintenance of professional competencies. This paper will provide an overview of the nature and scope of psychologist skill development through multisource feedback (MSF) or 360 degree evaluation, present a rationale for its use for assessing practicing psychologist performance, and advocate its use in psychology given the demonstrated model utility in other health professions. The paper will conclude that an international research design is needed to assess the feasibility, reliability, and validity of MSF system ratings intended to solicit feedback from mentors, colleagues, coworkers, and patients about psychologist competencies. If adopted, the MSF model could lead to enhanced skill development that fosters patient satisfaction within and across countries.
[1] Sala, F., & Dwight, S.A. (2002). Predicting executive performance within multirater surveys: Whom you ask makes a difference. Consulting Psychology Journal: Practice and Research, 54 (3), pp. 166-172.
[2] Violato, C., Worsfold, L., & Polgar, J. M. (2009). Multisource feedback systems for quality improvement in the health professions: Assessing occupational therapists in practice. Journal of Continuing Education in the Health Professions, 29, pp. 111-118.
[3] Falender, C.A., & Shafranske, E.P. (2004). 360-degree evaluation applied to psychology training. Manuscript in preparation.
[4] Cone, J.J. (2001). Evaluating outcomes: Empirical tools for effective practice. Washington, DC: American Psychological Association.
[5] Andrews, J.W., Violato, C., Al Ansari, A., Donnon, T., & Pugliese, G. (2013). Assessing psychologists in practice: Lessons from the health professions using multisource feedback. Professional Psychology: Research and Practice, Vol 44 (4), pp. 193-207.
[6] Andrews, J. & Violato, C. (2010). The assessment of school psychologists in practice through multisource feedback. Canadian Journal of School Psychology, 25, pp. 328-346.
[7] Sharpless, B. A., & Barber, J. P. (2009). A conceptual and empirical review of the meaning, measurement, development, and teaching of intervention competence in clinical psychology. Clinical Psychology Review, 29, pp. 47-56.
[8] American Psychological Association (2002). Ethical principles of psychologists and code of conduct. American Psychologist, 57, pp. 1060–1073.
[9] Canadian Psychological Association. (2000). Canadian code of ethics for psychologists (3rd ed.). Ottawa, ON: Author.
[10] Belar, C. D., Brown, R. A., Hersch, L. E., Hornyak, L. M., Rozensky, R. H., Sheridan, E. P., … Reed, G. W. (2001). Self-assessment in clinical health psychology: A model for ethical expansion of practice. Professional Psychology: Research and Practice, 32, pp. 135-141.
[11] Pope, K. S., Sonne, J. L., & Greene, B. (2006). What therapists don't talk about and why: Understanding taboos that hurt us and our clients. Washington, DC: American Psychological Association.
[12] Caverzagie, K. J., Shea, J. A., & Kogan, J. R. (2008). Resident identification of learning objectives after performing self-assessment based upon the ACGME core competencies. Journal of General Internal Medicine, 23, pp. 1024-1027.
[13] Kaslow, N. J., Rubin, N. J., Bebeau, M., Leigh, I. W., Lichtenberg, J., Nelson, P. D., … Smith, I. L. (2007a). Guiding principles and recommendations for the assessment of competence. Professional Psychology: Research and Practice, 38, pp. 441-451.
[14] Dunning, D., Heath, C., & Suls, J. M. (2004). Flawed self-assessment: Implications for health, education, and the workplace. Psychological Science in the Public Interest, 5, pp. 69-106.
[15] Eva, K. W., Cunnington, J. P. W., Reiter, H. I., Keane, D. R., & Norman, G.R. (2004). How can I know what I don't know? Poor self-assessment in a well-defined domain. Advances in Health Sciences Education, 9, pp. 211-224.
[16] Fletcher, C. E., & Baldry, C. (2000). A study of individual differences and self-awareness in the context of multi-source feedback. Journal of Occupational and Organizational Psychology, 73, pp. 303-319.
[17] Swick, S. Hall, S., & Beresin, E. (2006). Assessing the ACGME competencies in psychiatry training programs. Academic Psychiatry, 30, pp. 330-351.
[18] Rodolfa, E., Schaffer, J. B., & Webb, C. (2010). Continuing education: The path to life –long competence? Professional Psychology: Research and Practice, 41, pp. 295-297.
[19] Nutt, R. L. (2010). Are we meeting public expectations for competence? Professional Psychology: Research and Practice, 41, pp. 294-295.
[20] Carraccio, C., Wolfsthal, S. D., Englander, R., Ferentz, K., & Martin, C. (2002). Shifting paradigms: From Flexner to competencies. Academic Medicine, 77, pp. 361-367.
[21] Roberts, M., Borden, K., Christiansen, M., & Lopez, S. (2005). Fostering a culture shift: assessment of competence in the education and careers of professional psychologists Professional Psychology: Research and Practice, 36(4), pp. 355-361.
[22] Kaslow, N. J., Grus, C. L., Campbell, L. F., Fouad, N. A., Hatcher, R. L., & Rodolfa, E. R. (2009). Competency assessment toolkit for professional psychology. Training and Education in Professional Psychology, 3, pp. S27–S45.
[23] Violato, C., & Lockyer, J. (2006). Self and peer assessment of pediatricians, psychiatrists and medicine specialists: Implications for self-directed learning. Advances in Health Sciences Education, 11, pp. 235-244.
[24] Lockyer, J., Violato, C., & Fidler, H. (2003). Likelihood of change: A study assessing surgeon use of multi-source feedback data. Teaching and Learning in Medicine, 15, pp. 168-174.
[25] Swanson, D.B., Norman, G.R. & Linn R.L. (1995). Performance based assessments: Lessons from the health professions. Educational Researcher (June), pp. 5-11.
[26] Hall, W., Violato, C., Lewkonia, R., Lockyer, J., Fidler, H., Toews, J., Moores, D. (1999). Assessment of physician performance in Alberta: The physician achievement review. Canadian Medical Association Journal, 161, pp. 52-57.
[27] Ramsey, P.G., Wenrich, M.D., Carline, J.D., Inui, T.S., Larson, E.B., & LoGerfo, J.P (1993). Use of peer ratings to evaluate physician performance. Journal of the American Medical Association. 269, pp. 1655-1660.
[28] Violato, C., Lockyer, J., & Fidler, H. (2006). Assessment of pediatricians by a regulator authority. Pediatrics, 117, pp. 796-802.
[29] Lockyer, J., Violato, C., Fidler, H., & Alakija, P. (2009). The assessment of pathologists/ laboratory medicine physicians through a multisource feedback tool. Archives of Pathology and Laboratory Medicine, 133, pp. 1301-1308.
[30] Lockyer, J., Violato, C., & Fidler, H. (2006b). A multi source feedback program for anesthesiologists. Canadian Journal of Anesthesia, 53, pp. 33-39.
[31] Kogon, S.L & Stephens, R.G. (1998). The RCDSO Quality Assurance Program: Substance or Illusion. Ontario Dentist, (March), 61.
[32] Violato, C., Lockyer, J., & Fidler, H. (2007). Assessment of psychiatrists with multisource feedback. Canadian Journal of Psychiatry, 53, pp. 525-533.
[33] Fielding, D. W. et al (1992). Assuring continuing competence: Identification and validation of a practice-based assessment blueprint. American Journal of Pharmaceutical Education, 56, pp. 21-29.
[34] Zubin, A., Marini, A., Croteau, D. & Violato. C. (2004). Assessment of pharmacists’ patient care competencies: Validity evidence from Ontario (Canada)’s Quality Assurance and Peer Review Process. Pharmacy Education, 4 (1), pp. 23–32.
[35] Schmitt, K. (1990). Testing across the nation. CLEAR Exam Review, 1, 1-3 Special Committee on Maintenance of Licensure. (2008). Draft report on maintenance of licensure. Washington, DC: Federation of State Medical Boards.
[36] Bohnen, L.S. (1994). Regulated Health Profession Act: Practical Guide. Aurora, ON Canada Law Books.
[37] Bill 25, Health Professions (Regulatory Reform) Amendment Act 2008, 4th Session, 38th Parliament of the Legislative Assembly, British Columbia Third Reading, 27th May, 2008.
[38] Federation of Medical Licensing Authorities of Canada (1994). FMLAC position paper: On the Canadian model for the monitoring and enhancement of physician performance. FMLAC: Ottawa.
[39] Atkins, P. W. B., & Wood, R. E. (2002). Self- versus others' ratings as predictors of assessment center ratings: Validation evidence for 360-degree feedback programs Personnel Psychology, 55, pp.871-904.
[40] Fletcher, C., & Bailey, C. (2003). Assessing self-awareness: Some issues and methods. Journal of Managerial Psychology, 18, pp. 395-404.
[41] Kaslow, N. J., Rubin, N. J., Forrest, L., Elman, N. S., Van Horne, B. A., Jacobs, S. C., …Thorn, B. E. (2007b). Recognizing, assessing, and intervening with problems of professional competence. Professional Psychology: Research and Practice, 38, pp. 479-492.
[42] Maurer, T. J., Mitchell, D. R. D., & Barbeite, F. G. (2002). Predictors of attitudes toward a 360-degree feedback system and involvement in post-feedback management development activity. Journal of Occupational and Organizational Psychology, 75, pp. 87-107.
[43] Rodolfa, E.E. (2005). Competency education-competent professionals-ethical practice (Special Issue). Professional Psychology: Research and Practice, 36 (4).
[44] Rodolfa, E. R., Bent, R. J., Eisman, E., Nelson, P. D., Rehm, L., & Ritchie, P. (2005). A cube model for competency development: Implications for psychology educators and regulators. Professional Psychology: Research and Practice, 36, pp. 347-354.
[1] Sala, F., & Dwight, S.A. (2002). Predicting executive performance within multirater surveys: Whom you ask makes a difference. Consulting Psychology Journal: Practice and Research, 54 (3), pp. 166-172.
[2] Violato, C., Worsfold, L., & Polgar, J. M. (2009). Multisource feedback systems for quality improvement in the health professions: Assessing occupational therapists in practice. Journal of Continuing Education in the Health Professions, 29, pp. 111-118.
[3] Falender, C.A., & Shafranske, E.P. (2004). 360-degree evaluation applied to psychology training. Manuscript in preparation.
[4] Cone, J.J. (2001). Evaluating outcomes: Empirical tools for effective practice. Washington, DC: American Psychological Association.
[5] Andrews, J.W., Violato, C., Al Ansari, A., Donnon, T., & Pugliese, G. (2013). Assessing psychologists in practice: Lessons from the health professions using multisource feedback. Professional Psychology: Research and Practice, Vol 44 (4), pp. 193-207.
[6] Andrews, J. & Violato, C. (2010). The assessment of school psychologists in practice through multisource feedback. Canadian Journal of School Psychology, 25, pp. 328-346.
[7] Sharpless, B. A., & Barber, J. P. (2009). A conceptual and empirical review of the meaning, measurement, development, and teaching of intervention competence in clinical psychology. Clinical Psychology Review, 29, pp. 47-56.
[8] American Psychological Association (2002). Ethical principles of psychologists and code of conduct. American Psychologist, 57, pp. 1060–1073.
[9] Canadian Psychological Association. (2000). Canadian code of ethics for psychologists (3rd ed.). Ottawa, ON: Author.
[10] Belar, C. D., Brown, R. A., Hersch, L. E., Hornyak, L. M., Rozensky, R. H., Sheridan, E. P., … Reed, G. W. (2001). Self-assessment in clinical health psychology: A model for ethical expansion of practice. Professional Psychology: Research and Practice, 32, pp. 135-141.
[11] Pope, K. S., Sonne, J. L., & Greene, B. (2006). What therapists don't talk about and why: Understanding taboos that hurt us and our clients. Washington, DC: American Psychological Association.
[12] Caverzagie, K. J., Shea, J. A., & Kogan, J. R. (2008). Resident identification of learning objectives after performing self-assessment based upon the ACGME core competencies. Journal of General Internal Medicine, 23, pp. 1024-1027.
[13] Kaslow, N. J., Rubin, N. J., Bebeau, M., Leigh, I. W., Lichtenberg, J., Nelson, P. D., … Smith, I. L. (2007a). Guiding principles and recommendations for the assessment of competence. Professional Psychology: Research and Practice, 38, pp. 441-451.
[14] Dunning, D., Heath, C., & Suls, J. M. (2004). Flawed self-assessment: Implications for health, education, and the workplace. Psychological Science in the Public Interest, 5, pp. 69-106.
[15] Eva, K. W., Cunnington, J. P. W., Reiter, H. I., Keane, D. R., & Norman, G.R. (2004). How can I know what I don't know? Poor self-assessment in a well-defined domain. Advances in Health Sciences Education, 9, pp. 211-224.
[16] Fletcher, C. E., & Baldry, C. (2000). A study of individual differences and self-awareness in the context of multi-source feedback. Journal of Occupational and Organizational Psychology, 73, pp. 303-319.
[17] Swick, S. Hall, S., & Beresin, E. (2006). Assessing the ACGME competencies in psychiatry training programs. Academic Psychiatry, 30, pp. 330-351.
[18] Rodolfa, E., Schaffer, J. B., & Webb, C. (2010). Continuing education: The path to life –long competence? Professional Psychology: Research and Practice, 41, pp. 295-297.
[19] Nutt, R. L. (2010). Are we meeting public expectations for competence? Professional Psychology: Research and Practice, 41, pp. 294-295.
[20] Carraccio, C., Wolfsthal, S. D., Englander, R., Ferentz, K., & Martin, C. (2002). Shifting paradigms: From Flexner to competencies. Academic Medicine, 77, pp. 361-367.
[21] Roberts, M., Borden, K., Christiansen, M., & Lopez, S. (2005). Fostering a culture shift: assessment of competence in the education and careers of professional psychologists Professional Psychology: Research and Practice, 36(4), pp. 355-361.
[22] Kaslow, N. J., Grus, C. L., Campbell, L. F., Fouad, N. A., Hatcher, R. L., & Rodolfa, E. R. (2009). Competency assessment toolkit for professional psychology. Training and Education in Professional Psychology, 3, pp. S27–S45.
[23] Violato, C., & Lockyer, J. (2006). Self and peer assessment of pediatricians, psychiatrists and medicine specialists: Implications for self-directed learning. Advances in Health Sciences Education, 11, pp. 235-244.
[24] Lockyer, J., Violato, C., & Fidler, H. (2003). Likelihood of change: A study assessing surgeon use of multi-source feedback data. Teaching and Learning in Medicine, 15, pp. 168-174.
[25] Swanson, D.B., Norman, G.R. & Linn R.L. (1995). Performance based assessments: Lessons from the health professions. Educational Researcher (June), pp. 5-11.
[26] Hall, W., Violato, C., Lewkonia, R., Lockyer, J., Fidler, H., Toews, J., Moores, D. (1999). Assessment of physician performance in Alberta: The physician achievement review. Canadian Medical Association Journal, 161, pp. 52-57.
[27] Ramsey, P.G., Wenrich, M.D., Carline, J.D., Inui, T.S., Larson, E.B., & LoGerfo, J.P (1993). Use of peer ratings to evaluate physician performance. Journal of the American Medical Association. 269, pp. 1655-1660.
[28] Violato, C., Lockyer, J., & Fidler, H. (2006). Assessment of pediatricians by a regulator authority. Pediatrics, 117, pp. 796-802.
[29] Lockyer, J., Violato, C., Fidler, H., & Alakija, P. (2009). The assessment of pathologists/ laboratory medicine physicians through a multisource feedback tool. Archives of Pathology and Laboratory Medicine, 133, pp. 1301-1308.
[30] Lockyer, J., Violato, C., & Fidler, H. (2006b). A multi source feedback program for anesthesiologists. Canadian Journal of Anesthesia, 53, pp. 33-39.
[31] Kogon, S.L & Stephens, R.G. (1998). The RCDSO Quality Assurance Program: Substance or Illusion. Ontario Dentist, (March), 61.
[32] Violato, C., Lockyer, J., & Fidler, H. (2007). Assessment of psychiatrists with multisource feedback. Canadian Journal of Psychiatry, 53, pp. 525-533.
[33] Fielding, D. W. et al (1992). Assuring continuing competence: Identification and validation of a practice-based assessment blueprint. American Journal of Pharmaceutical Education, 56, pp. 21-29.
[34] Zubin, A., Marini, A., Croteau, D. & Violato. C. (2004). Assessment of pharmacists’ patient care competencies: Validity evidence from Ontario (Canada)’s Quality Assurance and Peer Review Process. Pharmacy Education, 4 (1), pp. 23–32.
[35] Schmitt, K. (1990). Testing across the nation. CLEAR Exam Review, 1, 1-3 Special Committee on Maintenance of Licensure. (2008). Draft report on maintenance of licensure. Washington, DC: Federation of State Medical Boards.
[36] Bohnen, L.S. (1994). Regulated Health Profession Act: Practical Guide. Aurora, ON Canada Law Books.
[37] Bill 25, Health Professions (Regulatory Reform) Amendment Act 2008, 4th Session, 38th Parliament of the Legislative Assembly, British Columbia Third Reading, 27th May, 2008.
[38] Federation of Medical Licensing Authorities of Canada (1994). FMLAC position paper: On the Canadian model for the monitoring and enhancement of physician performance. FMLAC: Ottawa.
[39] Atkins, P. W. B., & Wood, R. E. (2002). Self- versus others' ratings as predictors of assessment center ratings: Validation evidence for 360-degree feedback programs Personnel Psychology, 55, pp.871-904.
[40] Fletcher, C., & Bailey, C. (2003). Assessing self-awareness: Some issues and methods. Journal of Managerial Psychology, 18, pp. 395-404.
[41] Kaslow, N. J., Rubin, N. J., Forrest, L., Elman, N. S., Van Horne, B. A., Jacobs, S. C., …Thorn, B. E. (2007b). Recognizing, assessing, and intervening with problems of professional competence. Professional Psychology: Research and Practice, 38, pp. 479-492.
[42] Maurer, T. J., Mitchell, D. R. D., & Barbeite, F. G. (2002). Predictors of attitudes toward a 360-degree feedback system and involvement in post-feedback management development activity. Journal of Occupational and Organizational Psychology, 75, pp. 87-107.
[43] Rodolfa, E.E. (2005). Competency education-competent professionals-ethical practice (Special Issue). Professional Psychology: Research and Practice, 36 (4).
[44] Rodolfa, E. R., Bent, R. J., Eisman, E., Nelson, P. D., Rehm, L., & Ritchie, P. (2005). A cube model for competency development: Implications for psychology educators and regulators. Professional Psychology: Research and Practice, 36, pp. 347-354.
@article{"International Journal of Business, Human and Social Sciences:67119", author = "Jac W. Andrews and J. B. Hale", title = "Evaluating Psychologist Practice Competencies through Multisource Feedback: An International Research Design ", abstract = "Effective practicing psychologists require ongoing skill development that is constructivist and recursive in nature, with mentor, colleague, co-worker, and patient feedback critical to successful acquisition and maintenance of professional competencies. This paper will provide an overview of the nature and scope of psychologist skill development through multisource feedback (MSF) or 360 degree evaluation, present a rationale for its use for assessing practicing psychologist performance, and advocate its use in psychology given the demonstrated model utility in other health professions. The paper will conclude that an international research design is needed to assess the feasibility, reliability, and validity of MSF system ratings intended to solicit feedback from mentors, colleagues, coworkers, and patients about psychologist competencies. If adopted, the MSF model could lead to enhanced skill development that fosters patient satisfaction within and across countries.
", keywords = "Psychologist, multisource feedback, psychologist competency, professionalism.", volume = "8", number = "5", pages = "1324-5", }
