The Assessment of School Psychologists in Practice Through Multisource Feedback

Abstract

In this article we provide an overview of the nature and scope of multisource feedback (MSF) and provide empirical evidence of its reliability, validity, and feasibility in one of the health professions. The overall internal consistency reliability (Cronbach alpha) of MSF instruments is generally greater than .96 for self and informants such as patients, coworker, and colleague surveys. Generalizability coefficients for the assessors across persons are approximately 0.80. There is also substantial evidence of content, criterion-related and some evidence of construct validity of the MSF instruments applied in the health professions. Based on these findings, we recommend the development and use of a MSF system for practicing school psychologists, present information about how MSF instruments can be constructed, and provide examples of what these instruments could look like.

Cet article vous offre un aperçu de la nature et de la portée des multisource feedback (MSF) et vous fournit des preuves empiriques de sa fiabilité, de sa validité et la faisabilité dans une des professions de la santé. La fiabilité de la cohérence interne globale (Cronbach alpha) des outils de MSF est généralement plus grande que, 96 pour des sondages effectués sur soi-même, sur des individus tels que des patients, des collègues de travail ou collaborateurs. Les coefficients de généralisibilité pour les évaluateurs à travers les personnes sont d’environ 0,80. Il y a aussi des preuves substantielles du contenu, les critères liés et certains éléments de preuve de la validité des outils MSF appliqués dans les professions de la santé. Sur la base de ces résultats, nous recommandons l’élaboration et l’utilisation d’un système de MSF pour la pratique de la psychologie en milieu scolaire; présenter des informations sur la façon dont les outils MSF peuvent être développés et fournir des exemples de ce que ces outils peuvent être.

Keywords

multisource feedback school psychologist assessment 360-degree evaluation professional development

Get full access to this article

View all access options for this article.

References

Accreditation Council for Graduate Medical Education Outcome Project. (2004). Accreditation Council for Graduate Medical Education web site. Retrieved November 2, 2004, from http://www.acgme.org/Outcome/

Archer, J. , Norcini, J. , & Davies, H.A. ( 2005). Peer review of paediatricians in training using SPRAT , British Medical Journal, 330, 1251-1253.

Atkins, P.W.B. , & Wood, R.E. ( 2002). Self- versus others’ ratings as predictors of assessment center ratings: Validation evidence for 360-degree feedback programs. Personnel Psychology, 55, 871-904.

Bracken, D. W. , Timmreck, C. W. , & Church, A. H. (Eds.). (2001). The handbook of multisource feedback: The comprehensive resource for designing and implementing MFS processes. San Franscisco: Jossey-Bass.

Bill 25, Health Professions (Regulatory Reform) Amendment Act 2008, 4th Session, 38th Parliament of the Legislative Assembly, British Columbia, Third Reading (2008, May 27).

Church, A.H. ( 1997). Do you see what I see? An exploration of congruence in ratings from multiple perspectives. Journal of Applied Social Psychology, 27, 983-1020.

Cone, J.J. ( 2001). Evaluating outcomes: Empirical tools for effective practice . Washington, DC: American Psychological Association.

Connors G.P. , & Munro, T.W. ( 2001). 360-degree physician evaluations. Healthcare Executive, 27(5), 58-59.

Falender, C.A. , & Shafranske, E.P. (2004). 360-degree Assessment applied to psychology in training. Manuscript in preparation.

10.

Fantuzzo, J.W. , Sisemore, T.A. , & Spradlin, W.H. (1983). A competency-based model for teaching skills in the administration of intelligence tests. Professional Psychology: Research and Practice, 14, 224-231.

11.

Fidler, H. , Lockyer, J. , Toews, J. , & Violato, C. ( 1999). Changing physicians’ practices: The effect of individual feedback. Academic Medicine, 74, 702-714.

12.

Fletcher, C. , & Bailey, C. ( 2003). Assessing self-awareness: Some issues and methods. Journal of Managerial Psychology, 18, 395-404.

13.

Griffin, E. , Sanders, G. , Craven, D. , & King, J. ( 2000). A computerized 360 feedback tool for personal and organizational development in general practice. Health Informatics Journal , 6, 71-80.

14.

Hall, W. , Violato, C. , Lewkonia, R. , Lockyer, J. , Fidler, H. , & Toews, J. ( 1999). Assessment of Physician performance in Alberta: The physician achievement review project. Canadian Medical Association Journal , 161, 52-57.

15.

Health Professions Act 2009, Alberta Legislature, Alberta Queens Printer.

16.

Kaslow, N.J. , Rubin, N.J. , Forrest, L. , Elman, N.S. , Van Horne, B.A. , Jacobs, S.C. , et al. (2007). Recognizing, assessing, and intervening with problems of professional competence. Professional Psychology: Research and Practice, 38, 479-492.

17.

Kaslow, N.J. ( 2004). Competencies in professional psychology. American Psychologist, 59, 774-781.

18.

Levine, A.M. ( 2002). Medical professionalism in the new millennium: A physician charter. Annals of Internal Medicine, 136, 243-246

19.

Lockyer, J. , Violato, C. , Fidler, H. , & Alakija, P. ( 2009). The assessment of pathologists/Laboratory medicine physicians through a multisource feedback tool. Archives of Pathology and Laboratory Medicine, 133, 1301-1308.

20.

Lockyer, J. , & Clyman, G.S. ( 2008). Multisource feedback (360-degree Assessment). In E.S. Holmboe , & R. E. Hawkins (Eds.), Practical guide to the Assessment of clinical competence (pp 75-85). Philadelphia: Mosby.

21.

Lockyer, J. , Violato, C. , & Fidler, H. ( 2008). Assessment of radiology physicians by regulatory Authority . Radiology, 247, 771-778.

22.

Lockyer, J. , Violato, C. , & Fidler, H. ( 2003). Likelihood of change: A study assessing surgeon use of multi-source feedback data. Teaching, Learning, and Medicine , 15, 168-174.

23.

Lockyer, J. , & Violato, C. ( 2004). An examination of the appropriateness of using a common peer Assessment instrument to assess physician skills across specialties. Academic Medicine, 79, 5-8.

24.

Maurer, T.J. , Mitchell, D.R.D. , & Barbeite, F.G. (2002). Predictors of attitudes toward a 360-degree feedback system and involvement in post-feedback management development activity. Journal of Occupational and Organizational Psychology, 75, 87-107.

25.

Norcini, J.J. ( 2003). Peer assessment of competence. Medical Education, 37, 539-543.

26.

Rodolfa, E. E. (Ed.). (2005). Competency education-competent professionals-ethical practice [Special Issue] . Professional Psychology: Research and Practice, 36, 347-375.

27.

Rodolfa, E.R. , Bent, R.J. , Eisman, E. , Nelson, P.D. , & Ritchie, P. ( 2005). A cube model for Competency development: Implications for psychology educators and regulators. Professional Psychology: Research and Practice, 36, 347-354.

28.

Ramsey, P.G. , Wenrich, M.D. , Carline, J.D. , Inui, T.S. , Larson, E. , & Logerfo, J.P. ( 1993). Use of peer ratings to evaluate physician performance . Journal of American Medical Association, 269, 1655-1660.

29.

Roberts, M. , Borden, K. , Christiansen, M. , & Lopez, S. ( 2005). Fostering a culture shift: Assessment of competence in the education and careers of professional psychologists. Professional Psychology: Research and Practice, 36, 355-361.

30.

Sala, F. , & Dwight, S.A. ( 2002). Predicting executive performance with multirater surveys: Whom you ask makes a difference. Consulting Psychology Journal: Practice and Research, 54, 166-172.

31.

Shaw, B.F. , & Dobson, K.S. ( 1988). Competency judgments in the training and Assessment of psychotherapists. Journal of Consulting and Clinical Psychology , 56, 666-672.

32.

Societal Needs Working Group Report. (1996 ). Skills for the New Millennium: CanMEDS 2000 Project. Annals of Royal College of Physicians and Surgeons of Canada, 29, 206-221

33.

Stern, D. T. (Ed.). (2006). Measuring medical professionalism. Oxford, UK: Oxford University Press.

34.

Violato, C. , Worsfold, L. , & Polgar, J.M. ( 2009). Multisource feedback systems for quality improvement in the health professions: Assessing occupational therapists in practice. Journal of Continuing Education in the Health Professions, 29(2), 111-118.

35.

Violato, C. , Lockyer, J. , & Fidler, H. ( 2003). Multisource feedback: A method of assessing surgical practice. British Medical Journal, 326, 546-548.

36.

Violato, C. , Lockyer, J. , & Fidler, H. ( 2008a). Assessment of psychiatrists with multisource feedback . Canadian Journal of Psychiatry, 53, 525-533

37.

Violato, C. , Lockyer, J. , & Fidler, H. ( 2008b). Changes in performance: A 5-year longitudinal study of participants in a multi-source feedback program. Medical Education, 42, 1007-1013.

38.

Violato, C. , Marini, A. , Toews, J. , Lockyer, J. , & Fidler, H. ( 1997). Feasibility and psychometric properties of using peers, consulting physicians, co-workers, and patients to assess physicians. Academic Medicine, 72, 82-84.