Evaluation research and outcomes measurement in the arena of behavioral health services for children must be adapted for the community agency setting. Through evaluation research, it is possible to address service goals as well as more traditional academic research goals. This article examines a variety of activities that have been implemented to evaluate children’s behavioral and educational services in a Northern California non-profit community agency. It is noted that there are multiple formats for collecting information from and providing comments to children’s parents, their clinicians, and program administration staff, all of which can be used to effectively address service-focused evaluation research goals. Challenges to doing scientifically rigorous research in a community setting require additional considerations regarding organizational culture and structure. Based on the experiences of the authors and the experiences of others, the article describes general principles that can guide evaluation research and outcomes measurement with children and their families in the community health agency setting.
Get full access to this article
View all access options for this article.
References
1.
Abidin, R. R.
(1990). Parenting stress index—Manual. Charlottesville: University of Virginia, Pediatric Psychology Press.
2.
Achenbach, T.
(1991). Child Behavior Checklist for ages 4-18. Burlington: University of Vermont.
3.
Alkin, M. C.
, Hofstetter, C. H., & Ai, X. (1998). Stakeholder concepts. In A. J. Reynolds & H. J. Walberg (Eds.), Advances in educational productivity, Vol. 7: Evaluation research for educational productivity (pp. 87-113). Stamford, CT: JAI.
4.
Attkisson, C. C.
, & Zwick, R. (1982). The Client Satisfaction Questionnaire. Psychometric properties and correlations with service utilization and psychotherapy outcome. Evaluation and Program Planning, 5(3), 233-237
5.
Bergman, D. A.
(1995). Thriving in the 21st century: Outcome assessment, practice parameters, and accountability. Pediatrics, 96(4), 831-835.
6.
Bird, H. R.
, Canino, G., Rubio-Stipec, M., Gould, M. S., Ribera, J., Sesman, M., et al. (1988). Estimates of the prevalence of childhood maladjustment in a community survey in Puerto Rico. The use of combined measures. Archives of General Psychiatry, 45(12), 1120-1126.
7.
Birleson, P.
(1998). Learning organisations: A suitable model for improving mental health services?Australian & New Zealand Journal of Psychiatry, 32(2), 214-222.
8.
Birleson, P.
, & Luk, E. S. (1997). Continuing the debate on a separate adolescent psychiatry. Australian & New Zealand Journal of Psychiatry, 31(4), 447-451.
9.
Bryk, A.
(Ed.). (1983). Stakeholder based evaluation: New directions for program evaluation 17. San Francisco: Jossey-Bass.
10.
Costello, E. J.
, Messer, S. C., Bird, H. R., Cohen, P., & Reinherz, H. Z. (1998). The prevalence of serious emotional disturbance: A re-analysis of community studies. Journal of Child & Family Studies, 7(4), 411-432.
11.
Cousins, J. B.
(1992). The case for participatory evaluation. Educational Evaluation and Policy Analysis, 14, 397-418.
12.
Cousins, J. B.
, & Earl, L. M. (1992). The case for participatory evaluation. Educational Evaluation and Policy Analysis, 14(4), 397-418.
13.
Crane, V. S.
(1988). Economic aspects of clinical decision making: Applications of clinical decision analysis. American Journal of Hospital Pharmacy, 45(3), 548-553.
14.
Cummings, O. W.
, Nowakowski, J. R., Schwandt, T. A., Eichelberger, R. T., Kleist, K. C., Larson, C. L., et al. (1988). Business perspectives on internal/external evaluation. In J. A. McLaughlin, L. J. Weber, R. W. Covert, & R. B. Ingle (Ed.), Evaluation utilization (Vol. 39, pp. 59-73). San Francisco: Jossey-Bass.
15.
Epstein, A. M.
(1997). Medicaid managed care and high quality: Can we have both?Journal of the American Medical Association, 278(19), 1617-1621.
16.
Epstein, M. H.
, & Sharma, J. M. (1998). Behavioral and Emotional Rating Scale: A strength-based approach to assessment—Examiner’s manual. Austin, TX: Pro-ed.
17.
Epstein, R. S.
, & Sherwood, L. M. (1996). From outcomes research to disease management: A guide for the perplexed. Annals of Internal Medicine, 124(9), 832-837.
18.
Fetterman, D. M.
(1994). Steps of empowerment evaluation: From California to Cape Town. Evaluation & Program Planning, 17(3), 305-313.
19.
Forrest, C. B.
, Simpson, L., & Clancy, C. (1997). Child health services research. Challenges and opportunities. Journal of the American Medical Association, 277(22), 1787-1793.
20.
Goldfried, M. R.
, & Wolfe, B. E. (1998). Toward a more clinically valid approach to therapy research. Journal of Consulting and Clinical Psychology, 66(1), 143-150.
21.
Heneghan, A. M.
, Horwitz, S. M., & Leventhal, J. M. (1996). Evaluating intensive family preservation programs: A methodological review. Pediatrics, 97(4), 535-542.
22.
Hernandez, M.
, Hodges, S., & Cascardi, M. (1998). The ecology of outcomes: System accountability in children’s mental health. Journal of Behavioral Health Services and Research, 25(2), 136-150.
23.
Hoagwood, K.
, Jensen, P. S., Petti, T., & Burns, B. J. (1996). Outcomes of mental health care for children and adolescents: I. A comprehensive conceptual model. Journal of the American Academy of Child and Adolescent Psychiatry, 35(8), 1055-1063.
24.
Hodges, K.
, & Wong, M. M. (1996). Psychometric characteristics of a multidimensional measure to assess impairment: The Child and Adolescent Functional Assessment Scale. Journal of Child & Family Studies, 5(4), 445-467.
25.
House, E. R.
(1993). Professional evaluation: Social impact and political consequences. Newbury Park, CA: Sage.
26.
Huffman, L.
, & Kerivan A. (1999). CHC Functioning Scale. Palo Alto, CA: The Children’s Health Council.
27.
Kumpfer, K. L.
(1999). Outcome measures of interventions in the study of children of substance-abusing parents. Pediatrics, 103(5), 1128-1144.
28.
Lavigne, J. V.
, Gibbons, R. D., Christoffel, K. K., Arend, R., Rosenbaum, D., Binns, H., et al. (1996). Prevalence rates and correlates of psychiatric disorders among preschool children. Journal of the American Academy of Child and Adolescent Psychiatry, 35(2), 204-214.
29.
Lemoine, R. L.
, & Carney, A. (1983). The Louisiana mental health client-outcome evaluation project: An initial progress report. Community Mental Health Journal, 20(1), 90-100.
30.
Lerner, R. M.
, Sparks, E. E., & McCubbin, L. D. (1999). Family diversity and family policy. In D. H. Demo, K. R. Allen, & M. A. Fine (Ed.), Handbook of family diversity (pp. 380-401). New York: Oxford University Press.
31.
Lewin, L. S.
(1995). A single standard for measuring community benefit is needed. Modern Healthcare, 25(50), 51-51.
32.
Liang, M. H.
, & Shadick, N. (1997). Feasibility and utility of adding disease-specific outcome measures to administrative databases to improve disease management. Annals of Internal Medicine, 127(8), 739-742.
33.
Lieu, T. A.
, & Newman, T. B. (1998). Issues in studying the effectiveness of health services for children. Health Services Research, 33(4), 1041-1058.
34.
Maton, K. I.
(2000). Making a difference: The social ecology of social transformation. American Journal of Community Psychology, 28(1), 25-57.
35.
Moffitt, T. E.
(1996). Measuring children’s antisocial behaviors. Journal of the American Medical Association, 275(5), 403-404.
36.
Narrow, W. E.
, Regier, D. A., Goodman, S. H., Rae, D. S., Roper, M. T.Bourdon, K. H., et al. (1998). A comparison of federal definitions of severe mental illness among children and adolescents in four communities. Psychiatry Services, 49(12), 1601-1608.
Preskill, H.
, & Torres, R. T. (1999). Evaluative inquiry for learning in organizations. Thousand Oaks, CA: Sage.
39.
Provus, M. N.
(1971). Discrepancy evaluation. Berkeley, CA: McCutcheon.
40.
Recchia, K. C.
, Petros, T. M., Spooner, S. A., & Cranshaw, J. L. (1995). Implementation of the community outpatient practice experience in a large pediatric residency program. Pediatrics, 96(1), 90-98.
41.
Rosenblatt, A.
(1998). A primer on outcome measures for children’s services. Behavioral Healthcare Tomorrow, 7(4), 41-43.
42.
Santos, A. B.
, Henggeler, S. W., Burns, B. J., Arana, G. W., & Meisler, N. (1995). Research on field-based services: Models for reform in the delivery of mental health care to populations with complex clinical problems. American Journal of Psychiatry, 152(8), 1111-1123.
43.
Schainblatt, A. H.
(1980). What happens to the clients?Community Mental Health Journal, 16(4), 331-342.
44.
Schumacher, J. E
, Milby, J. B., Engle, M., & Raczynski J. M. (2000). Linking practice and science in the substance abuse treatment of homeless persons. Journal of Applied Behavioral Science, 36, 297-313.
45.
Senge, P.
(1990). The fifth discipline: The art and practice of the learning organization. New York: Currency Doubleday.
46.
Sisk, J. E.
, Gorman, S. A., Reisinger, A. L., Glied, S. A., DuMouchel, W. H., & Hynes, M. M. (1996). Evaluation of Medicaid managed care. Satisfaction, access, and use. Journal of the American Medical Association, 276(1), 50-55.
47.
Sonnichsen, R. C.
(1988). Advocacy evaluation: A model for internal evaluation offices. Evaluation & Program Planning, 11(2), 141-148.
48.
Sparks, E.
(1996). The challenges facing community health centers in the 1990s: A voice from the inner city. In M. B. Lykes, A. Banuazizi, R. Liem, & M. Morris (Eds.), Myths about the powerless: Contesting social inequalities (pp. 237-257). Philadelphia: Temple University Press.
49.
Strayhorn, J. M.
, & Weidman, C. S. (1988). A parent practices scale and its relation to parent and child mental health. Journal of the American Academy of Child and Adolescent Psychiatry, 27(5), 613-618.
50.
Thayer, C. E.
, & Fine, A. H. (2001). Evaluation and outcome measurement in the non-profit sector: Stake-holder participation. Evaluation & Program Planning, 24, 103-108.
51.
Torres, R. T.
(1994). Linking individual and organizational learning: The internalization and externalization of evaluation. Evaluation & Program Planning, 17(3), 327-337.
52.
Weinstein, C. E.
(1992). Learning and Study Strategies Inventory. Lincoln: University of Nebraska–Lincoln, Buros Institute of Mental Measurements.
53.
Weiss, B.
, Catron, T., Harris, V., & Phung, T. M. (1999). The effectiveness of traditional child psychotherapy. Journal of Consulting Clinical Psychology, 37(1), 82-94.
54.
Weiss, H. B.
, & Greene, J. C. (1992). An empowerment partnership for family support and education programs and evaluation. Family Science Review, 5, 131-148.
55.
Whyte, W. F.
(1991). Participatory action research. Newbury Park, CA: Sage.
