Evaluating Performance Measurement Systems in Nonprofit Agencies: The Program Accountability Quality Scale (PAQS)

Abstract

The drive for accountability in human services puts pressure on nonprofit agencies to develop performance measurement systems. But efforts to build capacity in this area have been hindered by the lack of instruments to evaluate the quality of proposed performance measurement systems. The Performance Accountability Quality Scale (PAQS) attempts to fill this gap. The instrument was field-tested on 191 program performance measurement systems developed by nonprofit agencies in Central Florida. Preliminary findings indicate that PAQS provides a structure for obtaining expert opinions based on a theory-driven model about the quality of a proposed measurement system in a not-for-profit agency. The instrument also is useful for assessing agency needs for technical assistance and for evaluating progress in the development of performance measurement systems. Further study is needed to test PAQS in other settings and to explore new areas of research in outcome evaluation.

Get full access to this article

View all access options for this article.

References

Cascio, W. E. , Alexander, R. A. , & Barrett, G. V. (1988). Setting cutoff scores: Legal, psychometric, and professional issues and guidelines. Personnel Psychology, 41, 1-24.

Crocker, L. , & Algina, J. (1986). Introduction to classical and modern test theory. Fort Worth, TX: Holt, Rinehart & Winston.

Hatry, H. (1997). Outcome measurement and social services: Public and private sector perspectives. In E. J. Mullen & J. L. Magnabosco (Eds.), Outcomes measurement in the human services (pp. 3-19). Washington, DC: NASW Press.

Hatry, H. , van Houten, T. , Plantz, M. C. , & Greenway, M. T. (1996). Measuring program outcomes: A practical approach. Alexandria, VA: United Way of America.

Hodgkinson, V. , & Weitzman, M. (1996). Nonprofit almanac 1996-1997. San Francisco: Jossey-Bass.

Kaiser, H. F. (1974). An index of factorial simplicity. Psychometrika, 39, 31-36.

Kerlinger, F. N. (1986). Foundations of behavioral research (3rd ed.). New York: Holt, Reinhart & Winston.

Lambur, M. (1993, November). Can evaluability assessment be usefully shortened for use in small independent organizations? Paper presented at the American Evaluation Association Annual Meeting, Dallas, Texas.

Newcomer, K. E. (1997). Using performance measurement to improve programs. In K. E. Newcomer (Ed.), Using performance measurement to improve public and nonprofit programs (pp. 5-13). San Francisco: Jossey-Bass.

10.

Nunnally, J. C. , & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.

11.

Plantz, M. C. , Greenway, M. T. , & Hendricks, M. (1997). Outcome measurement: Showing results in the nonprofit sector. In K. E. Newcomer (Ed.), Using performance measurement to improve public and nonprofit programs (pp. 15-30). San Francisco: Jossey-Bass.

12.

Reisman, J. (1994). A field guide to outcome-based program evaluation. Seattle, WA: The Evaluation Forum, Organizational Research Services, Clegg & Associates.

13.

Reisman, J. , & Clegg, J. (1999). Outcomes for success 2000. Seattle, WA: The Evaluation Forum, Organizational Research Services, Clegg & Associates.

14.

Rubin, A. , & Babbie, E. (1997). Research methods for social work (3rd ed.). Pacific Grove, CA: Brooks/Cole.

15.

Rutman, L. (1980). Planning useful evaluations: Evaluability assessment. Beverly Hills: Sage.

16.

Schmidt, R. E. , Scanlon, J. W. , & Bell, J. B. (1979). Evaluability assessment: Making public programs work better. Rockville, MD: U.S. Department of Health, Education and Welfare, Project Share, Human Services Monograph 14.

17.

Segal, S. P. (1997). Outcomes measurement systems in mental health: A program perspective. In E. J. Mullen & J. L. Magnabosco (Eds.), Outcomes measurement in the human services (pp. 149-159). Washington, DC: NASW Press.

18.

Stevens, J. (1986). Applied multivariate statistics for the social sciences. Hillsdale, NJ: Lawrence Erlbaum.

19.

U. S. Government Accounting Office (1996, June). Effectively implementing the Government Performance and Results Act (GAO/GGD-96-118). Washington, DC.

20.

U. S. Government Accounting Office (1999, February). Agency performance plans: Examples of practices that can improve usefulness to decision makers (GAO/GGD/AIMD-99-69). Washington, DC.

21.

Weiss, H. B. (1997). Results-based accountability for child and family services. In E. J. Mullen & J. L. Magnabosco (Eds.), Outcomes measurement in the human services (pp. 173-180). Washington, DC: NASW Press.

22.

Wholey, J. S. (1979). Evaluation: Promise and performance. Washington, DC: The Urban Institute.

23.

Wholey, J. S. (1983). Evaluation and effective public management. Boston: Little, Brown.

24.

Wong-Rieger, D. , & David, L. (1995). Using program logic models to plan and evaluate education and prevention programs. In A. J. Love (Ed.), Evaluation methods sourcebook II (pp. 120-135). Ottawa, Ontario: Canadian Evaluation Society.