Multilevel confirmatory factor analysis was used to evaluate the factor structure underlying the 12-item, three-factor Interagency Collaboration Activities Scale (ICAS) at the informant level and at the agency level. Results from 378 professionals (104 administrators, 201 service providers, and 73 case managers) from 32 children’s mental health service agencies supported a correlated three-factor model at each level and indicated that the item loadings were not significantly (p > .05) different across levels. Reliability estimates of the three factors (Financial and Physical Resource Activities, Program Development and Evaluation Activities, and Collaborative Policy Activities) at the agency level were .81, .60, and .72, respectively, whereas these estimates were .79, .82, and .85 at the individual level. These multilevel results provide support for the construct validity of the scores from the ICAS. When the ICAS was examined in relation to Level 1 and Level 2 covariates, results showed that participants’ characteristics (i.e., age, job role, gender, educational level, and number of months employed) and agency characteristics (i.e., state location and number of employees) were not significantly ( p > .05) related to levels of interagency collaboration.
American Educational Research Association, American Psychological Association, National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
2.
Andrews, F.M. ( 1984). Construct validity and error components of survey measures: A structural modeling approach. Public Opinion Quarterly, 48, 409-442.
3.
Bentler, P.M. ( 1992). On the fit of models to covariances and methodology to the Bulletin. Psychological Bulletin, 112, 400-404.
4.
Bergstrom, A., Clark, R., Hogue, T., Iyechad, T., Miller, J., Mullen, S., . . . Thurston, F. ( 1995). Collaboration framework- Addressing community capacity . Retrieved from http://crs.uvm.edu/nnco/collab/framework.html
5.
Bliese, P.D. ( 2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In K. J. Klein & W. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions and new directions (pp. 349-381). San Francisco, CA: Jossey-Bass.
6.
Brown, E.C., Hawkins, J.D., Arthur, M.W., Abbott, R.D., & Van Horn, M.L. (2008). Multilevel analysis of a measure of community prevention collaboration. American Journal of Community Psychology, 41, 115-126.
7.
Bruner, C. ( 1991). Thinking collaboratively: Ten questions and answers to help policy makers improve children’s services. Washington, DC: Education and Human Services Consortium .
8.
Calloway, M., Morrissey, J.P., & Paulson, R.I. ( 1993). Accuracy and reliability of self-reported data in interorganizational networks. Social Networks, 15, 377-398.
9.
Chan, D. ( 1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology, 83, 234-246.
10.
Cumblad, C., Epstein, M.H., Keeney, K., Marty, T., & Soderlund, J. ( 1996). Children and adolescents network: A community-based program to serve individuals with serious emotional disturbance. Special Services in the Schools, 11, 97-118.
11.
Darlington, Y., Feeney, J.A., & Rixon, K. ( 2005). Interagency collaboration between child protection and mental health services: Practices, attitudes and barriers. Child Abuse & Neglect, 29, 1085-1098.
12.
Dyer, N.G., Hanges, P.J., & Hall, R.J. ( 2005). Applying multilevel confirmatory factor analysis techniques to the study of leadership. Leadership Quarterly, 16, 149-167.
13.
Friedman, S.R., Reynolds, J., Quan, M.A., Call, S., Crusto, C.A., & Kaufman, J.S. ( 2007). Measuring change in interagency collaboration: An examination of the Bridgeport Safe Start Initiative. Evaluation and Program Planning, 30, 294-306.
14.
Glisson, C. ( 2002). The organizational context of children’s mental health services. Clinical Child and Family Psychology Review , 5, 233-253.
15.
Glisson, C. ( 2007). Assessing and changing organizational culture and climate for effective services. Research on Social Work Practice, 17, 736-747.
16.
Glisson, C., & Hemmelgarn, A. (1998). The effects of organizational climate and interorganizational coordination on the quality and outcomes of children’s service systems. Child Abuse & Neglect , 22, 401-421.
17.
Graham, J.W. ( 2009). Missing data analysis: Making it work in the real world . Annual Review of Psychology, 60, 549-576.
18.
Greenbaum, P.E., & Dedrick, R.F. ( 2007). Multilevel analysis of interagency collaboration of children’s mental health agencies. In C. Newman, C. Liberton, K. Kutash, & R. M. Friedman (Eds.), The 19th annual research conference proceedings, a system of care for children’s mental health: Expanding the research base (pp. 445-448). Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, Research and Training Center for Children’s Mental Health.
19.
Greenbaum, P.E., Lipien, L., & Dedrick, R.F. ( 2004, March). Developing an instrument to measure interagency collaboration among child-serving agencies. Paper presented at the annual meeting of A System of Care for Children’s Mental Health: Expanding the Research Base, Tampa, FL.
20.
Harrod, J.B. ( 1986). Maine child and adolescent service system profile. Unpublished manuscript.
21.
Health Resources and Services Administration. (2000). Community health status report: Data sources, definitions, and notes. Washington, DC: Author.
22.
Hodges, S., Nesman, T., & Hernandez, M. ( 1999). Promising practices: Building collaboration in systems of care. Systems of care: Promising practices in children’s mental health , 1998 series (Vol. 6). Washington, DC: Center for Effective Collaboration and Practice, American Institutes for Research.
23.
Hox, J. ( 2002). Multilevel analysis: Techniques and applications. Mahwah, NJ: Lawrence Erlbaum.
24.
Hu, L., & Bentler, P.M. ( 1999). Cutoff criteria for fit indices in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.
25.
Jones, N., Thomas, P., & Rudd, L. ( 2004). Collaborating for mental health services in Wales: A process evaluation. Public Administration, 82, 109-121.
26.
Kozlowski, S.W.J., & Klein, K.J. ( 2000). A multilevel approach to theory and research in organizations: Contextual, temporal, and emergent processes. In K. J. Klein & W. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions and new directions (pp. 3-90). San Francisco, CA: Jossey-Bass.
27.
Lippitt, R., & Van Til, J. (1981). Can we achieve a collaborative community? Issues, imperatives, potentials. Journal of Voluntary Action Research, 10, 7-17.
28.
Little, R.J.A., & Rubin, D.B. ( 2002). Statistical analysis with missing data. Hoboken, NJ: John Wiley.
29.
Longoria, R.A. ( 2005). Is inter-organizational collaboration always a good thing ? Journal of Sociology and Social Welfare, 32, 123-138.
30.
Lüdtke, O., Marsh, H.W., Robitzsch, A., Trautwein, U., Asparouhov, T., & Muthén, B. (2008). The multilevel latent covariate model: A new, more reliable approach to group-level effects in contextual studies. Psychological Methods, 13, 203-229.
Mattessich, P.W., Murray-Close, M., & Monsey, B.R. ( 2001). Collaboration: What makes it work (2nd ed.). Saint Paul, MN: Amherst H. Wilder Foundation.
33.
Morrissey, J.P., Calloway, M., Bartko, W.T., Ridgely, M.S., Goldman, H.H., & Paulson, R.I. ( 1994). Local mental health authorities and service system change: Evidence from the Robert Wood Johnson program on chronic mental illness. Milbank Quarterly, 72, 49-80.
Muthén, L.K., & Muthén, B.O. (2007). Multilevel modeling with latent variables using Mplus. Unpublished manuscript.
36.
Nunnally, J.C. ( 1978). Psychometric theory (2nd ed.). New York, NY: McGraw-Hill.
37.
Robinson, W.S. ( 1950). Ecological correlations and the behavior of individuals . American Sociological Review, 15, 351-357.
38.
Salmon, G. ( 2004). Multi-agency collaboration: The challenges for CAMHS . Child and Adolescent Mental Health, 9, 156-161.
39.
Schwartz, G.T. ( 1978). Estimating the dimensions of a model. Annals of Statistics, 6, 461-464.
40.
Smith, B.D., & Mogro-Wilson, C. (2007). Multi-level influences on the practice of inter-agency collaboration in child welfare and substance abuse treatment. Children and Youth Services Review, 29, 545-556.
41.
Steiger, J.H., & Lind, J.C. ( 1980, May). Statistically-based tests for the number of common factors. Paper presented at the annual meeting of the Psychonomic Society, Iowa City, IA.
42.
Stroul, B.A., & Friedman, R.M. (1986). A system of care for children and adolescents with severe emotional disturbance (Rev. ed.). Washington, DC: Georgetown University, Child Development Center, National Technical Assistance Center for Child Mental Health.
43.
Wasserman, S., & Faust, K. ( 1994). Social network analysis: Methods and applications. New York, NY: Cambridge University Press.
44.
Zyphur, M.J., Kaplan, S.A., & Christian, M.S. (2008). Assumptions of cross-level measurement and structural invariance in the analysis of multilevel data: Problems and solutions. Group Dynamics: Theory, Research, and Practice, 12, 127-140.
