Development and Validation of the Family Feedback on Child Welfare Services (FF-CWS)

Abstract

Objective:

The measurement of Family Feedback on Child Welfare Services (FF-CWS) is gaining prominence as an efficacy indicator and is coherent with concerns about family-centered practice and empowerment. The aim of this study was to develop and validate an instrument that would overcome the scarcity of psychometrically sound measures in this field.

Methods:

Following item construction and selection, exploratory and confirmatory factor analyses with a sample of 263 Spanish (52%) and Portuguese (48%) caregivers of children involved with CWS were conducted.

Results:

Three subscales were identified: Intervention Efficacy, Perception of Workers, and Satisfaction with the Intervention Process. In general, all dimensions showed good reliability, convergent, and criterion-related validity results. Multigroup analyses confirmed measurement invariance for both countries.

Conclusions:

The FF-CWS Questionnaire is a brief self-report measure that can be a useful assessment tool to frontline practitioners, agency managers, and policy makers for program evaluation and planning.

Keywords

family feedback child welfare at-risk families satisfaction questionnaire validation

Although the feedback and opinions of child welfare recipients should arguably be taken into account in evidence-based interventions, there are still considerable conceptual and methodological shortcomings that hinder gathering and harnessing such information.

In both Spain and Portugal, the laws that regulate state intervention with underaged at-risk children and their families (Boletín Oficial del Estado, 1996; Lei nº 147/99, 1999) recognize that children’s out-of-home placement must be a last resort action. At-risk families are defined as those whose contextual or personal adverse circumstances hinder their parenting competences, thereby jeopardizing children’s development (Rodrigo & Byrne 2011). Therefore, both countries share a philosophy of action with at-risk families based on support and family preservation, aiming to ensure children’s well-being within their families of origin. Nevertheless, the way in which family preservation services (i.e., Child Welfare Services [CWS]) are organized is different in each country. In Spain, Community Social Services are responsible for the interventions with at-risk children, which have been centralized in the Autonomous Communities since 1987. In Portugal, the enforcement of child protective measures is shared among the Commissions for the Protection of Children and Youth, Juvenile Court, Social Services, and Health and Education Services. Notwithstanding these differences, in both countries, there is a significant dearth of information regarding the perceptions and feelings of at-risk families toward CWS. Previous research comparing Portuguese and Spanish at-risk families assisted by family preservation services showed that they share several sociodemographic characteristics, such as economic stress, low educational levels, high levels of unemployment, and being multiassisted by community services (Pérez-Padilla et al., 2015).

Taking families’ views and opinions about services into account is of great importance in shaping and refining evidence-based interventions. In fact, family feedback on CWS is being increasingly viewed as a relevant indicator to assess service functioning and efficacy (Baker, 2007; Cortis, 2007; Kapp & Vela, 1999; Tilbury, Osmond, & Crawford, 2010). This concern is consistent with the recent emphasis on agencies’ accountability and transparence as well as with a family-centered and strengths-based perspective (Connolly, 2007). This perspective, in social work practice, translates into recognizing family as the most adequate environment for child development. European policies acknowledge governments’ responsibility in promoting positive parenting through parental and family support, especially for vulnerable families. The need to “foster a dialogue with stakeholders as well as the public on the outcomes and general satisfaction of the child and family friendliness of social services” has also been recognized (Committee of Ministers of the Council of Europe, 2006, 2011).

There are several reasons to assess systematically family feedback on CWS. First, due to their firsthand experience, families have a wealth of information regarding the functioning of services and programs. Thus, analyzing their views provides valuable insights about how they experience services, which gives CWS an opportunity to change those aspects that are not helpful (Harris & Poertner, 1998). This means an opportunity to modify and enhance existing programs and services (Baker, 2007) as well as to design new ones that are responsive to families’ needs. For evidence-based practice, agencies must integrate the best available knowledge about what works with parents’ expectations, values, and skills (i.e., what works for whom, and under what circumstances; Tilbury et al., 2010). It has been argued that the social validity of human services depends, at a great extent, on user satisfaction (McMurtry & Hudson, 2000). Second, assessing parents’ service experience can serve as a means to refine the conceptualization of satisfaction and to identify its core components as well as contributing to build knowledge about the predictors of case outcome (Alpert, 2005; Baker, 2007). Third, the process of being asked about one’s opinions may change favorably the client’s perceptions about the staff and the agency (Baker, 2007), which could lead to an improvement in treatment engagement and adherence and eventually in outcomes. Fourth, from an ethical point of view, giving a voice to parents involved in CWS means acknowledging them as citizens with rights to equity, representation, and participation (Pollitt, 1998). This is especially important in nonvoluntary services, in which there is an accentuated agency users power asymmetry. If the point of view of those in the most vulnerable position is not considered, the democratic functioning of institutions is compromised.

In the family feedback literature, users’ service satisfaction has been by far the most studied component. Although the link between CWS users’ perceptions and case outcomes has not been solidly established, there is some evidence that points to positive associations. For instance, Trotter (2008) found that more satisfied users received higher practitioner estimates of client progress, fewer further notifications, and lower rates of children placement. Evidence is more robust regarding the association between client satisfaction and intermediate outcomes, such as greater likelihood of service completion (Damashek, Doughty, Ware, & Silovsky, 2011). Also, the strength of the parent–worker relationship—one of the central elements of parents’ experience with CWS—predicted service completion (Girvin, DePanfilis, & Daining, 2007), family involvement (Korfmacher, Green, Spellmann, & Thornburg, 2007), child and family well-being (Johnson & Ketring, 2006; Southerland, Mustillo, Farmer, Stambaugh, & Murray, 2009), and improvements in child safety (Johnson & Ketring, 2006; Lee & Ayón, 2004).

In spite of these findings and the aforementioned institutional recognition, parents’ feedback with CWS has rarely been studied beyond service satisfaction, and there is a notable scarcity of instruments to perform this assessment (Ayala-Nunes, Jiménez, Hidalgo, & Jesus, 2014; Baker, 2007; Harris & Poertner, 1998; Kapp & Vela, 1999). Within the CWS field, most efforts in instrument development have been directed toward the assessment of service satisfaction of parents with children in foster services (e.g., Alpert & Britner, 2009; Harris, Poertner, & Joe, 2000; Kapp & Vela, 2003). However, in child welfare, lower risk situations in which parental rights termination is not required constitute the greatest percentage of cases (Comissão Nacional de Proteção das Crianças e Jovens em Risco, 2014; U.S. Department of Health and Human Services, 2015). Moreover, in the current context of economic recession in Southern Europe countries, it is expectable that previously well-functioning families face financial hardship and family stress, which may augment the number of low-risk children entering CWS. Therefore, the development of an instrument suitable for parents who keep children’s custody would be useful for frontline practitioners, managers, and policymakers.

Previous instruments have allowed evaluating specific programs, but the practice of using ad hoc, program-specific questionnaires, without attempts to frame them into a theoretical framework or to ensure content validity and reliability, is common (Ayala-Nunes et al., 2014; Kapp & Vela, 1999). The use of qualitative methods to evaluate user feedback prevails (e.g., Buckley, Carr, & Whelan, 2011; Cortis, 2007). Qualitative methods can be a valuable source of information but preclude comparisons over time and across different services and programs. Also, performing systematic service assessment and large-scale surveys is not feasible using these methods.

The claim made by Pascoe (1984) decades ago about the lack of a conceptual model that frames empirical findings in user satisfaction studies still holds true for satisfaction in particular and for family feedback in general. In a systematic review of instruments assessing family feedback on CWS and family preservation programs, it was found that 88% were not explicitly based on a theoretical model nor did they provide a construct definition (Ayala-Nunes et al., 2014). Also, the tendency to assess only service satisfaction without including wider aspects of the perception of CWS (e.g., empowerment) was identified.

From a methodological point of view, measures in this field have been criticized for having unexamined or inadequate psychometric properties (Ayala-Nunes et al., 2014; Harris & Poertner, 1998). The identified questionnaires assessing families’ experiences in CWS—Strengths-Based Practices Inventory (SBPI, Green, McAllister, & Tarte, 2004), Customer Satisfaction Survey (CSS, Huebner, Jones, Miller, Custer, & Critchfield, 2006), and Current Client Satisfaction with Agency Staff (CCSAS, Winefield & Barlow, 1995)—although having shown satisfactory reliability indexes (Cronbach’s α ranging from .69 to .94) have serious conceptual and psychometric limitations, mainly because many of these aspects were unreported and therefore remain unknown. For instance, the SBPI (Green et al., 2004) did not include a definition of the construct being assessed nor did it include external evidence for discriminant validity. The CSS (Huebner et al., 2006) and the CCSAS (Winefield & Barlow, 1995) shared the limitations of the previous instrument and had some additional ones, such as not reporting or underreporting content validity analysis, statistical analysis of the items, or evidences for external validity. Additionally, the CCSAS used a small sample (N = 24) and did not perform a dimensionality analysis. Notably, almost all of the identified instruments were developed in the United States. The important cultural and organizational differences that exist between North American and Southern European family preservation institutions impel us to consider that their use in countries such as Spain and Portugal might not be appropriate. Therefore, the development of a sound, valid, and reliable instrument to assess parents’ experience with CWS is needed.

Taking into account the scarcity of measures to assess quantitatively parents’ experience with CWS as well as the limitations and drawbacks of the existing instruments, our aim was to develop and validate a family feedback questionnaire that would be (a) service-specific enough to suit the characteristics of this type of intervention and (b) sufficiently broad to be used in agencies (Child Welfare, Child Protection, or other type of services who attend at-risk families) and programs (family preservation and child abuse prevention) that deal with a wide range of risky situations and diverse populations. Carretero-Dios and Pérez’s (2005) and DeVellis’ (2003) recommendations for scale development were followed.

Method

Procedure

Development of a conceptual framework

Currently—to the best of authors’ knowledge—there is no empirically supported theoretical framework available that explains family feedback in the context of CWS. The present questionnaire was, therefore, developed by adapting Pascoe’s (1984) and Gerkensmeyer, Austin, and Miller’s (2006) conceptual framework of patient and consumer satisfaction, respectively. It must be noted that these models were the result of the integration and improvement in several conceptualizations and definitions in patient and consumer satisfaction literature—mainly, theories of assimilation and contrast—and that they only focus on satisfaction, a narrower construct within the experience with a given service. However, some elements of these theoretical frameworks may be applicable for users of CWS. We define feedback as the user’s reaction to salient aspects of the context, process, and outcomes of his or her experience with a given service. Evaluation is seen as a comparison between the salient features of the individual’s experience and a subjective standard. In this comparative process, two interrelated psychological processes occur: a cognitive evaluation and an emotional response to the service structure, process, and outcomes. The pattern used for judging the experience with the service may be a subjective ideal, a sense of what is deserved, a subjective average of past experiences in similar situations, or a minimal acceptable level. Pascoe (1984) assumes in his model that users are generally capable of discriminating and judging the quality of several aspects of their experience with the service. Based on this model, feedback is conceptualized as a dynamic process that may modify the user’s subjective standard over time.

Gerkensmeyer and colleagues (2006) noted that a desired service is also a relevant psychological standard to explain satisfaction levels and that both negative and positive expectations about services must be taken into account. Experience with CWS is considered a multidimensional construct. It seems plausible that perceptions of services vary according to the aspect being assessed. For instance, a parent may be pleased with his or her social worker but feel that the intervention has not accomplished the expected results. If a comprehensive assessment of the experience with the service is to be performed, measures should take into consideration distinct relevant aspects of this process. After a review of the literature (e.g., Baker, 2007; Green et al., 2004; Harris & Poertner, 1998; Huebner et al., 2006; Kapp & Vela, 1999; Winefield & Barlow, 1995), we identified six domains deemed important in the experience of CWS: expectations about the services, general satisfaction with the service, evaluation of service characteristics, evaluation of intervention outcomes, perception of workers, and perceptions of the agency intervention. Some of these domains, namely, perception of workers, perception of the agency intervention, and intervention outcomes, had been previously pointed out by Kapp and Vela (1999) as important dimensions to be assessed. Expectations address the subjective standard regarding service quality before the beginning of the intervention. General satisfaction refers to overall feeling with the experience. Evaluation of service characteristics includes issues such as service availability, accessibility, and atmosphere. Evaluation of intervention outcomes addresses progress made on goals, coping skills learned, comparison of service outcomes to those expected, and helpfulness of service. The PW domain includes empathy, respect, competency, communication quality, availability, assurance, and confidentiality. Lastly, the agency intervention domain addresses issues such as user involvement in decisions, agreement about child needs, intrusiveness, and agency consideration of the child’s best interests.

Scale construction and item assessment

Taking these dimensions into account, an initial pool of 75 items was generated, weighting the number of items per dimension according to its theoretical importance and complexity, that is, the number of relevant aspects to be taken into account. An effort was made to obtain a balance between positively and negatively worded items in order to neutralize the acquiescence bias, a pervasive problem in client satisfaction research (Pascoe, 1984). Items were formulated in a primary school reading level. This pool was initially evaluated by three experts in Child Welfare who were encouraged to make suggestions about how the instrument could be improved. Items were kept, modified, or deleted taking their input into account, and 35 items resulted from this initial selection. Because the instrument was to be validated in Portuguese and Spanish simultaneously, a back-forward translation was performed by the first author and one of the experts who were bilingual. The response format was anchored in a 4-point scale (1 = nothing at all; 2 = a little; 3 = fairly; and 4 = a lot) to prevent a neutral response effect (DeVellis, 2003).

To reinforce content validity and item quality, a second panel of 12 content experts (6 for each country, 3 scholars specialized in family studies and developmental psychology and 3 CWS workers per country) was consulted. A definition of the construct and its domains was provided in an evaluation sheet, and experts were requested to assess item clarity, vocabulary adequacy, relevance for the construct, and degree of correspondence with the proposed domain in a 5-point scale. They were also asked to rank the items according to their theoretical relevance for each domain. Following this assessment, items with a mean lower than 3.5 or that ranked in the last position within each domain were deleted and items with means between 3.6 and 4 were modified to enhance clarity. As a result, 27 items remained in the pilot version of the questionnaire (Table 1), which was titled as the Family Feedback on Child Welfare Services (FF-CWS).

Table 1.

Preliminary Version of Questionnaire Items, Descriptive and Correlation With CSQ-8.

Item	Domain	M	SD	Skewness	Kurtosis	Correlation With CSQ-8
1. I thought that things in my family would get worse by coming here	Expectations	3.59	0.92	−2.11	2.96	.166**
2. At the beginning I did not expect much from the help I was going to get here		2.79	1.14	−0.37	−1.30	.200**
3. Before starting to come here, I already knew that the help we were going to get would be helpful for my child		2.86	1.10	−0.50	−1.08	.297**
4. I am satisfied with the services I get here	General satisfaction	3.13	1.02	−0.94	−0.32	.595**
5. In general, I am satisfied with the help we get here		3.29	0.82	−0.99	0.31	.626**
6. Me and my family are not getting the kind of help we need		3.08	1.12	−0.80	−0.86	.271**
7. It’s easy to get the workers to meet with us	Service characteristics	3.32	0.87	−1.19	0.64	.312**
8. The location of the institution is convenient for us		3.49	0.78	−1.54	1.80	.084
9. I do not feel at ease talking about my matters in the room in which they see me		3.43	0.94	−1.43	0.74	.275**
10. The service office hours are convenient for us		3.46	0.78	−1.30	0.90	.180**
11. Things have not improved in our family since we are here	Intervention outcomes	3.12	1.06	−0.84	−0.67	.380**
12. The help I get here is better than I expected		2.89	1.01	−0.49	−0.87	.661**
13. I learned a lot here about how to deal with my problems		2.82	1.08	−0.45	−1.08	.615**
14. Here they have taught me how to look for and to get help elsewhere		2.59	1.12	−0.22	−1.32	.541**
15. I feel that the services here are not useful		3.44	0.92	−1.52	1.10	.499**
16. The workers understand how I feel	Perception of workers	3.16	0.915	−0.95	0.12	.386**
17. People here treat me disrespectfully		3.73	0.792	−2.82	6.48	.137*
18. I am happy with the work done by the workers		3.44	0.718	−1.21	1.14	.574**
19. I feel that workers really listen to me		3.46	0.675	−1.17	1.32	.475**
20. I know I can count on workers to talk about what is really worrying me		3.40	0.804	−1.26	0.91	.531**
21. Workers gave us all the information we needed		3.33	0.835	−1.21	0.88	.448**
22. Workers treat me correctly and nicely		3.65	0.553	−1.42	1.83	.375**
23. I know that the things I tell workers will not leave the room		3.44	0.840	−1.40	1.07	.390**
24. Workers take my opinion into account when it comes to make decisions about my child	Agency intervention	3.39	0.816	−1.26	0.89	.380**
25. I agree with workers about what they think is best for my child		3.30	0.822	−1.10	0.70	.490**
26. Workers are too intrusive in our life		3.36	0.993	−1.33	0.43	.375**
27. Workers act thinking about my child’s best interest		3.49	0.736	−1.43	1.60	.477**

Note. Correlations were computed using Spearman’s ρ (r_s ). Deleted items in bold were eliminated after the first selection process. Items in italics were eliminated after the second selection process. N = 263. CSQ-8 = Client Satisfaction Questionnaire; M = mean; SD = standard deviation.

*correlation is significant at .05 level. **correlation is significant at .01 level.

This version was administered to 10 CWS users (5 in Spain and 5 in Portugal) in order to detect items that generated comprehension problems. Participants’ feedback was solicited after the pilot administration to check whether the wording of items was confusing, and none of them reported comprehension difficulties. Because no changes were made on the wording of the items, the scores from these participants were included in the analyses. Questionnaire administration lasted approximately 10 min.

Measures

In order to characterize participants and to obtain evidences of external validity, the following instruments were used:

Sociodemographic profile: This questionnaire collected data on participant’s individual (gender, age, kinship with target child, academic level, immigrant status, and employment status); family (size and structure); and target child (age and gender) variables.

Satisfaction with CWS: The Client Satisfaction Questionnaire (CSQ-8, Larsen, Attkisson, Hargreaves, & Nguyen, 1979) measures client satisfaction as a single, broad construct and has good psychometric properties and brevity. It was originally developed for use in mental health programs (Larsen et al., 1979), but its nonspecificity has allowed its application in different types of human services. It is a brief, 8-item scale anchored in a 4-point scale (e.g., “How satisfied are you with the amount of help you received?”) with good internal consistency and concurrent validity (Attkisson & Zwick, 1982). Internal consistency in this study was satisfactory (α = .88). This instrument was used to test the convergent validity of the questionnaire of interest.

Progress in CWS: Referents from the institutions in which participants were recruited were asked to provide information about the case entry date and previous child placement. Workers were also asked to rate participants’ progress on a scale from 1 to 10 regarding their adherence to workers’ recommendations, engagement with the intervention, progress in the intervention, and current level of risk for the child.

Data Collection

Questionnaire administration

This study was part of a larger research project aimed at assessing child well-being in at-risk families. Approval from the Ethics Board of the participating universities was obtained prior to data collection. Rural and urban, region-representative child welfare agencies in the Algarve (south of Portugal) and Andalusia (south of Spain) were contacted via letter and subsequently by telephone and asked to collaborate in this project. As a result, 7 agencies from Portugal (Commissions for the Protection of Children and Youth) and 15 from Spain (Community Social Services) participated. Participants’ selection criteria were (1) being enrolled in CWS for family preservation reasons for at least 3 months, (2) having a medium-risk profile (i.e., no child out-of-home measures were to be enforced), and (3) not being in a crisis. Participants who fulfilled these criteria participated voluntarily in the study and were given an appointment for an interview in CWS facilities by their case manager. Prior to the interview, participants signed an informed consent form specifying the voluntary nature of their participation, the anonymity and confidentiality of their answers, and the option to leave the study at any stage without receiving any negative consequences. Participants were also informed that the interviewer was external to the agency and that their answers would not be revealed to the personnel of the agency. Directions were provided to the respondents concerning the items that mentioned agency workers, clarifying that they should think about the worker(s) with whom they had had most contact with. No monetary incentives were offered. Total administration length of the three questionnaires was in average 25 min. Confidentiality was a major concern throughout the study in order to preserve response veracity; therefore, workers from the participating agencies did not have access to participants’ responses.

Participants’ characteristics

The sample consisted of 263 caregivers (74.9% women) with at least one child receiving CWS, 51.7% of whom lived in Spain and the remaining 48.3% in Portugal. In most cases (94.7%), caregivers were children’s biological parents, therefore the term “parents” will be used throughout the article. Participants’ average age was 38.85 years (SD = 8.61). Educational level was mainly low, since 73.8% of the participants had not completed school beyond compulsory education. Only 14.8% were immigrants. More than half (51.7%) of the participants were unemployed when the interview took place; and families had an average income of 9,834.37€ per year (SD = 7,269.26). Participants’ households had in average four members (M = 4.01; SD = 1.45), and among these nearly two were underage children (M = 1.86; SD = 1.00). In exceptional cases (2.7%), children were living with the other parent due to a separation, but the respondent maintained a close relationship with the child. Most families were two-parent (55.9%) and among these, 55.8% were blended. Concerning participants’ marital status, 35.2% were cohabiting and 22.1% were married. Target children within each family were mainly boys (65%) with a mean age of 10 years (SD = 4.67). Only 6.0% of the families had experienced previous child placement. Participants had been enrolled in CWS in average 17.18 months since the moment of the interview (SD = 23.70). Participants from both countries were comparable in gender, age, marital status, and household size. Significant differences were observed, however, in family structure, χ²(1) = 3.97, p = .048, and educational level χ²(3) = 13.12, p = .004, with Spanish families having a significantly higher proportion of one-parent households and a higher educational level than Portuguese families.

Results

Preliminary Item Analysis

A double process of preliminary item analysis was conducted with the aim of selecting items that would maximize the variance in the instrument (e.g., allow observing interindividual differences). Ideally, items should have a high discrimination power, high SDs, and average scores around the medium point of the scale (Nunnally & Bernstein, 1994). First, an analysis of the descriptive statistics of the scale— – mean, SD, range, skewness, and kurtosis—for the whole sample was performed (Table 1). Items that did not comply with the following criteria were considered for elimination: (a) Mean between 1.5 and 3.5, (b) SD above 0.5, (c) range equal to 3, (d) skewness and kurtosis between −3 and 3, and (e) positive, significant correlation with CSQ-8. Second, the same analysis was performed separately for each country, with the aim of selecting items with similar psychometric characteristics in both groups and reinforcing item quality. The same criteria as mentioned above were applied together with (g) skewness and kurtosis in the same direction for the two groups, (h) intergroup difference in skewness lower than 1, and (i) variances ratio lower than 10. Items that did not comply with at least two of the aforementioned criteria were considered for elimination. After this selection process, Items 1, 8, 9, 10, 17, 18, 19, 20, 22, 26, and 27 were removed from the questionnaire.

Scale Dimensionality

In order to examine the factorial structure of the scale, the sample was randomly split into two equivalent halves. With the first half (n = 131), an exploratory factor analysis (EFA) was conducted with the statistical package FACTOR Version 9.2 (Lorenzo-Seva & Ferrando, 2006); with the second half (n = 132), a confirmatory factor analysis (CFA) was performed with EQS Version 6.1 (Bentler & Wu, 2002). There were no significant differences between the two halves regarding gender, age, educational level, marital status, family type, or household size. The possible influence of univariant and multivariant extreme cases was examined attending to interquartile distance from the mean and Mahalanobis distance, respectively (Tabachnick & Fidell, 2007). Nine cases (about 3%) were identified as multivariate outliers (Mahalanobis distance p < .001) and deleted from subsequent analyses.

EFA

Due to the ordinal nature of the items, the polychoric correlations matrix was used. The unweighted least squares method was chosen for factor extraction, and the oblique rotation with Promin method was used to increase interpretability, given the expected relation between the underlying matrix factors. In order to validate the correlation matrix structure, Bartlett’s sphericity test (χ² = 683.3; p < .001) and the Kaiser–Meyer–Olkin (= .85) were calculated as measures of sample adequacy, revealing good results (Tabachnick & Fidell, 2007). Criteria to retain a solution were (a) a minimum of 3 items per factor, (b) substantive coherence, (c) loadings higher than 0.45, (d) communalities not too low (< 0.4) nor higher than 1, and (e) a difference higher than 0.10 between item factor loadings (Stevens, 2002). The initial EFA indicated four factors with eigenvalues above 1, although one of the factors had only 2 items. Additionally, item 2 showed low communalities compared to other items (0.165) and loaded similarly on two factors (0.998 and 0.994). Subsequently, a 3-factor solution was forced without Item 2, explaining 63.27% of the variance. The goodness-of-fit indicators obtained (goodness-of-fit index [GFI] = .99, root mean square of residuals = .05, Bentler’s simplicity index = .86, loading simplicity index = .29) were satisfactory (Hair, Anderson, Tatham, & Black, 2009). At item level, reliability criteria for retention were (1) item-total corrected correlation with its subscale above .30 and (2) subscale Cronbach’s α decreased if item was deleted. Following the reliability analysis, Item 3 was dropped. The factors were labeled Intervention Efficacy (IE; Items 6, 11, 12, and 15); Perception of Workers (Items 7, 16, 21, 23, 24, and 25); and Satisfaction with the Intervention Process (SP; Items 4, 5, 13, and 14). All the factors had good standardized α and ω reliability coefficients (IE ω = .81, α = .81; PW ω = .82, α = .82; SP ω = .84, α = .83).

CFA

With the aim of confirming the factorial structure obtained with the exploratory analysis, a CFA was performed with the second half of the sample. All variables were defined as categorical, and the maximum likehood estimation method was used. Mardia’s coefficient indicated that the multivariate normality assumption had been violated (Mardia = 50.72). Therefore, robust estimators were used following Hair, Anderson, Tatham, and Black’ (2009) recommendations. Given the high correlations between the factors (see Table 2), two alternative models were tested: (M1) a model with interfactor correlations with a second-order factor and (M2) another model with interfactor correlations without a second-order factor. The M1 had identification problems, therefore it was rejected. According to Hair and colleagues (2009), a Satorra-Bentler χ²/df value ≤2 is considered good and values = 1 are considered very good; nonnormed fit index (NNFI) and comparative fit index (CFI) values above .90 indicate an adequate model fit; and root mean square error of approximation (RMSEA) values ≤.06 indicate good model fit, around .08 indicate adequate fit, and ≥.10 a poor fit. M2 showed acceptable fit indices, with S–Bχ² ₍₇₄₎ = 134.38, p < .001, NNFI = .92, CFI = .94, and RMSEA = .08, allowing us to confirm the solution obtained in EFA. All items had loadings well above .45 and, thus, none were removed from the model. Nonstandardized regression coefficients were significant at a p < .001 in all cases, with t values ranging from 6.58 to 25.40. Adjusted R ² values ranged from .27 to .87. No high coefficient standardized errors were observed (Hair et al., 2009).

Table 2.

Questionnaire Dimensions and Items, Reliability Coefficients, Corrected Item-Total Correlations for Each Dimension, Interdimension Correlation, and External Validity Indicators.

Dimension	Items	Reliability Coefficients (ω/std. α)	Item-Total Correlations (Range)	Inter-Dimension Correlation			Correlation With CSQ-8	Correlation With Participants’ Progress
Dimension	Items	Reliability Coefficients (ω/std. α)	Item-Total Correlations (Range)	1	2	3	Correlation With CSQ-8	Adherence	Engagement	Positive Evolution	Risk Level
1. Intervention Efficacy	6, 11, 12, 15	.82/.82	.49–.58	—	.43**	.57**	.57**	.19**	.23**	.29**	−.05
2. Perception of Workers	7, 16, 21, 23, 24, 25	.83/.83	.35–.54	—	—	.65**	.62**	.05	.07	.12	−.04
3. Satisfaction with Intervention Process	4, 5, 13, 14	.86/.86	.44–.74	—		—	.74**	.07	.16*	.15*	−.03

Note. N = 254. CSQ-8 = Client Satisfaction Questionnaire.

*Correlation is significant at .05 level. **Correlation is significant at .01 level. Correlations were calculated using Spearman’s ρ (r_s ).

Measurement invariance

In order to ensure measurement invariance, the four-step process recommended by Byrne (2006) was followed. First, the baseline models were tested for each country separately with satisfactory results (Spain: S–Bχ² ₍₇₄₎ = 189.44, NNFI = .89, CFI = .91, and RMSEA = .11; Portugal: S–Bχ² ₍₇₄₎ = 106.28, NNFI = .94, CFI = .95, and RMSEA = .06). Second, the configural equivalence was analyzed, estimating the baseline models within the framework of a multigroup model. Goodness-of-fit statistics revealed a well-fitting multigroup model, with the S–Bχ² ₍₁₄₈₎ value of 303.01 closely representing the sum of the baseline models, NNFI = .90, CFI = .92, and RMSEA = .06. Third, measurement invariance was confirmed, as no significant differences in GFIs emerged when specifications of equality constraints for factor loadings were included in the model (S–Bχ² ₍₁₆₂₎ = 330.93, NNFI = .90, CFI = .91, RMSEA = .06). Finally, structure invariance was also demonstrated when adding specifications of equality constraints for factor covariances (S–Bχ² ₍₁₆₅₎ = 333.62, NNFI = .90, CFI = .91, and RMSEA = .06).

Reliability and external validity

For reliability estimates, the one-test method was used, calculating the internal consistency of the dimensions, and good reliability coefficients were observed (see Table 2). Concerning external validity, all dimensions correlated highly with participants’ CSQ-8 scores (see Table 2), a scale that measures the same construct, confirming the questionnaire’s convergent validity. In order to test for criteria validity, correlations between the dimensions and workers’ reports of participants’ progress (adherence to workers’ recommendations, engagement with the intervention, positive evolution, and current level of risk for the child) were calculated.

As displayed in Table 2, IE had a low but significant correlation with workers’ reports of participants’ adherence to the intervention and was moderately correlated with participants’ engagement with the intervention and with their positive evolution during the intervention. PW’ scores were not significantly correlated with any of the indicators of workers’ reports of participants’ progress. As for the SP, the subscale had a low but significant correlation with participants’ engagement and with positive evolution. None of the dimensions were significantly correlated with participants’ risk level. To assess discriminant validity, we hypothesized that all dimensions should be unrelated to participants’ background characteristics. Hence, correlations between the dimensions and participants’ age, a t test comparing differences in dimensions scores between genders and an ANOVA comparing differences among educational levels, were performed. Results were overall satisfactory: Age was significantly but weakly associated with PW (r = .13, p = .042) and nonsignificantly correlated with the rest of the dimensions (IE r = .08, not statistically significant [ns]; SP r = .11, ns). No differences were observed in the dimensions’ scores between genders (t scores range = −.75–.25, ns; r = .01–.05) nor among educational levels (F scores range = .13–.86, ns; η² = .00–.01).

Discussion and Applications to Practice

The aim of this study was to develop and validate a conceptually driven questionnaire to assess parents’ perceptions and views about CWS. From a literature review, a multidimensional model of family feedback that included several aspects of users’ experience with CWS (expectations, general satisfaction, service characteristics, intervention outcomes, perception of workers, and agency intervention) was developed. From the pilot version with 27 items, those with the most adequate psychometric properties were selected, and the final form consisted of 14 items. The EFA, performed in a randomly split half of the sample, allowed us to identify three distinct, theoretically meaningful factors (IE, PW, and SP). This structure was later confirmed with the other half of the sample through a CFA with satisfactory results. All the dimensions showed good reliability indexes. Multigroup comparisons across countries allowed us to confirm instrument invariance, meaning that the interpretation of the items did not vary for each group, and therefore it is legitimate to validate the instrument simultaneously in these two cultures.

The multidimensional nature of the construct was thus confirmed; however, not all the factors we had identified through the literature review were fully replicated. All items from the General Satisfaction subscale were kept, although they split into two different dimensions: IE and SP. It thus seems that general satisfaction, a traditional dimension that has been by far the most studied judging from the content of existing questionnaires (e.g., McMurtry & Hudson, 2000; Larsen et al., 1979; Winefield & Barlow, 1995), is an important element to assess the experience with CWS. Nonetheless, in the dimensionality analysis, a factor that combined elements of satisfaction with others focused on the experience with the intervention emerged. This dimension accounts for aspects related to the process, beyond the intervention outcomes, that is, how the intervention has been experienced in terms of learning new tools and skills. This finding underlines the importance of assessing not only outcomes but also the way in which the process is experienced and the coping skills acquired during the intervention (Cortis, 2007).

The hypothesized dimension outcomes were partially replicated in the EFA and relabeled as IE, since the items in this factor referred to perceptions of changes occurred in family’s problems due to the intervention and its usefulness. Regarding the factor PW, although some items from other proposed subscales loaded on this dimension (e.g., items 24 and 25, which were conceptualized as part of the subscale Intervention), all of them referred to agency staff and therefore is not surprising that these grouped into the same factor. The importance of the relationship quality with CWS workers has been emphasized by other researchers (see Marsh, Angell, Andrews, & Curry, 2012 for a review), and the strength of the parent–worker relationship has been shown to predict outcomes as relevant to CWS as improvements in child safety (Johnson & Ketring, 2006; Lee & Ayón, 2004). Previous research with Portuguese and Spanish CWS users has pointed to the significant presence of social agencies workers in the emotional social support networks of at-risk families (Ayala-Nunes, Nunes, & Lemos, 2017; Rodrigo & Byrne, 2011). It seems therefore advisable to emphasize the therapeutic relational aspects in workers’ training.

Almost all items referring to service characteristics (such as location and office hours) were discarded due to their unsatisfactory results, revealing that this dimension was not as relevant to parents’ evaluations of CWS as other aspects. It is worth noting that questions concerning the more tangible and practical aspects of the services are more often than not measured as core components of user satisfaction (Kapp & Vela, 1999). Finally, all the items pertaining to the initial subscale Expectation were dropped, suggesting that this aspect of the experience does not contribute significantly to the evaluation of the services, contrarily to what the extant literature posits for community populations (e.g., Pascoe, 1984). It is of course possible that this is due to flaws in item design, but another possible explanation is the complexity of this construct, which may be difficult to grasp for participants who have a low educational level. Reporting one’s expectations retrospectively implies not only recalling thoughts and feelings about a vague entity (e.g., an unknown service) but also comparing them with the current ones and deciding whether their expectations have been met or not. It may also be that some families hold misadjusted expectations about this type of services, since CWS still has strong negative social representations in some south European countries. Perhaps this is due to the deficit perspective that predominated in family and child community services until the late 1980s (Hidalgo, Menéndez, Sánchez, Lorence, & Jiménez, 2010); thus, these interventions may be initially seen as unnecessary, invasive, or even threatening.

The FF-CWS showed good external validity, since it correlated with a previously validated measure of the same construct and with external indicators reported by other informants. Results concerning its discriminant validity were also satisfactory: Consistent with McMurtry and Hudson (2000), we found that in general the dimensions of the FF-CWS were unrelated to participants’ background characteristics, with the exception of PW, which was significantly but weakly associated with participants’ age. Regarding criterion validity, IE and SP were moderately related to three of the four aspects of parents’ progress in the intervention assessed by agency workers: adherence, engagement, and positive evolution. These findings support the external validity of the scales. However, none of these variables was significantly related to PW. It has been previously noted that there is a tentative relationship between user satisfaction and some outcomes (Larsen et al., 1979). It is not surprising that progress indicators are more linked to participants’ views about the utility of the intervention than to how they assess workers at a personal level and their relationship with them. Additionally, none of the dimensions correlated significantly with current risk level of participants’ children. Probably, there are other factors that explain children’s risk level and are unrelated to parental level of engagement or willingness to follow workers’ recommendations. This could indicate that the FF-CWS allows the assessment families’ feedback about services irrespective of their risk level. It must be highlighted the variables chosen for testing external validity were reported by different informants; and multiple informants’ reports typically exhibit low-to-moderate levels of correspondence (Achenbach, McConaughy, & Howell, 1987).

Despite the relevance of these results, some limitations should be noted. The assessment of FF-CWS poses many theoretical and methodological challenges, in aspects such as identifying core constructs, sampling bias, participants’ recruitment and accessibility, timing of data collection, maintaining confidentiality, ensuring interviewer impartiality, and validity and reliability problems (Baker, 2007; Tilbury et al., 2010). In line with these challenges, this study had some limitations, most of them related to the fact that CWS users constitute a hard-to-reach population. First, the sample size is small. Second, it is possible that we oversampled participants who were satisfied with CWS. High participant mobility, low literacy skills, mental health or addiction problems, or participants having no telephone or changing number frequently are common sampling issues (Tilbury et al., 2010). Difficulties in establishing contact and limited access to CWS users are also frequent and may compromise the generalizability of the results to all service users because it is likely that less satisfied families will be harder to reach (Gain & Young, 1998).

Furthermore, it would have been desirable to include a focus group of CWS users in order to identify core constructs relevant to evaluate their experience with these services. For instance, Green, McAllister, and Tarte (2004) based the conceptual model that guided the development of the SBPI on both a review of the literature and focus groups with parents involved with a strengths-based family support program. Hsieh (2012) also followed an interesting line of work, advocating the incorporation of the perceived importance of service elements into client satisfaction measures.

The piloting of the initial form of the questionnaire was conducted with a small sample due to the accessibility issues that characterize research with clinical groups. Perhaps if a bigger sample would have been consulted regarding item readability, additional improvements could have been made in the questionnaire. Additionally, high user satisfaction rates are pervasive in research (Larsen et al., 1979; Pascoe, 1984), and this study was no exception, with many items being negatively skewed. If participants believe that their responses will be revealed to the agency staff, they may minimize or omit criticisms (Haight et al., 2002). We were aware of the importance of maintaining confidentiality, and participants were reassured before the interview that workers would not have access to their individual survey responses. It may be that acquiescence or social desirability effects were not fully neutralized, despite the precautions taken to avoid them.

Future investigations should aim to reinforce the questionnaire’s discriminant validity by following longitudinally participants’ outcomes and testing whether the FF-CWS succeeds at differentiating those cases in which child out-of-home placement is decided from those who maintain child custody. It would also be interesting to adapt the questionnaire to children and adolescents as a way of giving them a voice in a system in which they are the protagonists, but their opinions and views are even less solicited that those of their parents (Committee of Ministers of the Council of Europe, 2011). In sum, incorporating users’ views, increasing sample size, piloting the questionnaire with more participants in order to refine it, administering the questionnaire outside the agencies’ facilities in order to neutralize the social desirability bias, reinforcing discriminant validity through a follow-up, developing and validating a child and adolescent form of the questionnaire as well as adapting it to other countries could be the next steps for future researches.

Despite the aforementioned limitations, this study has also several strengths: It has developed, to the best of our knowledge, one of the few conceptually driven, validated questionnaires that assesses families’ views of their experience with CWS. Although consensus has not yet been reached concerning the core constructs of feedback on CWS (Baker, 2007), hopefully, this study has contributed to highlight some of the relevant elements to measure. The FF-CWS has shown to be valid for two countries with different service structure and organization. Also, fathers who are frequently absent in child welfare research and practice (Brown, Callahan, Strega, Walmsley, & Dominelli, 2009) were included in the sample, which reinforces the generalizability of the questionnaire to both genders.

In sum, the FF-CWS is a brief, freely accessible, self-administrable questionnaire with a basic reading level with good reliability and validity indicators that provide information about three important aspects of families’ perceptions of CWS: How CWS users perceive the efficacy of the intervention, how they view workers, and how satisfied they are with the process (see Appendix). Therefore, it constitutes an easy-to-administrate, cost-effective, and useful assessment tool to frontline practitioners and agency managers. Relying on sound assessment instruments to evaluate users’ perceptions contributes to institutional transparency and accountability, to evidence-based interventions, and to the delivery of family-centered and strengths-based family interventions.

Footnotes

Appendix

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by the Portuguese Foundation for Science and Technology (Fundação para a Ciência e a Tecnologia; Grant SFRH/BD/86172/2012), under the program QREN—POPH with co-financing of the European Social Fund and the Portuguese Ministry of Education and Science (MEC) through national funds.

References

Achenbach

T. M.

McConaughy

S. H.

Howell

C. T.

(1987). Child/adolescent behavioral and emotional problems: Implications of cross-informant correlations for situational specificity. Psychological Bulletin, 101, 213–232.

Alpert

L. T.

(2005). Research review: Parents’ service experience—a missing element in research on foster care case outcomes. Child & Family Social Work, 10, 361–366. doi:10.1111/j.1365-2206.2005.00387.x

Alpert

L. T.

Britner

P. A.

(2009). Measuring parent engagement in foster care. Social Work Research, 33, 135–145. doi:10.1093/swr/33.3.135

Attkisson

C. C.

Zwick

(1982). The Client Satisfaction Questionnaire: Psychometric properties and correlations with service utilization and psychotherapy outcome. Evaluation and Program Planning, 5, 299–314.

Ayala-Nunes

Jiménez

Hidalgo

Jesus

(2014). Family feedback in Child Welfare Services: A systematic review of measures. Children and Youth Services Review, 44, 299–306. doi:10.1016/j.childyouth.2014.07.004

Ayala-Nunes

Nunes

Lemos

(2017). Social support and parenting stress in at-risk Portuguese families. Journal of Social Work, 17, 207–225.

Baker

A. J.

(2007). Client feedback in child welfare programs: Current trends and future directions. Children and Youth Services Review, 29, 1189–1200. doi:10.1016/j.childyouth.2007.05.003

Bentler

(2002). EQS for Windows user’s guide. Encino, CA: Multivariate Software.

Boletín Oficial del Estado. (1996). Ley Orgánica 1/1996, de 15 de enero, de Protección Jurídica del Menor, de modificación parcial del Código Civil y de la Ley de Enjuiciamiento Civil. BOE number 15 January 17th 1996. Madrid, Spain: Ministerio de la Presidencia.

10.

Brown

Callahan

Strega

Walmsley

Dominelli

(2009). Manufacturing ghost fathers: The paradox of father presence and absence in child welfare. Child & Family Social Work, 14, 25–34.

11.

Buckley

Carr

Whelan

(2011). ‘Like walking on eggshells’: Service user views and expectations of the child protection system. Child & Family Social Work, 16, 101–110. doi:10.1111/j.1365-2206.2010.00718.x

12.

Byrne

(2006). Structural equation modeling with EQS. Mahwah, NJ: Lawrence Erlbaum.

13.

Carretero-Dios

Pérez

(2005). Normas para el desarrollo y revisión de estudios instrumentales. International Journal of Clinical and Health Psychology, 5, 521–551.

14.

Comissão Nacional de Proteção das Crianças e Jovens em Risco. (2014). Avaliação da atividade das Comissões de proteção das crianças e jovens. Relatório anual 2014. Retrieved from http://www.cnpcjr.pt/preview_documentos.asp?r=5603&m=PDF

15.

Committee of Ministers of the Council of Europe. (2006). Recommendation 19. Retrieved from https://wcd.coe.int/ViewDoc.jsp?id=1073507

16.

Committee of Ministers of the Council of Europe. (2011). Recommendation 12. Retrieved from https://wcd.coe.int/ViewDoc.jsp?id=1872121

17.

Connolly

(2007). Practice frameworks: Conceptual maps to guide interventions in child welfare. British Journal of Social Work, 37, 825–837. doi:10.1093/bjsw/bcl049

18.

Cortis

(2007). What do service users think of evaluation? Evidence from family support. Child & Family Social Work, 12, 399–408. doi:10.1111/j.1365-2206.2007.00495.x

19.

Damashek

Doughty

Ware

Silovsky

(2011). Predictors of client engagement and attrition in home-based child maltreatment prevention services. Child Maltreatment, 16, 9–20. doi:10.1177/1077559510388507

20.

DeVellis

R. F.

(2003). Scale development: Theory and applications (2nd ed). Thousand Oaks, CA: Sage.

21.

Gain

Young

(1998). Outcome measurement in child protection: International literature review and critical analysis of child protection and alternative placement outcome measures. Melbourne, Australia: Steering Committee for the Review of Commonwealth/State Service Provision.

22.

Gerkensmeyer

J. E.

Austin

J. K.

Miller

T. K.

(2006). Model testing: Examining parent satisfaction. Archives of Psychiatric Nursing, 20, 65–75. doi:10.1016/j.apnu.2005.09.001

23.

Girvin

DePanfilis

Daining

(2007). Predicting program completion among families enrolled in a child neglect preventive intervention. Research on Social Work Practice, 17, 674–685. doi:10.1177/1049731507300285

24.

Green

B. L.

McAllister

C. L.

Tarte

J. M.

(2004). The Strengths-Based Practices Inventory: A tool for measuring strengths-based service delivery in early childhood and family support programs. Families in Society: The Journal of Contemporary Social Services, 85, 326–334. doi:10.1606/1044-3894.1493

25.

Haight

W. L.

Black

J. E.

Mangelsdorf

Giorgio

Tata

Schoppe

S. J.

Szewczyk

(2002). Making visits better: The perspectives of parents, foster parents, and child welfare workers. Child Welfare, 81, 173–202.

26.

Hair

J. F.

Anderson

R. E.

Tatham

R. L.

Black

W. C.

(2009). Multivariate data analysis: A global perspective (7th ed). Upper Saddle River, NJ: Prentice Hall.

27.

Harris

Poertner

(1998). Measurement of client satisfaction: The state of the art. Urbana, IL: Children and Family Research Center.

28.

Harris

Poertner

Joe

(2000). The parents with children in foster care satisfaction scale. Administration in Social Work, 24, 15–27. doi:10.1300/J147v24n02_02

29.

Hidalgo

M. V.

Menéndez

Sánchez

Lorence

Jiménez

(2010). Intervention with at-risk families: Contributions from a psycho-educational perspective. Psychology in Spain, 14, 48–56.

30.

Hsieh

C. M.

(2012). Incorporating perceived importance of service elements into client satisfaction measures. Research on Social Work Practice, 22, 93–99.

31.

Huebner

R. A.

Jones

B. L.

Miller

V. P.

Custer

Critchfield

(2006). Comprehensive family services and customer satisfaction outcomes. Child Welfare, 85, 691–714.

32.

Johnson

L. N.

Ketring

S. A.

(2006). The therapy alliance: A moderator in therapy outcome for families dealing with child abuse and neglect. Journal of Marital and Family Therapy, 32, 345–354. doi:10.1111/j.1752-0606.2006.tb01611.x

33.

Kapp

S. A.

Vela

R. H.

(1999). Measuring consumer satisfaction in family preservation services: Identifying instrument domains. Journal of Family Strengths, 4, 19–36.

34.

Kapp

S. A.

Vela

R. H.

(2003). The parent satisfaction with foster care services scale. Child Welfare, 83, 263–287.

35.

Korfmacher

Green

Spellmann

Thornburg

K. R.

(2007). The helping relationship and program participation in early childhood home visiting. Infant Mental Health Journal, 28, 459–480. doi:10.1002/imhj.20148

36.

Larsen

D. L.

Attkisson

C. C.

Hargreaves

W. A.

Nguyen

T. D.

(1979). Assessment of client/patient satisfaction: Development of a general scale. Evaluation and Program Planning, 2, 197–207.

37.

Lee

C. D.

Ayón

(2004). Is the client-worker relationship associated with better outcomes in mandated child abuse cases? Research on Social Work Practice, 14, 351–357. doi:10.1177/1049731504265833

38.

Lei nº 147/99 (1999) de 1 de Setembro. Lei de protecção de crianças e jovens em perigo. Diário da República nº 204/99, I série A. Assembleia da República. Lisboa.

39.

Lorenzo-Seva

Ferrando

P. J.

(2006). Factor: A computer program to fit the exploratory factor analysis model. Behavioral Research Methods, Instruments and Computers, 38, 88–91. doi:10.3758/BF03192753

40.

Marsh

J. C.

Angell

Andrews

C. M.

Curry

(2012). Client–provider relationship and treatment outcome: A systematic review of substance abuse, child welfare, and mental health services research. Journal of the Society for Social Work and Research, 3, 233–267. doi:10.5243/jsswr.2012.15

41.

McMurtry

S. L.

Hudson

W. W.

(2000). The Client Satisfaction Inventory: Results of an initial validation study. Research on Social Work Practice, 10, 644–663.

42.

Nunnally

J. C.

Bernstein

I. H.

(1994). Psychometric theory (3rd ed.). New York, NY: McGraw-Hill.

43.

Pascoe

G. C.

(1984). Patient satisfaction in primary health care: A literature review and analysis. Evaluation and Program Planning, 6, 185–210.

44.

Pérez-Padilla

Ayala-Nunes

Hidalgo

M. V.

Nunes

Lemos

Menéndez

(2015). Parenting and stress: A study with Spanish and Portuguese at-risk families. International Social Work. Advance online publication. doi:10.1177/0020872815594220

45.

Pollitt

(1998). Bringing consumers into performance measurement: Concepts, consequences and constraints. Policy and Politics, 16, 77–87.

46.

Rodrigo

M. J.

Byrne

(2011). Social support and personal agency in at-risk mothers. Psychosocial Intervention, 20, 13–24.

47.

Southerland

D. G.

Mustillo

S. A.

Farmer

E. M. Z.

Stambaugh

L. F.

Murray

(2009). What’s the relationship got to do with it? Understanding the therapeutic relationship in therapeutic foster care. Child and Adolescent Social Work Journal, 26, 49–63. doi:10.1007/s10560-008-0159-4

48.

Stevens

(2002). Applied multivariate statistics for the social sciences (4th ed.). Hillsdale, NJ: Lawrence Erlbaum.

49.

Tabachnick

B. G.

Fidell

L. S.

(2007). Using multivariate statistics. Boston: Pearson/Allyn & Bacon.

50.

Tilbury

Osmond

Crawford

(2010). Measuring client satisfaction with child welfare services. Journal of Public Child Welfare, 4, 77–90. doi:10.1080/15548730903563160

51.

Trotter

(2008). What does client satisfaction tells us about effectiveness? Child Abuse Review, 17, 262–274. doi:10.1002/car.1038

52.

U.S. Department of Health and Human Services, Administration for Children and Families, Administration on Children, Youth and Families, Children’s Bureau. (2015). Child maltreatment 2013. Retrieved from http://www.acf.hhs.gov/programs/cb/research-data-technology/statistics-research/child-maltreatment

53.

Winefield

H. R.

Barlow

J. A.

(1995). Client and worker satisfaction in a child protection agency. Child Abuse & Neglect, 19, 897–905.