Abstract
This study evaluated the reliability and criterion validity of the Byumba Social Functioning Questionnaire (BSFQ), an instrument to measure social functioning in Rwanda. The instrument was locally composed in concordance with a well-described method for culture-specific and sex-specific function assessment. Respondents in a Northern Province of Rwanda (N = 393) were assessed with the BSFQ and a 19-item scale (SF-19) drawn from the Medical Outcomes Study 36-Item Short-Form (SF-36). The BSFQ’s internal consistency was just acceptable for women but poor for men, while the SF-19 had good to excellent internal consisteny. BSFQ total scores showed a strong floor effect, while the SF-19 showed more variation in total score distribution. The BSFQ did not perform as well as we expected, and appears not to be suitable for measuring social functioning in the study context. This outcome seems to reflect the conceptualization of social functioning used in constructing the BSFQ. Implications for the development of culture-specific measures of functional status are discussed.
Introduction
In Rwanda, the genocide in 1994 severely damaged social cohesion. From 2006 onwards, a community-based psychosocial intervention program has been conducted in Byumba, northern Rwanda. This makes use of a therapeutic group approach called sociotherapy (Richters, Dekker, & Scholte, 2008; Scholte et al., 2011; Verduin, Scholte, Rutayisire, & Richters, 2010). As the intervention primarily aims at social bonding, it might be expected that it also impacts on social functioning. As a step toward examining this outcome, the present study examined the validity of an instrument developed to measure social functioning as defined by members of the local population.
To date, there is no standardized, widely accepted definition of social functioning (Hirschfeld et al., 2000). The combination of the words “social” and “functioning” suggests it refers to social role behaviours related to individuals’ desired or preferred social positions (Blakely & Dziadosz, 2007). Social scientists like Bartlett (1970) and Boehm (1958), have stated that social functioning is defined as normative behaviour in social situations. A practical definition was provided by Paykel (1999, p. 10), who described social functioning as “an individual’s ability to function within their usual environment.” Diagnostic instruments generally include the following domains of social functioning: occupation, household rule, marital functioning, parental role, family/kinship role, social role, leisure/general interest, and self-care (Hirschfeld et al., 2000; Wiersma, 1996).
Social norms and role behaviours vary among groups and cultures, and an instrument to assess social functioning needs to have validity for the chosen population. In this regard, many existing instruments pose major problems. For example, instruments developed in Western countries contain many culture-bound questions that are difficult to adapt to other contexts, and do not acknowledge the major differences in the roles of men and women that still predominate in most low-income countries (Bolton & Tang, 2002).
Seeking to avoid these problems, Bolton and Tang (2002) developed a method for cross-cultural and sex-specific functional assessment that can be used within any population. They realized that highly trained clinical personnel familiar with the local culture are often not available (Ertl et al., 2010). By taking an interactive construction approach (Bullinger, Anderson, Cella, & Aaronson, 1993), they created a new method that was applicable with limited resources. This method uses a template that lists tasks and activities that adults in the particular culture are expected to perform regularly. Separate lists are constructed for men and women. Respondents are then asked to rate how much difficulty they experience in completing each of these tasks, rated on a 5-point scale labeled: None = 0, Little = 1, Moderate = 2, A Lot = 3, and Often Can't Do Task = 4. For each task or activity, the reasons for difficulty are also recorded. Bolton and Tang applied this method to design a functional assessment tool for use in a setting in southern Rwanda. For validation purposes, they established the instrument’s internal reliability, test–retest reliability, and the accuracy of self-assessment using the instrument.
Given that the original validation process had taken place in Rwanda, it seemed logical to adapt Bolton and Tang’s instrument for our study in northern Rwanda. We developed an adapted version using their approach, and called it the Byumba Social Functioning Questionnaire (BSFQ). In line with the original instrument, the domains of social functioning assessed by the BSFQ are: (a) care for oneself, (b) care for family, and (c) care for the community.
The present study aimed to test the reliability and the criterion validity of the BSFQ. Usually criterion validity would be done by comparing the instrument locally against another measure or indicator of known accuracy, but no such instrument exists for the Rwandan context. We therefore compared the BSFQ to the Medical Outcomes Study’s 36-Item Short-Form (SF-36).
Methods
Setting
The intervention area Byumba is located in the north of Rwanda, bordering Uganda. The region suffered heavily from the war that resulted from Rwandan Patriotic Front attacks (1990–1994) and the genocide (1994). Community life is largely dominated by the Anglican Church in Rwanda (Eglise Anglicane du Rwanda [EAR]), the Diocese of Byumba. Under the new administrative scheme of Rwanda, the Northern Province covers five districts, 40 sectors, hundreds of cells, and thousands of villages (Imidugudu), the latter being the smallest administrative unit in Rwanda. Up to 150 households on average form a village; each village is headed by a village leader. Ten to 12 of these villages form a cell, and each cell also has a leader. Five to six of these cells form a sector headed by an executive secretary.
Baseline measurements for this study took place in October 2007. Approval was gained from the Medical Ethics Committee of the Academic Medical Center in Amsterdam. Trial Registration: Netherlands Trial Register NTR1120.
Instruments
The Byumba Social Functioning Questionnaire (BSFQ)
Our methodology started with focus group discussions on the existing list of Bolton and Tang (2002) for Rwanda, and proceeded with free listing of items to be added to the existing list. Finally, all items were ranked. This resulted in 10 items for men and 10 for women to form our questionnaire.
Sociodemographic characteristics of all respondents (N = 393).
The leading question for each item would be “Do you have difficulty completing the following activity?” For each activity listed, respondents would be asked to indicate how much difficulty they experienced in completing the activity. Response categories would be read out loud to the respondent. To illustrate the meaning of each response category, interviewers could make use of a card with a series of drawings similar to the one used by Bolton and Tang (2002); this nonverbal response card shows five drawings of a person carrying an increasingly heavy weight (Bolton & Tang, 2002, Figure 2.).
The original template included a section on causes for difficulty on each task (Bolton & Tang, 2002). As the sole purpose of this section was to determine whether the difficulty in functioning was health-related or due to economic limitations, it was decided to simplify the template and handle this issue differently in the present study. The section in question was left out and interviewers were trained to always probe what the cause of the difficulty was whenever respondents would report one. If the nature of the cause appeared to be economic, the interviewer would ask whether the difficulty would persist if this economic limitation were removed. For example, such a question could be: “If you had soap, would you still have difficulty washing yourself?” If the answer were negative, this would be interpreted as “no difficulty” and the interviewer would tick “None.” For the cause of validity of the responses, the concept of health was extensively discussed with the interviewers first. The Kinyarwandan word for “health” appeared to cover the general notion of physical and psychological well-being. Lastly, “Not applicable” was added as a response option in the questionnaire. This would apply if a respondent had no tasks in taking care of children or no cattle to take care of.
From SF-36 to SF-19
The SF-36 is a multipurpose, short-form instrument which contains 36 questions. It is a generic measure of health status, as opposed to one that targets a specific age, disease, or treatment group. It was constructed to provide a basis for comparisons about population health in terms of functional health and well-being (Ware & Gandek, 1998). It has been widely used all over the world (Hoopman, Terwee, Devillé, Knol, & Aaronson, 2009; Ware & Gandek, 1998), not only in Western contexts, but also in low-income countries (Cross, Edwards, Nyeko, & Edwards-Jones, 2009; Fan, Kuo, Kao, Morisky, & Chen, 2011; Maharaj & Chetty, 2011). The SF-36 has been translated for use in over 50 countries and its results have been reported in over 4,000 publications. Studies to date have yielded content, concurrent, criterion, construct, and predictive evidence of validity (Ware, 2000) in many varied populations, which is suggestive of the instrument’s validity in Rwanda as well. The SF-36 yields an eight-scale profile of scores. We used four of the eight scales of the SF-36, carefully selecting the scales that address social functioning and daily activities (Ware & Sherbourne, 1992) and cover the same domains as the BSFQ.
The SF-36 Health Survey represents multidimensional health concepts and measurement of the full range of health states (during the previous 4 weeks), including levels of well-being and personal evaluations of health. The instrument is suitable for self-administration or administration by a trained interviewer. It has eight scales formulated as statements or questions. There is one version, to be applied for both sexes. A score from 0 to 100 is calculated for each scale, with higher scores indicating a better health-related quality of life (HRQOL; McHorney, Ware, Lu, & Sherbourne, 1994; Ware, 1988). The SF-36 items and scales were constructed using the Likert method of summated ratings. Analysis and interpretation of the resulting linear scales assumes that item scores, on average, are linearly related to the underlying health concept being measured (Hays, Sherbourne, & Mazel, 1993; Ware & Sherbourne, 1992). The median reliability coefficient for each of the eight scales was equal or greater than .80, except for the Social Functioning scale (SF), which had a median reliability of .76 (McHorney, Ware, & Raczek, 1993). Relative to the longer measures that the scales of SF-36 were constructed to reproduce, they have been shown to achieve 80–90% of their empirical validity.
As we aimed to measure the same construct (social functioning) in both instruments, we needed to select items from the SF-36 that matched with the BSFQ. We identified four of the eight scales of the SF-36 that showed qualitative similarity, comprising five questions with 19 items in total. The items in the four selected scales are most comparable to the tasks listed in the BSFQ, addressing health-related limitations, either physical or emotional, in daily activities, work, and social activities (see Appendix C, available online with this article). The four scales have shown reliability estimates (Cronbach’s alphas) ranging from .83 to .93 (McHorney et al., 1994). In view of the number of items, we refer to this newly created instrument as the SF-19. The four scales selected for the SF-19 are:
Physical functioning (Question 3 in the SF-36, Question 1 in the SF-19, with 10 items): health-related limitations on daily activities such as bathing, getting dressed, and lifting shopping bags. Physical role functioning (Question 4 in SF-36, Question 2 in the SF-19, with four items): problems with work and other daily activities due to physical health problems. Emotional role functioning (Question 5 in the SF-36, Question 3 in the SF-19, with three items): problems with work and other daily activities due to emotional problems. Social functioning (Questions 6 and 10 in the SF-36, Questions 4 and 5 in the SF-19, with one item each): limitations on social activities such as visiting friends and relatives due to physical health or emotional problems.
The selected questions were back-translated to Kinyarwanda (Verduin et al., 2010). Like in the IQOLA project (International Quality of Life Assessment; Wagner et al., 1998), the most difficult items to translate and reword in a context-sensitive way proved to be “physical functioning” items that refer to activities like vacuum cleaning, playing golf or bowling, and walking a block or a mile. These items were discussed with two key informants; one was a highly ranked local leader at the Diocese of the Episcopal Church, who was not involved in the research; the other one (ThR, one of the authors) was born and raised in rural Rwanda, and is fluent in Kinyarwanda and English. The activities mentioned were changed to activities with higher local relevance, such as sweeping and fetching water (Item 1b). Climbing flights of stairs was changed to climbing a hill or a terrace (Items 1d and 1e). Walking blocks or miles was changed to walking average distances related to villages or settlements (Items 1g, 1h, and 1i).
Interviews
The longitudinal intervention study prompting the present study followed a quasi-experimental design (Scholte et al., 2011; Verduin et al., 2010). A total of 486 respondents were questioned by eight trained Rwandan interviewers. All interviews were taken in Kinyarwanda. Preceding each interview, informed consent was obtained by use of an explanatory text, which, because of the high illiteracy rate, was read aloud. A structured interview form was used, composed of several questionnaires including the BSFQ and the SF-19. The BSFQ was administered after a screener for mental health (the SRQ-20) and the SF-19 was administered right after that. Both the BSFQ and the SF-19 took about 5 minutes to administer.
Data analysis
The BSFQ response categories were given a 0 to 4 Likert scale: 0 for “No degree of difficulty” and 4 for “Often can’t do.” The total score of the BSFQ was the simple sum score of the 10 responses. Before analysis, all scales of the SF-19 were recoded following the instructions by Ware (Hays et al., 1993; Ware, 1988). Each raw score was transformed to fit a 0–100 scale by using a standard formula, with the higher scores representing better functioning. Data were used from all respondents of the longitudinal intervention study mentioned above, except those who had indicated that one or more tasks of the BSFQ were not applicable (n = 393). There was much variation in items that were indicated as nonapplicable; men indicated “Advising the family” most often; for women this was “Caring for children.” The sample was stratified for gender, because the BSFQ was different for men and women. Internal consistencies of the BSFQ and the SF-19 were assessed by calculating Cronbach’s alphas and conducting item analyses. The association between both instruments was established by calculating Spearman’s rho. The relation between the BSFQ and the SF-19 is presented through a scatter plot of total scores on both instruments, to provide information on floor and ceiling effects. As the category “Other” in the BSFQ yielded very few additional tasks, these were not included in the analysis.
Results
The final study sample comprised 181 men and 212 women. Table 1 shows their sociodemographic characteristics. More than 50% were married farmers without formal education.
For the BSFQ, a higher total score indicates more difficulties in performing tasks. The highest possible total BSFQ score is 40, meaning a person has major difficulties in performing all 10 tasks. In our sample, maximum total BSFQ scores were 16 for men and 26 for women. Thirty-nine percent of the sample (153/393) scored 0, indicating no difficulties at all. Women scored slightly higher in general.
For the SF-19, a higher total score indicates better functioning. The highest possible SF-19 total score is 400, meaning a person has no difficulty in performing any task. In our sample, the lowest SF-19 total score was 30, the highest 400.
Distribution of total scores (n, %) on BSFQ and SF-19 across all respondents.
Low score = good social functioning.
High score = good social functioning.
Reliability
The internal consistency of the BSFQ was acceptable for women (Cronbach’s alpha .75) and questionable for men (Cronbach’s alpha .64). The internal consistency of the SF-19 was good (Cronbach’s alpha .90). When calculated for men and women separately, the SF-19’s Cronbach’s alpha was .90 for women and .88 for men. Cronbach’s alphas are not substantially increased by the removal of any item in either of the instruments.
Association between the two instruments
There was a small linear association between the two instruments. Spearman’s rho was −.426 for men (p < .001) and −.512 for women (p < .001). To examine the association between the two instruments further, total scores of the BSFQ and the SF-19 were plotted against each other (see Figure 1). For respondents scoring 0 on the BSFQ, SF-19 total scores ranged between 72.5 and 400 (see Figure 2). Both Figure 1 and 2 represent scores of women only; the figures for men look similar.
Scatter plot of BSFQ and SF-19 total scores of all female respondents (n = 212). Frequency of SF-19 total scores of female respondents scoring 0 on BSFQ (n = 79).
Score distribution of SF-19
SF-19 scores (n, %) of all respondents scoring 0 on BSFQ (n = 153).
Discussion
This study evaluated the reliability and criterion validity of an instrument (BSFQ) for culture-specific assessment of social functioning. The development of the BSFQ at our study site in Rwanda followed an approach described by Bolton and Tang (2002). On the basis of their findings it was plausible to expect sufficient reliability and validity of the instrument. We compared the BSFQ with another instrument (the SF-19), which was composed of relevant scales of the SF-36. Although proof is also lacking for the SF-36’s validity for use in Rwanda, its proven validity in over 50 other contexts is sufficiently suggestive of it. The present study shows that the four selected scales of the SF-36 also have strong internal consistency. The BSFQ shows disappointing properties, which seems to be caused by both validity issues and the conceptualization of social functioning.
The BSFQ’s internal consistency is just acceptable for women but questionable for men, versus good to excellent values for the SF-19. It should be noted, however, that the SF-19’s internal consistency may be inflated by its smaller number of response options. BSFQ total scores show a strong floor effect, thereby proving incapable of showing variation at the bottom of the scale. The SF-19 shows more variation in total score distribution, even though it consists of seven questions with only “Yes/No” response options.
The latter two findings may have various possible causes. It may relate to the activities the respective instruments address, the way items are worded, and the instruments’ respective places in the order of questionnaires administered during the present study.
The range of human activities addressed when using the term “social functioning” as defined for this study encompasses two domains: caring activities for oneself or others and activities to take part in community life. The latter is commonly referred to as “social participation.” Both instruments may differ in the weight put on either of these two domains. Out of the 10 activities addressed by the BSFQ version for men, six (1–4, 6, and 10) refer to caring, and four (5, 7–9) to social participation. For the BSFQ version for women, this is seven (1–6, 10) versus three (7–9). The SF-19 may work differently. Five (c–e, g–i) out of the 10 activities addressed in Question 1 may refer to either domain. The same applies to the way Questions 2 and 3 are phrased, while Questions 4 and 5 unambiguously refer to social participation. Thus, while the BSFQ seems to put more weight to the ability to perform caring tasks than to participation in communal life, the SF-19 may work the opposite way.
In addition, the atmosphere of distrust and social fragmentation still prevailing in postgenocide Rwanda adds to the relevance of social participation as a domain to be addressed when measuring social functioning. Specifically for this reason, the wider range of responses yielded by the SF-19 may be related to its greater focus on social participation than the BSFQ. The fact that Question 3 (focusing on emotional problems) of the SF-19 contributes less to the variety in responses than Question 2 (focusing on physical health), may relate to the fact that, in this culture, emotional problems are more readily expressed through physical complaints. This applies especially to men, who tend to deny emotional problems completely (Scholte et al., 2011).
Another factor of relevance may be formed by the specificity of activities addressed by the BSFQ; activities mentioned may be too specific to capture social functioning in general. Listing a selected group of activities means leaving other activities unmentioned. Although the focus group discussions serving the instrument’s development were extensive, tasks may have been left out which are relevant for younger adults or unmarried individuals, or may have been added while irrelevant to them. Likewise, activities included as an item of the instrument, such as participating in a burial ceremony or a folkloric evening, may not apply for these groups. Also, the BSFQ items addressing social participation may have been too specific. In a context of distrust these items may have caused respondents to be restrained and to respond in a way they thought would be desirable.
In total, 80 men and 61 women mentioned one or more tasks as not applicable to them. For men, the most nonapplicable item (n = 37) was “Advising the family”; for women this was “Taking care of children” (n = 34). Eight and seven other items were only incidentally considered nonapplicable by male and female respondents, respectively.
The activities mentioned in the original SF-36 constitute culture-bound items which may be difficult to interpret across populations living in contexts with varying physical requirements (Bolton & Tang, 2002). Although this is obviously the case for a Rwandan population, the problem could be preempted sufficiently by adapting items to the local context whenever needed, as described in the Methods section.
In the BSFQ, the wording of the leading question may have complicated the differentiation between health and economic reasons as the cause of difficulties in performing tasks. We experienced that this remained problematic, even while interviewers were trained to only record a difficulty if respondents indicated causes related to physical, mental, or emotional problems. In the wording of the SF-19 items, physical and emotional problems are explicitely mentioned. In future studies it might be helpful to rephrase the leading question of the BSFQ, mentioning health reasons for each difficulty.
In the order of questionnaires within the complete interview administered during our overall study, the placing of the two questionnaires on social functioning may have impacted the responses. The SF-19 was taken right after the BSFQ. While responding to the BSFQ, respondents may have been sensitized to focus on their limitations in functioning. When their limitations were then further questioned through the SF-19, the validity of responses to the SF-19 may have increased.
Conclusion
The BSFQ was developed in concordance with a well-described and convincing method, by use of an interactive construction approach, and it is a culture- and sex-specific instrument. However, it did not perform as well as we expected, and it appeared not to be suitable to measure social functioning in the study context. It may measure limitations in practical activities of daily living rather than in social participation. Although possible differences in the roles of men and women in low-income countries are not acknowledged in the SF-19, the wording of its items prove to be appropriate for both sexes. This study therefore suggests that rewording the SF-19 items whenever needed seems to sufficiently compensate for its possible cultural inadequacy.
The approach described by Bolton and Tang has already been used in other cultures and settings (Ertl et al., 2010; Murray et al., 2006), and more such studies might follow. Therefore, our findings urge replication of our study. Similar outcomes would have implications for future use of the approach in question. In any context it should first be decided if indeed, depending on the method, one wants to assess functioning by inquiring three kinds of tasks in particular, notably: tasks that men/women must do regularly to care for themselves, for their family, and for their community, respectively. If another conceptualization of social functioning would be preferred, the approach does not apply. If this one is actually decided on, one might be better off using a standard instrument like the SF-19. If, however, the template in question is used, it seems that validity of the resulting instrument needs to be tested in every new context, criterion validity included.
Funding
This study was partly supported by a grant to Femke Verduin from the Health Research Development Counsel, Department Prevention Program (ZonMW), OOG-Geestkracht (ZonMW: 60-60105-98-117). We also thank Cordaid for funding the research and Prins Bernhard Cultuurfonds for their grant to F. Verduin.
Footnotes
Acknowledgements
We thank Paul Bolton for his generous permission to use the Rwandan functional assessment template. We thank Prof. J. M. Richters, of the Division of Culture, Health, and Illness of the Leiden University Medical Center (LUMC) for her contributions to the qualitative research and C. Dekker, who trained and supervised all the sociotherapy group leaders. We thank Ngendahayo Emmanuel, Sarabwe Emmanuel, and Hategekimana Balthazar for helping us organize and conduct the research. We also thank our loyal interviewers: Rukundo Ange, Mugisha-Bitanuzire John-Peter, Umuhoza Adelin, Uwizeye Beata, Ingabire Bernadette, Uwizeyimana Albertine, Mukanyiligira Marie Grace, and Muhongwanseko Scholastique.