Abstract
BACKGROUND:
Assessing functioning and disability among individuals with mental and behavioral health disorders has historically relied on deriving accurate psychiatric diagnoses and assessing symptoms. However, growing empirical evidence suggests that this approach is inadequate to determine real world performance, particularly with respect to work.
OBJECTIVE:
We examined a performance-based approach to the assessment of work functioning and its relationship to mental and behavioral health status.
METHODS:
A cross-sectional study was conducted at two mental health programs. Trained employment providers conducted performance-based assessments of work function and ratings of mental and behavioral health while study participants self-reported their mental/behavioral health functioning. We hypothesized that participant and provider ratings of mental/behavioral health would be moderately correlated with performance-based assessments of work function.
RESULTS:
We found no significant correlation between participants’ self-report of their mental and behavioral health and performance-based assessments of work. Employment providers’ ratings of participants’ mental/behavioral health were moderately correlated with performance-based measures of work. Finally, we found low concordance between employment providers and study participants’ with respect to ratings of their mental/behavioral health.
CONCLUSIONS:
Contrary to our hypotheses, ratings of mental/behavioral health were only moderately correlated with performance-based measures of work. Results confirm earlier research suggesting that it is difficult to predict work performance from participants’ self-reports of their mental/behavioral health alone. Performance-based assessments of work capacity as well as ratings of mental and behavioral health may both be needed for a more complete and complimentary picture of the ability of individuals with mental and behavioral health disorders to function in the work place.
Keywords
Introduction
Assessing functioning and disability among individuals with mental and behavioral health disorders is complex. Historically, the medical model, with a focus on measuring symptoms and deriving accurate psychiatric diagnoses, was considered sufficient to understand and quantify functioning and disablement [1]. However, over the past two decades, a growing body of research has sought to better understand the relationship between functioning and psychiatric symptomatology [2, 3]. Empirical evidence has emerged suggesting that symptoms and functioning in general, and work performance more specifically, cannot be wholely predicted by symptoms, diagnosis, co-morbid conditions, or cognitive impairment [4–11]. In a recent study of almost 500 individuals with psychiatric disabilities receiving employment services, Corbiére and colleagues corroborated recent research suggesting that clinical factors, such as psychiatric diagnoses and severity of symptoms, were not useful in predicting employment outcomes [6].
With this accumulating empirical evidence has come a growing emphasis on functional assessments of mental/behavioral health [12] and work performance [13–15] with a concomitant focus on “real-world” functioning, as opposed to psychiatric symptoms and diagnosis alone [16]. In a recent review of performance-based measures, Harvey and colleagues [13] concluded that direct assessment of functional capacity has substantial advantages over other traditional measures of psychiatric status and provides a more valid estimate of functional disability. Leifker, [16] along with an expert panel, conducted a review of current performance measures across social, residential, and vocational domains (VALERO; “Validating Measures of Real-World Outcome”). From a total of 59 mental health assessments, recommendations of functional measures included those examining social and interpersonal behavior, self-care, life skills, and quality of life. No assessments focused on work function. Two additional performance-based assessments have been the object of study recently [14, 17] and both centered on self-care and activities of daily living among individuals with psychiatric disabilities. In addition, many of the studies that have been conducted on performance-based measures have been limited to the subpopulation of individuals with a diagnosis of schizophrenia alone. Together, the extant literature suggests that more research is needed to examine performance-based measures of work capacity and how they correlate with or diverge from measures of psychiatric symptoms, diagnoses, and status.
In this study, we sought to obtain empirical data about the relationship between a short form of the Work Disability Functional Assessment Battery, designed as a self-report assessment of behavioral and mental health (BH-FAB) [12] and a performance-based assessment of work capacity conducted by trained employment providers. We also wanted to examine the concordance between employment providers ratings of mental and behavioral health functioning and individuals’ concurrent self-reported ratings of the same. We hypothesized that: 1) employment provider ratings and participant self-report ratings of mental and behavioral health would each be moderately correlated with performance-based assessments of work capacity, 2) participants’ self-report of their mental and behavioral health functioning would be correlated with concurrent ratings by their employment providers.
Methods
We conducted a cross-sectional study to address study hypotheses. We recruited employment providers and clients they served to complete ratings of mental and behavioral health and to participate in structured, in vivo work performance assessments. All research procedures were reviewed and approved by two governing Institutional Review Boards (Boston University and the state Department of Mental Health). Data were collected from 2013 to 2016.
Sites
Research activities were conducted at two mental health programs in Massachusetts. The programs were chosen because they had an active employment services component and employment providers and were willing to participate in a fairly burdensome research study. Employment providers within these programs were recruited for participation by describing the research activities, the training they would receive as part of that research, and the observational and data collection portion of the study. Participation by employment providers was not mandated by the agency, but was voluntary. After recruitment, screening and consenting, employment providers were trained in the performance-based work assessment protocol over several weeks. Clients they served were then recruited for participation in the study. Assessments of work performance were carried out either at a “clubhouse program” for individuals with mental or behavioral health conditions or in community-based employment settings where participating individuals were working and receiving support from their employment providers. A trained research liaison at each site carried out informed consent procedures and coordinated data collection with senior research staff.
Participants
Employment providers
Six employment providers were recruited and trained to conduct observations of work performance; five were White American and one was African American; one was male and the remainder female. Most employment providers were not licensed clinical professionals. Overall, they had worked between 1 and 11 years as employment providers.
Study participants
Demographics of study sample
Demographics of study sample
The following instruments were used for data collection:
Demographic background
Basic demographics were captured using an instrument for that purpose and included age, gender, race/ethnicity, marital status, income, current work status, work history, educational attainment, Social Security Disability status, and living situation of study participants. With authorization, we obtained participants’ DSM-IV mental or behavioral health diagnoses from their mental health program.
Mental health function
We used a short form of the Behavioral Health domains of the Work Disability Functional Assessment Battery (WD-FAB) [12] to assess mental and behavioral health. The Behavioral Health subscales of the FAB (BH-FAB) assess critical dimensions of mental and behavioral health and were developed using item response theory (IRT) methodology. Item content was conceptually grounded in the World Health Organization International Classification of Functioning [18] and using a rigorous process that included extensive literature review, cognitive testing and focus group input from both individuals with work disabilities as well as work disability experts.
Sample items from the Vocational Situational Assessment, the Behavioral Health Functional Assessment Battery, and the Behavior and Symptom Identification Scale-24®
Sample items from the Vocational Situational Assessment, the Behavioral Health Functional Assessment Battery, and the Behavior and Symptom Identification Scale-24®
Note: The BH FAB (Short Form) has a mean of 50 and a SD of 10 in the normative data; the VSA is scored on a 1–4 scale with higher scores indicating better functioning.
Higher scores on each scale indicate higher functioning. Internal consistency scores for the BH-FAB in this sample ranged from good to excellent with coefficients alpha for the participants self-report ratings ranging from 0.70 to 0.92 and from 0.70 to 0.95 for the employment provider ratings. There were predictable differences by diagnostic category which provided evidence for both the scale’s reliability and discriminant validity.
IIB [19] The Vocational Situational Assessment Scale (VSA), a 35-item scale developed to measure work performance among individuals with mental and behavioral health disorders, was used to guide in vivo observations and ratings by trained employment providers. The VSA is comprised of two subscales, Work Skills (21 items) and Interpersonal Skills for the Workplace (15 items), each measured on a 4-point rating scale (namely, 1 = cause to be fired to 4 = above acceptable work performance) and yielding a subscale score. Each scale point is anchored with descriptors which were carefully developed and evaluated based on the requirements of a minimum wage job (samples of the Work Skills subscale and the Interpersonal Skills for the Workplace subscale appear in Table 2).
The VSA has demonstrated reliability, concurrent and predictive validity in a small study of individuals with psychiatric disabilities [19]. Internal consistency reliability estimates for this sample and study of the VSA instrument was 0.97 overall with 0.95 for the Work Skills Subscale and 0.94 for the Interpersonal Skills Subscale.
Recruitment and training of employment providers
Research staff oriented and consented employment providers (informed consent of the employment providers was required by the governing IRBs). Remuneration was provided for their time and effort. Employment providers were trained extensively on procedures for observing work performance and on the ratings of the Vocational Situational Assessment Scale. They were provided an overview of performance-based measurement and were oriented to each item of the VSA using hypothetical scenarios. During the training, we capitalized on the employment providers’ considerable knowledge and experience in observing and assessing both adequate and problematic work performance, behaviors and skills; training helped them to operationalize their knowledge and skills using the scale items and ratings.
Secondly, we established inter-rater reliability and validity of the VSA ratings in two steps. First, we recruited and enrolled “practice” participants whom employment providers were able to observe and rate in pairs. Towards the end of their training, employment providers were paired with their supervisors to conduct an additional assessment. We took this step with the assumption that the supervisor was the “gold standard” for validity purposes and also to perform another check on inter-rater reliability. Employment providers were instructed to conduct unobtrusive observations and to keep concurrent notes. At the end of each training session, research staff calculated the agreement between raters, and reviewed the ratings and observational notes in detail with employment providers and their supervisors. We detected the most common sources of errors or differences between each pair, and addressed those differences by discussing the ratings and reaching consensus. We continued with practice participants until satisfactory correlations between raters were achieved. These training and consensus building steps allowed employment providers to achieve a satisfactory level of skill in conducting performance-based assessments of work before proceeding with the study.
Data collection
Research staff screened potential study participants based on eligibility criteria; participants were consented and enrolled by research liaisons at each site. Upon enrollment, participants completed the demographic questionnaire and the self-report version of the BH-FAB. Within 1–2 days of the study participant’s data collection, employment providers completed the BH-FAB about their respective client. Within 1–2 days of completing the BH-FAB, the employment provider began the performance-based work assessment by observing the participant in a natural work setting for at least 1 hour per day for a 3–5 day period. Employment providers kept notes about their observations and rated the participants at the conclusion of the observation period.
Data analysis
After verifying the data and examining it descriptively, we conducted analyses to examine concordance between providers and participants on mental and behavioral health functioning [20, 21] and to examine ratings of mental and behavioral health and performance-based assessments of work using both intra-class and Pearson correlation coefficients. We constructed a “folded empirical cumulative distribution plot” (i.e., mountain plots) [22] in order to visually depict and examine the magnitude and direction of discordance between ratings conducted by providers and self-report ratings by participants of their mental and behavioral health. A mountain plot provides information about the distribution of the differences between the two rating approaches, in this case, provider ratings vs. self-report rating. If the two ratings are unbiased with respect to each other, the mountain plot will be centered over zero. Long tails in the plot reflect large differences between the raters.
All data analyses were conducted in SPSS 20.0.
Results
Means and standard deviations of measures
Means and standard deviations of measures
Note: The BH FAB (Short Form) has a mean of 50 and a SD of 10 in the normative data; the VSA is scored on a 1–4 scale with higher scores indicating better functioning. Missing data on the Interpersonal Skills subscale of the VSA was due in part to some behaviors not being observable in certain work situations (e.g., interactions with the public).
Correlation coefficients between subscales of the BH FAB and the total score of the Vocational Situational Assessment, Work Skills and Interpersonal Skills subscales
Note: N’s vary because not all items of the VSA could be observed (e.g., “converses with co-workers … ” may have been unobservable if the participant was working alone).
= significant at p < 0.05;
= significant at p < 0.01.
Intraclass correlation coefficients of employment specialist and client ratings on subscales of the Behavioral Health Functional Assessment Battery
The magnitude and direction of discordance between participants and providers on ratings of mental/behavioral health are depicted graphically in mountain plots displayed in Fig. 1. We note that the median values of score difference between subject self-report scores and provider scores are –6.97, –6.68, and –3.32 respectively for Behavioral Control, Mood and Emotions and Social Interactions subscales, suggesting that providers rated particiants as less impaired when compared with self-report ratings for the same subscales. On the Self-Efficacy subscale, the median value is 7.05, suggesting that providers rated subjects as more impaired when compared to self-reported ratings by participants. Next, we examined the difference between 75th and 25th percentiles (interquartile range (IQR)) for each subscale. The Self-Efficacy scale had largest interquartile range (16.1) compared with others (Mood and Emotions: 15.35; Social Interactions: 14.18, Behavioral Control: 13.46), suggesting that scores on the Self-Efficacy scale had more variability compared to other subscales.
Mountain plots of BH-FAB subscales comparing participant self -report to employment provider ratings. Differences with a negative skew indicate employment provider rated participant as functioning less well than did the participant (BC, SI, ME). The one difference with a positive skew was Self Efficacy, suggesting participants perceived and rated themselves more positively than providers. Note: BC = Behavioral Control; SI = Social Interactions; SE = Self Efficacy; ME = Mood and Emotions.
Contrary to our hypotheses, we found no significant correlations between participants’ self-report of their mental and behavioral health and performance-based assessments of work made by trained employment providers. This was the most striking finding and parallels a significant body of extant research. Such discrepancies have contributed to controversies in the mental health field about the capacity of individuals with mental and behavioral health disabilities to work and to conclusions by many clinicians that they cannot work even when clients assert that they can and wish to be employed. Such discrepancies in the perspectives of mental health clients and their providers may help to explain in part the differing views that have been reported in the literature about the role of work in the lives of individuals with mental and behavioral health disorders [23, 24].
We found moderate correlations between provider assessments of participants’ mental/behavioral health and their performance-based assessments of work. However, the variance explained in work performance by multiple subscales of a mental and behavioral health assessment was modest, in the range of 20–25%. These data suggest that work performance and mental and behavioral health status are relatively independent and that mental health functioning is not highly predictive of work performance in this population. The absence of a strong relationship between diagnosis, symptoms and work has been asserted and demonstrated in numerous studies examining the predictors of employment outcomes among individuals with mental and behavioral health disorders [6, 8–11].
Several factors could be responsible for the finding that there was not a high correlation between work function and ratings of mental/behavioral health. First is the possibility that not all mental and behavioral health symptoms by their nature interfere with an individual’s work performance. Having adequate employment and interpersonal skills in the workplace may supersede difficulties with mood and emotion, behavioral control, social interactions, or self-efficacy, particularly when difficulties in these areas are not severe. Secondly, it is possible that the observations of work performance conducted for this study did not accurately capture the totality of the participants’ work capacity.
In terms of the discordance between participants and providers in ratings of mental and behavioral health ratings, there may be other factors at play. First, the education and training of employment providers does not generally emphasize assessment of mental and behavioral health functioning, which may make concordance between employment providers ratings and participant self-report more difficult to achieve. However, this lack of concordance was also noted by Marfeo and colleagues [25] in a study examining concordance between mental health providers and Social Security claimant self-report. It is possible that the 7-day rating window imposed by some of the BH-FAB items was problematic for employment providers who did not necessarily have knowledge of difficulties in that abbreviated time span, particularly for low-incidence, but important behaviors, such as those represented in the Behavioral Control subscale (i.e., “In the past seven days, I threatened violence toward people or property”). Further, providers and participants may have had different perspectives about and interpretations of certain mental/behavior health items, as was noted by Marfeo and her colleagues [25].
Our findings mirror research suggesting that comparing ratings of psychiatric symptoms, functioning, goals, or problems when completed through self-report or using only the client’s perspective and compared to providers, often do not yield high agreement [25–30]. In addition, studies suggest that certain symptom and function ratings may be more suitable for accurate self-report than others, such as self-efficacy and depression vs. psychotic symptoms [31] or developmental history vs current work functioning, [32] although that research is not unequivocal [33] or robust.
Results of this study confirm earlier research suggesting that it is difficult to predict work performance from participants’ self-reports of their mental/behavioral health. Performance-based assessments of work capacity as well as ratings of mental/behavioral health may be needed for a complete and complimentary picture of the ability of individuals with mental and behavioral health disorders to function in the work place. We conclude that understanding the relation between work function and mental and behavioral health status deserves further study, and that our knowledge suggests that a person’s psychiatric diagnosis and symptomatology is likely to be only moderately predictive of their work performance.
This study had several methodological limitations. First, we used a relatively small sample of individuals with mental and behavioral health disorders, about 50% of whom had schizophrenia-spectrum disorders and who evidenced varying degrees of impairment. A larger sample may have provided additional information about the relationship between the symptoms, functioning and work performance. We were able to conduct this study only within two mental health programs in one state; a different geographical representation could have yielded different findings. A significant number of the study participants were engaged in simulated work rather than competitive employment. Given the relatively small sample size, we were unable to fully exploit subset analyses of these two types of work, but such differences could lead to differences in performance. Employment providers were trained to criterion to conduct performance-based assessments of work capacity, however, this measure itself may be imperfect in its ability to capture work capacity. Work performance ratings were also relatively high, suggesting possible confounds from a ceiling effect. Our performance-based assessment of work function does not take into account other factors that could affect employment providers’ perceptions and ratings of work performance such as the participants’ beneficiary status, environmental pressures, and labor market issues. Lastly, it appears that these two measures tap distinctly different aspects of functioning with less correlation than hypothesized.
Conclusions
Research suggesting that mental and behavioral health status and symptoms are not highly predictive of work performance in this population [5, 8] is borne out by this study. Taken together, results of this study suggest that observations and ratings of work performance conducted by a trained employment provider, when compared to self-report by participants of their mental and behavioral health are not highly correlated. The variance explained in work performance by the mental and behavioral health ratings was low, suggesting that capacity to work and work performance cannot be explained by mental health status alone. Each of these measures may provide unique and incremental information. We did confirm our hypothesis that performance-based observations of work correlated moderately with assessments of mental and behavioral health when both were conducted by employment providers. We conclude that multiple modes of observation and assessment of symptoms, function and performance are needed to obtain comprehensive and accurate assessments of “real world” functioning among individuals with mental and behavioral health disorders, particularly with respect to their ability to function in the workplace. These findings, coupled with other studies, have implications for the determination of social benefits and programs that rely on the establishment of work disability.
Conflict of interest
None to report.
Footnotes
1
In order to assess the extent of concordance between the Employment Provider and the participant about behavioral health/psychiatric functioning, a parallel version of the scale was created. Thus, an item that read “I am good at making new friends” for self-report by the participant was altered to read “The client is good at making new friends” for the provider.
Acknowledgments
This research was supported by Social Security Administration and National Institutes of Health (SSA-NIH) Interagency Agreements (NIH contract # HHSN269201100009I) and by the NIH intramural research program.