Development and validation of the Vellore Occupational Therapy Evaluation Scale to assess functioning in people with mental illness

Abstract

Background:

Available occupational therapy assessment scales focus on specific areas of functioning. There is a need for comprehensive evaluation of diverse aspects of functioning in people with mental illness.

Aim:

To develop a comprehensive assessment scale to evaluate diverse aspects of functioning among people with mental illness and to assess its validity and reliability.

Methods:

Available instruments, which evaluate diverse aspects of functioning in people with mental illness, were retrieved. Relevant items, which evaluate specific functions, were selected by a committee of mental health experts and combined to form a comprehensive instrument. Face and content validity and feasibility were assessed and the new instrument was piloted among 60 patients with mental illness. The final version of the instrument was employed in 151 consecutive clients, between 18 and 60 years of age, who were also assessed using Global Assessment of Functioning (GAF), Occupational Therapy Task Observation Scale (OTTOS), Social Functioning Questionnaire (SFQ), Rosenberg Self Esteem Scale (RSES) and Pai and Kapur Family Burden Interview Schedule (FBIS) by two therapists. The inter-rater reliability and test–retest reliability of the new instrument (Vellore Occupational Therapy Evaluation Scale (VOTES)) were also evaluated.

Results:

The new scale had good internal consistency (Cronbach’s alpha = .817), inter-rater reliability .928 (.877–.958) and test–retest reliability .928 (.868–.961). The correlation between the general behaviour domain (Pearson’s Correlation Coefficient [PCC] = −.763, p = .000), task behaviour (PCC = −.829, p = .000), social skills (PCC = −.351, p = .000), intrapersonal skills (PCC = −.208, p = .010), instrumental activities of daily living (IADL) (PCC = −.329, p = .038) and leisure activities (PCC = −.433, p = .005) scores of VOTES with the corresponding domains in the scales used for comparison was statistically significant. The correlation between the total score of VOTES and the total scores of OTTOS, SFQ and RSES was also statistically significant suggesting convergent validity. The correlation between the total score of VOTES with the total score of FBI is not statistically significant, implying good divergent validity.

Conclusion:

VOTES seems to be a promising tool to assess overall functioning of people with mental illness.

Keywords

Occupational therapy assessment functioning mental illness

Introduction

Socio-occupational dysfunction is a hallmark indication of most mental illnesses (Saleem, Olié, & Loo, 2002). Psychosocial disabilities depend on multiple factors including social, physical and political environments and psychologic-personal determinants (Cieza et al., 2015; Sabariego et al., 2015). Personal determinants include the presence of comorbidity, health status, stressful life events, personality profile, adaptation, self-esteem, self-worth, social supports, built environment, weather and health problems in the family. The heterogeneity in symptomatology, treatment response, clinical course and outcome among mental illnesses in general and within diagnostic heads in particular necessitates an individually tailored approach to assessment and intervention.

Despite the availability of instruments to assess functionality, there are limitations because of their narrow focus like those specifically related to communication skills (Assessment of Communication and Interaction Skills; Forsyth, Lai, & Kielhofner, 1999), daily living skills (Milwaukee Evaluation of Daily Living Skills; Leonardelli, 1986), interests (Interest Checklist; Rogers, Weinstein, & Figone, 1978) or social roles (Role Checklist; Oakley, Kielhofner, Barris, & Reichler, 1986) and divergent frames of reference. The use of multiple tools is difficult in busy clinical settings. There is also the practical difficulty in taking a single conceptual framework-based assessment to evaluate overall functioning (Acharya & D’Souza, 2012). On the other hand, an integrative approach assumes that a single instrument, which is comprehensive in its evaluation of functioning and is based on more than one framework, can be used to assess patient dysfunction. It simultaneously employs multiple conceptual frames to evaluate people with mental illness, distress and disability (Hemphill, 2008).

This study attempted to develop a comprehensive scale based on several theories to simultaneously assess multiple areas of functioning.

Method

The details of the methodology are described below.

Study setting

The Department of Psychiatry, Christian Medical College, Vellore, India, is a 122-bed tertiary referral centre treating adults and children with mental and behavioural disorders. The centre cares for patients with a variety of mental illnesses including organic mental disorders, schizophrenia, affective illness, substance dependence, anxiety, intellectual disability and adjustment disorders. Psychiatrists, clinical psychologists, psychiatric social workers, psychiatric nurses and occupational therapists form the treatment team and employ a multidisciplinary approach in the care of patients with mental illness. Pharmacological treatments, electroconvulsive therapy and different psychological therapies (e.g. cognitive, behavioural, psychodynamic and supportive therapy) are used in an eclectic approach. Patients are admitted for an average of 4–6 weeks. Family members are required to stay with patients during hospitalisation. Patients spend around 5 hours a day in various activities like activity-oriented therapy, group therapy, recreational therapy and exercise therapy in the occupational therapy unit. Weekly goals are set by the occupational therapists to improve various domains of social and occupational functioning.

Instrument development

The instrument was mainly developed to aid in weekly assessment of patient progress. It was tested among inpatients attending the Adult Occupational Therapy Unit of the Department of Psychiatry, Christian Medical College, Vellore.

Item collection

The following scales were examined: (1) Executive Functioning Performance Test (EFPT; Baum et al., 2008), (2) Routine Task Inventory (RTI; Heimann, Allen, & Yerxa, 1989), (3) Assessment of Communication and Interaction Skills (ACIS; Forsyth et al., 1999), (4) Assessment of Occupational Functioning (AOF; Watts, Kielhofner, Bauer, Gregory, & Valentine, 1986), (5) Canadian Occupational Performance Measure (COPM; Law et al., 1990), (6) Interest Checklist (Rogers et al., 1978), (7) Role Checklist (Oakley et al., 1986), (8) Milwaukee Evaluation of Daily Living Skills (MEDLS; Leonardelli, 1986), (9) Social Functioning Scale (SFS; Birchwood, Smith, Cochrane, Wetton, & Copestake, 1990), (10) Lawton Instrumental Activities of Living Scale (Lawton & Brody, 1969), (11) Bay Area Functional Performance Evaluation (BAFPE; Houston, Williams, Bloomer, & Mann, 1989) and (12) Comprehensive Occupational Therapy Evaluation Scale (COTES; Brayman, Kirby, Meisenheimer, & Short, 1976). Most of these scales were developed to assess specific behaviours and particular diagnoses.

Each scale and their items were examined for relevance, adaptability and clarity by a multidisciplinary group of mental health professionals. Measures considered useful were incorporated in the new scale. Occupational therapists, clinical psychologists, social workers and psychiatrists were part of the panel of experts.

Item categorisation

The items were then broadly categorised into eight domains, namely, general behaviour, task behaviour, interpersonal behaviour, group skills, intrapersonal behaviour, roles and routines, activities of daily living (ADL) and leisure activities. The domain of ‘general behaviour’ and ‘intrapersonal behaviour’ was deemed necessary to ascertain the effect of psychiatric symptoms on functioning. The domain of ‘task behaviour’ was included as therapeutic activities are the mainstay of occupational therapy intervention. Social functioning was also felt to merit a distinct domain. The need to assess ‘interpersonal skills’ in a dyadic level independently of ‘group behaviour’ as patient behaviour was seen to change according to the context was also considered important. The domains of ‘roles and routines’, ‘activities of daily living’ and ‘leisure activities’ were included to assess occupational functioning. These domains were felt to encompass all relevant aspects of social and occupational functioning.

Scoring

Five-point scale was formulated for assessment with 0 indicating no dysfunction and 4 suggesting maximal dysfunction. Precise words without jargon were used in the scoring key so that ambiguity and vagueness could be avoided. Items formulated were specific and with a single idea to prevent confusion (Appendix 1).

Pilot study

The scale was piloted in a group of 60 consecutive inpatients attending the occupational therapy programme, aged between 18 and 60 years of age who gave written informed consent. Clients with a clinical diagnosis of moderate to profound intellectual disability, organic disorders and acute psychosis were excluded from the study. Vellore Occupational Therapy Evaluation Scale (VOTES) was scored by two occupational therapists concurrently to assess inter-rater reliability. It was also scored again after 3 weeks to assess test–retest reliability.

The instrument was found to have good internal consistency (Cronbach’s alpha = .973). The inter-rater reliability (intra-class correlation coefficient = .99) and test–retest reliability (intra-class correlation coefficient = .97) were found to be good.

The expert group reviewed the scale and the results of the pilot study following which alterations were made to the scale. Face and content validity of the scale, its feasibility for routine use and its coverage of different facets for functions were specifically examined. A few items were removed and others rearranged into different domains. The domains ‘roles and routines’, ‘activities of daily living’ and ‘leisure activities’ were clubbed into one single domain, called ‘activity configuration’. The final version of the scale has six domains with 35 items: five components for ‘general behaviour’, ’interpersonal skills’ and ‘group skills’; four items for ‘intrapersonal skills’ and ‘activity configuration’ and 12 for ‘task behaviour’ (Appendix 1).

Validation study

Study sample

Consecutive inpatients attending the inpatient occupational therapy programme in the Department of Psychiatry, Christian Medical College, aged between 18 and 60 years of age, who gave written informed consent were recruited for the study. Clients with a clinical diagnosis of moderate to profound intellectual disability, organic mental disorders and those with acute psychotic presentations were excluded from the study.

Instruments used

The following instruments in addition to VOTES were used for assessment:

Global Assessment of Functioning (GAF; Hall, 1995): GAF is a numeric scale (0–100) used by mental health clinicians and physicians to rate subjectively the social, occupational and psychological functioning of adults with psychiatric illness, with a score of 100 signifying maximum functioning. The scale is divided into 10 intervals, 1–10, 11–20 and so on up to 91–100.

Occupational Therapy Task Observation Scale (OTTOS; Margolis, Harrison, Robinson, & Jayaram, 1996): OTTOS evaluates patient performance during occupational therapy task groups. Each section of OTTOS is scored from 0 to 100, and each item within a section is scored from 0 (maximal dysfunction) to 10 (no evidence of dysfunction). It contains two parts, 10 items for evaluation of specific task functions and 5 items for rating general behaviour.

Social Functioning Questionnaire (SFQ; Tyrer, Nur, & Crawford, 2005): SFQ is an eight-item self-report scale (score range 0–24), a semi-structured interview which assesses perceived social function. It is divided into five sections, each containing eight items: Self-care Skills, Domestic Skills, Community Skills, Social Skills and Responsibility.

Rosenberg Self Esteem Scale (RSES; Mimura & Griffiths, 2007; Rosenberg, 1965): RSES is a self-esteem measure consisting of 10 items with total scores ranging from 10 to 40, with a higher score indicating higher level of self-esteem.

Pai and Kapur Family Burden Interview Schedule (FBIS; Pai & Kapur, 1981): FBIS is an instrument for assessing family burden through a semi-structured interview. It has 24 items arranged in six categories of burden rated on a 3-point scale with 2 being severe burden and 0 being no burden.

Procedure

Patients were assessed using all the instruments. They were also reassessed using VOTES after a gap of 3 weeks. The assessors were blind to the diagnosis of patients and rated them independently.

Statistical analyses

The mean and standard deviation were used to describe continuous variables, while frequency and percentages were obtained for categorical variables. The chi-squared test and the Student’s t-test were used to assess the statistical significance of categorical and continuous variables, respectively. The statistical significance of the association between two continuous variables was evaluated using the Pearson’s correlation coefficient. Inter-rater and test–retest reliabilities were evaluated using the intra-class correlation coefficient. Kappa statistic was used to assess agreement between groups. The internal consistency of the scale was assessed using the Cronbach’s alpha.

Ethical approval

The study was approved by the Institutional Review Board, Christian Medical College, Vellore (institutional review board (IRB) Min. No. 8502).

Results

In all, 151 subjects were recruited. The socio-demographic and clinical characteristics of the sample are shown in Table 1. The majority of the patients were male, young adults, single, with primary education, middle socio-economic status, from nuclear families and currently unemployed.

Table 1.

Characteristics of the study population (n = 151).

Characteristics	M (SD)	N (%)
Age (years)	33.51 (11.53)
Education (years)	4.05 (1.90)
Duration of illness (years)	7.92 (7.06)
Sex
Male		105 (69.5)
Female		46 (30.5)
Diagnosis
Schizophrenia		67 (44.7)
Bipolar		27 (18.0)
Substance dependence		18 (12.0)
Personality disorder		13 (8.7)
Obsessive compulsive disorder		11 (7.3)
Anxiety disorder		6 (4.0)
Dissociative/somatoform disorder		8 (5.3)
Marital status
Married		60 (39.7)
Single		79 (52.3)
Divorced/separated/widow/widower		11 (8.0)
Employment
Employed		51 (33.8)
Currently unemployed		51 (33.8)
Student		17 (11.3)
Homemaker		16 (10.6)
Never employed		16 (10.6)
Socio-economic status
Low		36 (23.8)
Middle		89 (58.9)
Upper		25 (16.6)
Living situation
Nuclear family		91 (60.3)
Joint family		46 (30.5)
Living alone		11 (7.3)
GAF – total score (100)	40.76 (13.39)
VOTES – total score (140)	54.67 (22.32)
VOTES – general behaviour (20)	4.75 (3.013)
VOTES – task behaviour (48)	19.51 (9.39)
VOTES – interpersonal skills (20)	6.83 (3.92)
VOTES – group skills (20)	6.46 (3.27)
VOTES – intrapersonal skills (16)	10.13 (3.00)
VOTES – role performance (4)	2.38 (0.99)
VOTES – BADL (4)	0.76 (0.79)
VOTES – IADL (4)	2.10 (0.99)
VOTES – leisure skills (4)	1.75 (1.29)

SD: standard deviation; GAF: Global Assessment of Functioning; VOTES: Vellore Occupational Therapy Evaluation Scale; BADL: basic activities of daily living; IADL: instrumental activities of daily living.

Internal consistency

The internal consistency between all the domains of VOTES is found to be good with Cronbach’s alpha .817.

Validity

The validity data are shown in Table 2.

Table 2.

Validity: correlation between various domains of VOTES with corresponding domains of OTTOS, SFQ, RSES and FBIS.

Correlation	Pearson’s correlation coefficient (p-value)^a
Comparison for concurrent validity
VOTES – general behaviour/OTTOS – general behaviour	−0.763 (.000)
VOTES – task behaviour/OTTOS – task behaviour	−.829 (.000)
VOTES – social skills/SFQ – social skills	−.351 (.000)
VOTES – intrapersonal skills/RSES	−.208 (.010)
VOTES – role performance/SFQ – responsibility	−.173 (.287)
VOTES – basic activities of daily living/SFQ – self-care skills	−.153 (.347)
VOTES – instrumental activities of daily living/SFQ – community skills	−.329 (.038)
VOTES – leisure/SFQ – leisure	−.433 (.005)
VOTES – total score/total of OTTOS + SFQ + RSES	−.905 (.000)
Divergent validity
VOTES – total score/FBIS – total score	.275 (.085)

VOTES: Vellore Occupational Therapy Evaluation Scale; OTTOS: Occupational Therapy Task Observation Scale; SFQ: Social Functioning Questionnaire; RSES: Rosenberg Self Esteem Scale; FBIS: Family Burden Interview Schedule.

Negative correlation on PCC as higher score in VOTES implies greater dysfunction, while lower scores in the other scales suggest increased impairment.

Criterion validity

The GAF was used as the gold standard. The mean GAF score among the study population was 40.76 (median = 35), which is interpreted as serious impairment in social, occupational or school functioning. The mean of VOTES score was 54.56 (of 140) with a median of 51. VOTES when compared with GAF (using their respective median values to divide into two categories as the former was a continuous scale while the latter employs intervals of 10 to categorise responses) had a kappa agreement of .682 suggesting good criterion validity.

Convergent validity

The correlation between the general behaviour domain, task behaviour, social skills behaviour, intrapersonal skills, instrumental ADL and leisure activities scores of VOTES with the corresponding domains in the scales used for comparison was statistically significant (Table 2). All these values show good convergent value between domains of the VOTES as compared with domains of other scales measuring same construct.

However, the correlation between basic ADL and role identification and performance domains of VOTES and self-care and responsibility domains of SFQ, respectively, did not reach statistical significance. Nevertheless, the correlation between the total score of VOTES and the total scores of OTTOS, SFQ and RSES was statistically significant.

Divergent validity

The VOTES was correlated with the FBI, which measures the unrelated construct of family burden. The correlation between the total score of VOTES with the total score of FBI is not statistically significant, implying good divergent validity.

Reliability

The reliability data are shown in Table 3. The inter-rater and test–retest reliabilities for individual domains of the VOTES are found to be good.

Table 3.

Reliability data for all domains of VOTES.

Domain	Test–retest reliability – intra-class correlation (95% CI)	Inter-rater reliability – intra-class correlation (95% CI)
General behaviour	.824 (.692–.903)	.799 (.674–.879)
Task behaviour	.731 (.546–.848)	.860 (.768–.917)
Interpersonal skills	.931 (.874–.963)	.888 (.813–.934)
Intrapersonal skills	.782 (.624–.878)	.739 (.585–.841)
Group skills	.800 (.653–.889)	.897 (.827–.939)
Role performance	.666 (.450–.808)	.603 (.362–.768)
Basic ADL	.835 (.710–.909)	.852 (.737–.919)
Instrumental ADL	.750 (.575–.860)	.638 (.410–.790)
Leisure skills	.764 (.596–.868)	.741 (.561–.854)
Total	.928 (.868–.961)	.928 (.877–.958)

CI: confidence interval; ADL: activities of daily living.

Discussion

People with mental illness have varying deficits in social and occupational functioning. Occupational therapy has the potential to help people with mental illness to remedy or compensate for these deficits. Instruments, which assess functioning, are needed to help therapists set appropriate goals for intervention. The currently available functional measures have been developed for other populations, which limit their utility in the Indian context. In India where there is dearth of mental health occupational therapists as well as intervention centres, there is dire need of developing and standardising measures specific to the population.

While many subsections of VOTES had high correlation with their corresponding scales, the correlation between basic ADL, role identification and performance domains of VOTES did not reach statistical significance. Cultural differences in perception of constructs like role identification, role performance and basic ADL could be responsible for limited correlation.

Preliminary data indicate that VOTES offers a reliable and valid method for rating the functioning of clients with mental illness. In addition, use of VOTES will improve the communication between occupational therapists and other health care providers. It will also facilitate the education of occupational therapy students and meet the requirements for documentation. The results of the study will need to be replicated in larger and more diverse samples.

The focus on and the evaluation of psychosocial functioning and disability can be used to support the implementation of cross-cutting interventions, social actions and policy tools to lower psychosocial disability experienced by persons with mental disease, illness and distress (Sabariego et al., 2015).

Footnotes

Appendix 1 Conflict of interest

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article:

This study was funded by a grant from the Christian Medical College, Vellore (IRB Min. No. 8502).

References

Acharya

D’Souza

D. D.

(2012). Problem identification grid: Assessment tool for acute mental health settings. Indian Journal of Occupational Therapy, 44, 11–14.

Baum

C. M.

Connor

L. T.

Morrison

Hahn

Dromerick

A. W.

Edwards

D. F.

(2008). Reliability, validity, and clinical utility of the Executive Function Performance Test: A measure of executive function in a sample of people with stroke. American Journal of Occupational Therapy, 62, 446–455.

Birchwood

Smith

Cochrane

Wetton

Copestake

(1990). The Social Functioning Scale: The development and validation of a new scale of social adjustment for use in family intervention programmes with schizophrenic patients. British Journal of Psychiatry, 157, 853–859.

Brayman

S. J.

Kirby

T. F.

Meisenheimer

A. M.

Short

M. J.

(1976).The Comprehensive Occupational Evaluation Scale. American Journal of Occupational Therapy, 30, 94–100.

Cieza

Anczewska

Ayuso-Mateos

J. L.

Baker

Bickenbach

Chatterji

… Pitkänen

(2015). Understanding the impact of brain disorders: Towards a ‘horizontal epidemiology’ of psychosocial difficulties and their determinants. PLoS ONE, 10, e0136271.

Forsyth

Lai

Kielhofner

(1999). The Assessment of Communication and Interaction Skills (ACIS): Measurement of psychometric properties. British Journal of Occupational Therapy, 62, 69–74.

Hall

R. C.

(1995).Global assessment of functioning, a modified scale. Psychosomatics, 36, 267–269.

Heimann

N. E.

Allen

C. K.

Yerxa

E. J.

(1989). The Routine Task Inventory: A tool for describing the function behavior of the cognitively disabled. Occupational Therapy Practice, 1, 67–74.

Hemphill

P. B.

(2008). Assessments in occupational therapy mental health – An integrative approach (2nd ed.). Thorofare, NJ: SLACK Incorporated.

10.

Houston

Williams

S. L.

Bloomer

Mann

W. C.

(1989). The Bay Area Functional Performance Evaluation: Development and standardization. American Journal of Occupational Therapy, 43, 170–183.

11.

Law

Baptiste

McColl

Opzoomer

Polatajko

Pollock

(1990).The Canadian Occupational Performance Measure: An outcome measure for occupational therapy. Canadian Journal of Occupational Therapy, 57, 82–87.

12.

Lawton

M. P.

Brody

E. M.

(1969). Assessment of older people: Self-maintaining and instrumental activities of daily living. Gerontologist, 9, 179–186.

13.

Leonardelli

C. A.

(1986).The process of developing a quantifiable evaluation of daily living skills in psychiatry. Occupational Therapy in Mental Health, 6, 17–26.

14.

Margolis

R. L.

Harrison

S. A.

Robinson

H. J.

Jayaram

(1996). Occupational Therapy Task Observation Scale (OTTOS): A rapid method for rating task group function of psychiatric patients. American Journal of Occupational Therapy, 50, 380–385.

15.

Mimura

Griffiths

(2007). A Japanese version of the Rosenberg Self-Esteem Scale: Translation and equivalence assessment. Journal of Psychosomatics Research, 62, 589–594.

16.

Oakley

Kielhofner

Barris

Reichler

R. K.

(1986). The Role Checklist: Development and empirical assessment of reliability. OTJR: Occupation, Participation and Health, 6, 157–170.

17.

Pai

Kapur

R. L.

(1981). The burden on the family of a psychiatric patient: Development of an interview schedule. British Journal of Psychiatry, 138, 332–334.

18.

Rogers

J. C.

Weinstein

J. M.

Figone

J. J.

(1978). The Interest Check List: An empirical assessment. American Journal of Occupational Therapy, 32, 628–630.

19.

Rosenberg

(1965). Society and the adolescent self-image. Princeton, NJ: Princeton University Press.

20.

Sabariego

Coenen

Ballert

Cabello

Leonardi

Anczewska

… Cieza

(2015). Determinants of psychosocial difficulties experienced by persons with brain disorders: Towards a ‘horizontal epidemiology’ approach. PLoS ONE, 10, e0141322. doi:10.1371/journal.pone.0141322

21.

Saleem

Olié

J. P.

Loo

(2002). Social functioning and quality of life in the schizophrenic patient: Advantages of amisulpride. International Clinical Psychopharmacology, 17, 1–8.

22.

Tyrer

Nur

Crawford

(2005). The Social Functioning Questionnaire: A rapid and robust measure of perceived functioning. International Journal of Social Psychiatry, 51, 265–275.

23.

Watts

J. H.

Kielhofner

Bauer

D. F.

Gregory

M. D.

Valentine

D. B.

(1986). The Assessment of Occupational Functioning: A screening tool for use in long-term care. American Journal of Occupational Therapy, 40, 231–240.