Transcultural Psychiatry Made Simple—Asynchronous Telepsychiatry as an Approach to Providing Culturally Relevant Care

Abstract

Objective: To examine the feasibility and diagnostic reliability of asynchronous telepsychiatry (ATP) consultations in Spanish and ATP consultation with Spanish-to-English translation. Subjects and Methods: Twenty-four interviews of Spanish-speaking patients were videorecorded by a bilingual clinician who also collected patient history data and gave the Structured Clinical Interview for DSM-IV Axis I Disorders (SCID-I) to each patient. The ATP data (video of the interview and patient history) were forwarded for psychiatric consultation and a diagnostic assessment by the investigators. The ATP data were then examined separately by two Spanish-speaking psychiatrists, before being translated into English and then re-examined by two English-speaking psychiatrists. Agreement between the expert diagnoses of the investigators and the diagnoses from the Spanish consultations, the Spanish-to-English translated consultations, and the SCID-I results was assessed using kappa statistics. Results: We found acceptable levels of agreement for major diagnostic groupings among the Spanish- and English-speaking psychiatrists. Kappa values for diagnostic agreement between the expert and the translated consultations, the original language consultations, and the SCID-I were at least 0.52 (percentage agreement, 79%) and higher. Conclusions: ATP consultations in Spanish, and those translated from Spanish to English, are feasible, and broad diagnostic reliability was achieved. The ATP process allows for rapid language translation. This approach could be useful across national boundaries and in numerous ethnic groups. Cross-language ATP may also offer significant benefits over the use of real-time interpreting services and has the potential to improve the quality of care by allowing for the addition of culturally relevant information.

Introduction

Individuals of minority groups are medically underserved in the United States, greatly lacking in mental healthcare services, and those with limited English proficiency (LEP) have the most limited access to healthcare and mental health services.¹ Lack of available translation services contributes to poor healthcare access and utilization for LEP patients.^1,2 It is well documented that access to psychiatric care is extremely limited in underserved communities, and because of language barriers the availability of psychiatric evaluation is even further limited for LEP patients.¹ Those who have LEP are “less likely to identify a need for mental health services, experience longer duration of untreated disorders, and use fewer healthcare services for mental disorders, particularly specialty mental healthcare.”³ The mental health community and public health officials have responded by attempting to increase patient access to medical interpreters and/or bilingual providers, and federal mandates now require that language services are available for LEP patients. However, cost and availability of interpreting services continues to limit access,⁴ and disparities in healthcare access are expanding between LEP patients and the English-proficient population. Many LEP patients are left to make due with available services, often struggling to communicate with physicians across language barriers or utilizing untrained bilingual staff or family members as interpreters.⁵

Traditionally there have been four methods of interaction when language incompatibility exists between doctors and patients. The first method involves a meeting between just the patient and the provider, where there may be many language problems on both sides. In this case, patients have reported the provider as being less friendly, less respectful, and less concerned.⁶ The second method involves the patient, the provider, and a family member acting as an interpreter for the patient. There are several issues with this second method. Family members and friends are not qualified interpreters and often do not understand medical terminology, so there is the potential for errors when interpreting. Moreover, patient confidentiality is breached with a family member interpreting, and patients may not fully disclose all pertinent medical information in the presence of a family member or friend.⁷ The third method is the interaction among the patient, provider, and another staff member, such as a bilingual nurse, acting as an “unofficial interpreter.” Bilingual nurses often take on the role of an interpreter, although they may not be trained as medical interpreters.⁸ Elderkin-Thompson et al.⁸ found four primary issues with the use of bilingual nurse interpreters: “(1) physicians resisted reconceptualizing the problem when contradictory information was mentioned; (2) nurses provided information congruent with clinical expectations but not congruent with patients' comments; (3) nurses slanted the interpretations, reflecting unfavorably on patients and undermining patients' credibility; and (4) patients explained the symptoms using a cultural metaphor that was not compatible with Western clinical nosology.” They concluded that interpretation by untrained nurse-interpreters during cross-language encounters led LEP patients to be misunderstood by their physicians. The fourth method is the interaction among the patient, provider, and a professional interpreter, all present at the appointment either in person or via telemedicine. This method has been found to improve access to care and mental health outcomes for LEP patients.^9,10 However, arranging professional interpreting services either in person or via telemedicine can be a challenging task, which may add to the difficulties of accessing care for LEP individuals. Many available interpreting services are not fully covered by insurers,⁴ adding yet another layer to the barriers that restrict access for LEP individuals. Moreover, despite federal regulations, many healthcare organizations are unable to meet an LEP patient's language service needs. One recent nationwide study found that only 13% of hospitals in the United States meet federal standards for providing language services, and 19% did to meet any of the federal language services standards.⁵ Over the several past decades, a technological revolution has taken place that has dramatically changed the way in which people communicate, seek information, and receive services. However, although advancements such as asynchronous telemedicine, electronic medical records, and mobile health applications are revolutionizing the field of healthcare in many areas, little attention has been paid to the use of technology to overcome language barriers between doctor and patient.

Asynchronous telemedicine has made it possible for a doctor in one location to get expert advice on a diagnosis and treatment plan from specialists around the world without the need for the doctor and the patient to be connected together in real time.^11,12 The inclusion of health information technology into the general medical field has allowed for the development and broad use of asynchronous medicine where patient information, such as pictures and written documents, is securely stored (e.g., in an electronic medical record) and later securely forwarded to a consulting specialist or party for review. This technology has been well received in many fields such as pathology, dermatology, and cardiology^13

–17 and has recently been used in psychiatry^18
–20 but has not been used with translation across languages in any discipline.

Asynchronous telemedicine has a unique advantage over real-time care in that patient data can be augmented en route by adding to the information, such as adding language interpretation, before sending it to a consulting specialist, thereby broadening the scope of providers who can evaluate it. With the asynchronous medicine process, a clinical interview recorded in a patient's native language could be translated into the language of the specialist who would be evaluating it, hence obviating the need for a real-time interpreter. Asynchronous telemedicine has the potential to overcome many of the barriers to care with the traditional forms of translation by eliminating the need for family, staff, or locally available professional interpreters to be available at the time of the original medical consultation.

Asynchronous telemedicine in the field of psychiatry or asynchronous telepsychiatry (ATP) is beginning to be evaluated as a potential service delivery system to increase access to specialty consultations for primary care physicians who serve patients with limited access to care. We have demonstrated the feasibility of ATP in 60 homogeneous-language consultations.¹⁸ Our next aim was to evaluate the use of ATP across languages, namely, with consultations occurring between Spanish-speaking patients with LEP and English-speaking providers. The primary goals were twofold. First, we intended to demonstrate that the use of ATP across languages was a feasible clinical process that could avoid the need for real-time interpreters. Second, we wished to establish preliminary reliability of ATP across languages. We intended to demonstrate that ATP can be feasible in cross-language settings and thus could greatly reduce the need for on-site translators and improve access to psychiatric consultation for non–English-speaking individuals.

Subjects and Methods

Participants

We received funding to conduct a feasibility and inter-rater reliability study of ATP, approved by the University of California, Davis Institutional Review Board. As part of a larger study, in 2008–2009, we conducted 24 ATP consultations in Spanish with patients older than 18 years of age, who were referred by their primary care provider (PCP) for a psychiatric consultation. Patients were referred from three Primary Care Centers in Tulare County, California. Advertisements for the study were posted in bulletin boards of the Primary Care Centers and other frequently visited places in the communities. The investigators traveled to all three Primary Care Centers, met referring PCPs, and discussed the project in detail.

Inclusion criteria involved the patient having a psychiatric problem that did not include cognitive impairment and warranted a non-urgent psychiatric evaluation, as identified by the PCP. Patients gave informed consent, following a complete description of the study, before admission to the study. All patients were informed that the video of their interview would be viewed by one or more psychiatrists and that a consultation opinion would be written and provided to their PCP. Patients were told that they would be able to see their consultation opinion by requesting it from their PCPs. All patients were given $100 gift vouchers for their time and were not charged for the consultation.

Application

A custom software application, PsychVACS, compliant with the requirements of the Health Insurance Portability and Accountability Act was designed specifically for this project in order to deliver ATP consultations.¹⁹ PsychVACS is a secure Web interface that allows the referring provider to log on and upload patient data and video where they are securely stored so that the consulting specialist can retrieve them at a later date and provide consultation feedback for the referring provider—an all-in-one secure ATP interface. PsychVACS is unique from several other “store-and-forward” applications on the market, both commercial and noncommercial, because patient video along with other patient data can be stored and transferred securely for consultation purposes.

Measures

We selected the widely used Structured Clinical Interview for DSM-IV Axis I Disorders (SCID-I),²¹ which is available in Spanish, to generate a control diagnosis for each patient to compare with the psychiatrists' ratings for diagnostic reliability only. The SCID-I data were not available to consulting psychiatrists for review. The SCID-I is a semistructured interview for DSM-IV Axis I diagnoses. This interview is widely considered to produce reliable and valid psychiatric diagnoses for clinical, research, and training purposes,²² with good to very good inter-rater reliability (kappa=0.61–0.83),²³ and has been used in over 1,000 research studies.²² The Spanish version of the Mini-International Neuropsychiatric Interview,²⁴ a short structured diagnostic interview, was used as part of the routine clinical interview undertaken with all patients, with the results being available to all assessing psychiatrists in both English and Spanish as part of the ATP dataset. For the purposes of this study, we developed a structured psychiatric history, which included a mental state assessment, illness, medical, and medication history and the Mini-International Neuropsychiatric Interview, which made up the “history” component of our ATP dataset.

Procedure

Twenty-four Spanish-speaking patients were interviewed by our Spanish-speaking clinician researcher. These interviews were videorecorded for later evaluation by psychiatrists (Spanish video). All original patient history documents were also written in Spanish in PsychVACS (Spanish history). All patients also undertook the SCID-I and the Mini-International Neuropsychiatric Interview in Spanish. Copies of the written history files were transcribed from Spanish into English by our research physician (translated history). Copies of the video files were sent for translation by the professional medical interpreting service at the University of California Davis Medical Center. The medical interpreters recorded an English language audio file for each of the interviews (translated audio).

Three groups of psychiatrists evaluated all 24 participants and offered diagnoses. (1) For expert diagnosis, two expert clinical telepsychiatrists were responsible for providing opinions and treatment recommendations to the referring PCPs within 2 weeks of the patient interviews for clinical purposes. They evaluated the “translated history” files and “Spanish video” files, and consultation reports were sent back to the PCP. For research purposes, to improve the process of diagnostic comparison, all original files and diagnoses were then reviewed by one expert telepsychiatrist to confirm a final “expert” diagnosis for each patient that was used for the reliability testing. (2) For original language consultations, two independent Spanish-speaking psychiatrists evaluated the “Spanish video” file and “Spanish history” file. (3) For Spanish-to-English translated consultations, two independent English-speaking psychiatrists evaluated the “translated history,” “Spanish video,” and “translated audio” files. All files could be viewed simultaneously with a multiple screen display set up, and the consultation note could be written at the same time. We found that consultation time across groups were comparable, taking about 20–40 min on average for psychiatrists to review the material and write the consultation note.

All psychiatrists recorded diagnoses and treatment recommendations in PsychVACS based on the video files and written information that they reviewed. Because of the small sample size and low prevalence of the individual DSM-IV diagnosis in this study, all diagnoses were categorized into four overarching diagnostic categories (Mood Disorders, Anxiety Disorders, Substance Abuse Disorders, and Psychotic Disorders). Following classification, the original language consultations and the translated consultations and the SCID results were each compared against the expert diagnoses for evaluation of diagnostic reliability.

Results

The sample consisted primarily of low-income or unemployed Spanish-speaking Hispanic individuals living in Tulare County, California. Middle-age females represented the largest group in the sample. The majority of the participants had not completed high school and were unemployed at the time of the study. Demographic data for the sample are presented in Table 1.

Table 1.

Demographic Characteristics for the 24 Participants

	MEAN (SD) OR FREQUENCY (%)
Age (years)	47.9±13.4
Gender
Female	15 (63%)
Male	9 (38%)
Education^a
Less than high school	10 (59%)
High school	5 (29%)
College	2 (12%)
Currently employed^b
Yes	7 (30%)
No	16 (70%)
Religion
Yes	20 (83%)
No	4 (17%)
Marital status^b
Married/relationship	16 (70%)
Not in a relationship^c	7 (30%)
Children (n)	3.8±3.1

Frequency missing=7.

Frequency missing=1.

Not in a relationship includes single, divorced, separated, and widowed.

SD, standard deviation.

Table 2 summarizes the diagnoses made by the investigator team. Most of the participants (71%) received only one diagnosis, and 20% received two diagnoses. One person received three diagnoses (4%), and one had no DSM-IV diagnosis (4%). The most prevalent diagnosis was mood disorder, with 71% of the participants having either a primary or a secondary diagnosis of a mood disorder.

Table 2.

Summary of Expert Diagnoses for the 24 Participants

	FREQUENCY (%)
Number of diagnoses
0	1 (4%)
1	17 (71%)
2	5 (21%)
3	1 (4%)
Type of diagnoses
Mood disorders	13 (54%)
Anxiety disorders	4 (17%)
Substance abuse/dependence disorders	1 (4%)
Mood disorders and anxiety disorders	3 (13%)
Mood disorders and substance disorders	1 (4%)
Psychotic disorders/schizophrenia	1 (4%)
No diagnosis	1 (4%)

The level of agreement on the presence or absence of each of the four major diagnoses between the expert psychiatrist diagnosis and SCID-I, the expert psychiatrist and the psychiatrists evaluating the Spanish data, and the expert psychiatrist and the psychiatrists evaluating the English-translated data was assessed using the kappa statistic, which accounts for agreement that may occur by chance. The ranges of kappa statistics showing agreement have been reported as poor (<0.2), fair (0.2–0.4), moderate (0.4–0.6), good (0.6–0.8), and very good (0.8–1.0).²⁵ Because the prevalence of the diagnoses (even after grouping) was still low and the kappa statistic is sensitive to the number of cases, we also reported the percentage of agreement.

Table 3 presents the results of the diagnostic agreement analyses. Kappa values for agreement with the expert psychiatrist were 0.52 or above (moderate–very good) across all raters and categories (percentage agreement 79% and higher). The highest agreements with the expert psychiatrist were in the substance abuse 0.63–1.00 (92–100%) and psychotic disorder 1.00 (100%) categories. Moderate to very good agreement was found in the anxiety 0.55–0.90 (79–96%) and mood disorder 0.52–0.78 (83–92%) categories.

Table 3.

Summary of Diagnosis Agreement (Kappa [Percentage]) with the Expert Psychiatrist Diagnoses for the Two English-Speaking Doctors, Two Spanish-Speaking Doctors, and the Structured Clinical Interview

		KAPPA (PERCENTAGE AGREEMENT)
		ENGLISH DOCTORS		SPANISH DOCTORS
DIAGNOSIS	FREQUENCY	E₁	E₂	S₁	S₂	SCID-I
Mood Disorders^a	17	0.68 (88%)	0.65 (88%)	0.52 (83%)	0.52 (83%)	0.78 (92%)
Anxiety Disorders^b	8	0.55 (79%)	0.90 (96%)	0.55 (79%)	0.73 (88%)	0.65 (83%)
Substance Abuse Disorders^c	4	0.78 (96%)	1.00 (100%)	0.63 (92%)	0.63 (92%)	0.65 (96%)
Psychotic Disorders	1	1.00 (100%)	1.00 (100%)	1.00 (100%)	1.00 (100%)	1.00 (100%)

Included Major Mood Disorder Due to General Medical Condition, Bipolar Disorders, Major Depressive Disorders, Dysthymic Disorder, Adjustment Disorder with Depressed Mood, Adjustment Disorder with Mixed Anxiety and Depressed Mood, Depressive Disorder Not Otherwise Specified.

Included Anxiety Disorder Not Otherwise Specified, Panic Disorder Without Agoraphobia, Generalized Anxiety Disorder, Panic Disorder with Agoraphobia, Social Phobia, Somatoform Disorder Not Otherwise Specified, Adjustment Disorder with Anxiety, and Posttraumatic Stress Disorder.

Included Alcohol Dependence, Polysubstance Dependence, Phencyclidine Dependence/Other (or Unknown) Substance Dependence, Alcohol Abuse, Cocaine Abuse, Amphetamine Abuse.

E₁ and E₂, English-speaking doctors; S₁ and S₂, Spanish-speaking doctors; SCID-I, Structured Clinical Interview for DSM-IV Axis I Disorders.

Discussion

We have demonstrated that ATP across languages is a feasible clinical process that can avoid the need for real-time interpreters. In this small pilot study we have also demonstrated the preliminary reliability of ATP diagnoses across languages and then compared it with a validated research diagnostic assessment tool. Further studies are needed to replicate these findings in larger samples with greater variation among diagnostic groupings.

We found evidence in this preliminary study that reliability is maintained following translation of the video interviews and brief online psychiatric histories into English. The diagnostic agreement percentage found in this sample is comparable to other studies evaluating the diagnostic reliability of telepsychiatry.²⁶ For the majority of the cases, the ranges of kappa statistics showing agreement in our study fell in the “good” to “very good” range.²⁵ Substance abuse and psychotic disorders showed higher diagnostic reliability than disorders with overlapping diagnostic criteria such as mood and anxiety disorders, although the numbers of our patients in these categories were small. Furthermore, demonstration of diagnostic reliability with patients and psychiatrists speaking the same language (Spanish–Spanish/English–English) indicates that this is a reliable consultation process to use in a same-language group as well as cross-language groups.

Additional studies are needed to replicate these results in larger samples and in non-research clinical settings, as several methodological limitations exist with this first cross-language ATP study. The presence of co-morbid or co-occurring disorders and categories with considerable diagnostic overlap (anxiety and depression) may also have limited diagnostic ability in some categories. Data collected for this study were collected by a researcher and not by the patient's PCP, as would occur in real-world practice, and replication in real-world settings is necessary. Audio files with interpreter voice translations of the Spanish interview were provided to supplement the Spanish video for the “translated audios.” In effect, the psychiatrists conducting the assessments in English reported that the affective tone of the interview was lost and that they tended to rely on the written history and listen to the original audio to observe the emotional inflection of the conversation. We believe that future applications would be more effective if subtitles were used to retain mood information while providing translation. Voice-to-text applications are available that could be used to generate the translations within the application and will be tested to see if they can improve the next iteration of the platform. The present study involved a limited sample size and diagnostic co-morbidity. Replication is necessary with larger and more diverse diagnostic groups, as well as with other languages. We also only attempted to do a language translation in this first study, and future work should involve cultural experts to more fully take account of cross-cultural social issues such as the acceptability of this approach among patients from various cultural groups and cultural-specific information that may be lost or misconstrued in the translation process. Ideally this approach would support the efficient and cost-effective addition of language translation as well and culturally relevant information provided by a professional with cultural expertise relevant to the patient. This application of ATP has the potential not only to expand access to basic mental healthcare to individuals with LEP, but also to improve the quality of care by making it more culturally relevant and inclusive. ATP has the potential to increase the accuracy of psychiatric consultation and the relevance to the patient's culture and to improve the relationship between patient and primary provider by increasing the PCP's cultural understanding of the patient and the influence of the patient's culture on his or her mental health needs.

Despite methodological issues, these data provide support for the feasibility of ATP and preliminary reliability evidence for ATP as an effective way of conducting distant psychiatric consultations across languages. Once the results of this first-ever cross-language study are validated, ATP could be used in many areas such as the prison system, the military, and other difficult-to-access groups where there are many patients for whom English is not a first language or for whom no alternative psychiatric access exists. Future iterations of the ATP platform could add other data enhancements such as facial and language analytic tools.²⁷ In the present study, the voice data were collected and translated into an audio file that could be played simultaneously with the muted video file to allow for interpretation in multiple languages. This consultation platform, with subtitles instead of an audio file, could be beneficial for multiple types of language translations, including sign language translation for providers of deaf individuals.

Future studies should focus on further refinement and validation of this consultation platform. Comparison studies should be conducted of ATP-translated consultations with interpretation occurring over videoconferencing as well as with in-person consultations. Outcome studies comparing ATP with face-to-face psychiatric care are also needed. We have already found that ATP is a more cost-effective clinical process than either real-time telepsychiatry or in person consultations.²⁰ Thus, it is important to determine the efficacy and economics of ATP when compared against telepsychiatry interpretation over videoconferencing and in-person consultations with an interpreter.

Footnotes

Acknowledgments

This research was funded by the Blue Shield of California Foundation (grant PYBSF01). We thank the anonymous patients and providers, as well as the medical interpreters at the University of California Davis Health System for their participation.

Disclosure Statement

No competing financial interests exist.

References

Kim

, Aguado Loi

, Chiriboga

et al. Limited English proficiency as a barrier to mental health service use: A study of Latino and Asian immigrants with psychiatric disorders. J Psychiatr Res, 2011; 45:104–110.

Flores

. The impact of medical interpreter services on the quality of healthcare: A systematic review. Med Care Res Rev, 2005; 62:255–299.

Bauer

, Chen

C-N

, Alegría

. English language proficiency and mental health service use among Latino and Asian Americans with mental disorders. Med Care, 2010; 48:1097–1104.

, Flores

. Pay now or pay later: Providing interpreter services in healthcare. Health Aff, 2005; 24:435–444.

Diamond

, Wilson-Stronks

, Jacobs

. Do hospitals measure up to the national culturally and linguistically appropriate services standards? Med Care, 2010; 48:1080–1087.

Ferguson

, Candib

. Culture, language, and the doctor-patient relationship. Fam Med, 2002; 34:353–361.

Wood

. Caring for a limited-English proficient patient. AORN J, 2002; 75:305–308.

Elderkin-Thompson

, Silver

, Waitzkin

. When nurses double as interpreters: A study of Spanish-speaking patients in a US primary care setting. Soc Sci Med, 2001; 52:1343–1358.

Beeber

, Lewis

, Cooper

et al. Meeting the “now” need: PMH-APRN–interpreter teams provide in-home mental health intervention for depressed Latina mothers with limited English proficiency. J Am Psychiatr Nurses Assoc, 2009; 15:249–259.

10.

Karliner

, Jacobs

, Chen

, Mutha

. Do professional interpreters improve clinical care for patients with limited English proficiency? A systematic review of the literature. Health Serv Res, 2007; 42:727–754.

11.

Yellowlees

, Odor

, Patrice

et al.

Disruptive innovation: The future of healthcare?

Telemed J E Health, 2011; 17:231–234.

12.

Deshpande

, Khoja

, Lorca

et al. Asynchronous telehealth: A scoping review of analytic studies. Open Med, 2009; 3:e69–e91.

13.

Baruah

, Rosa

. Optimal imaging in static telepathology. Indian J Pathol Microbiol, 2002; 45:367–370.

14.

Kayser

. Interdisciplinary telecommunication and expert teleconsultation in diagnostic pathology: Present status and future prospects. J Telemed Telecare, 2002; 8:325–330.

15.

Collins

, Nicolson

, Bowns

, Walters

. General practitioners' perceptions of store-and-forward teledermatology. J Telemed Telecare, 2000; 6:50–53.

16.

Whited

. Teledermatology. Current status and future directions. Am J Clin Dermatol, 2001; 2:59–64.

17.

Hooper

, Yellowlees

, Marwick

et al. Telehealth and the diagnosis and management of cardiac disease. J Telemed Telecare, 2001; 7:249–256.

18.

Yellowlees

, Odor

, Parish

et al. A feasibility study of the use of asynchronous telepsychiatry for psychiatric consultations. Psychiatr Serv, 2010; 61:838–840.

19.

Odor

, Yellowlees

, Hilty

et al. PsychVACS: A system for asynchronous telepsychiatry. Telemed J E Health, 2011; 17:299–303.

20.

Butler

, Yellowlees

. Cost analysis of store-and-forward telepsychiatry as a consultation model for primary care. Telemed J E Health, 2012; 18:74–77.

21.

First

, Gibbon

, Spitzer

et al.

Structured Clinical Interview for DSM-IV Axis II Personality Disorders, (SCID-II)

Washington, DC: American Psychiatric Press, 1997.

22.

Hersen

, Turner

, Beidel

. Adult psychopathology and diagnosis. 5th. Hoboken, NJ: John Wiley and Sons, 2007.

23.

Lobbestael

, Leurgans

, Arntz

. Inter-rater reliability of the Structured Clinical Interview for DSM-IV Axis I Disorders (SCID I) and Axis II Disorders (SCID II) Clin Psychol Psychother, 2011; 18:75–79.

24.

Sheehan

, Lecrubier

, Sheehan

et al. The Mini-International Neuropsychiatric Interview (M.I.N.I.): The development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. J Clin Psychiatry, 1998; 59,Suppl 20:22–33quiz 34–57.

25.

Landis

, Koch

. The measurement of observer agreement for categorical data. Biometrics, 1977; 33:159–174.

26.

Shore

, Savin

, Orton

et al. Diagnostic reliability of telepsychiatry in American Indian Veterans. Am J Psychiatry, 2007; 164:115–118.

27.

Diederich

, Yellowlees

, Al-Ajmi

. Ex-ray: Text classification and the assessment of mental health. Bruza

, Thom

. Proceedings of the Eighth Australian Document Computing Symposium. Canberra, Australia: CSIRO Discovery Centre, 2003; 75–76.