An Integrated Telehealth System for Remote Administration of an Adult Autism Assessment

Abstract

We developed a telehealth system to administer an autism assessment remotely. The remote assessment system integrates videoconferencing, stimuli presentation, recording, image and video presentation, and electronic assessment scoring into an intuitive software platform. This is an advancement over existing technologies used in telemental health, which currently require several devices. The number of children, adolescents, and adults with autism spectrum disorders (ASDs) has increased dramatically over the past 20 years and is expected to continue to increase in coming years. In general, there are not many clinicians trained in either the diagnosis or treatment of adults with ASD. Given the number of adults with autism in need, a remote assessment system can potentially provide a solution to the lack of trained clinicians. The goal is to make the remote assessment system as close to face-to-face assessment as possible, yet versatile enough to support deployment in underserved areas. The primary challenge to achieving this goal is that the assessment requires social interaction that appears natural and fluid, so the remote system needs to be able to support fluid natural interaction. For this study we developed components to support this type of interaction and integrated these components into a system capable of supporting the entire autistic assessment protocol. We then implemented the system and evaluated the system on real patients. The results suggest that we have achieved our goal in developing a system with high-quality interaction that is easy to use.

Introduction

Videoteleconferencing is the most widely used technology for telemental health applications.^1,2 Many telemental health applications require additional functions beyond interactive videoconferencing, such as stimuli presentation and recording capabilities. A typical videoteleconferencing suite in telemental health consists of a videoconferencing system such as Polycom (San Jose, CA) and Tandberg (a Cisco Systems Company, Oslo, Norway) with a TV display at each location, a document reader for capturing hardcopy stimuli, a VHS videotape player for recording, and a computer system for transmitting still images and full-motion video.³ We have developed an integrated remote assessment system that integrates all these functions (videoconferencing, stimuli presentation, recording, image and video presentation) and electronic assessment scoring into an intuitive software. The integrated system is used to support remote assessment of adolescents and adults with possible autism spectrum disorder (ASD).

Adolescents and adults with ASDs primarily experience impairments in social interaction, impairments in communication, and restricted and repetitive patterns of behavior and interests.⁴ Accurate diagnosis is important because it has implications for treatment (e.g., access to appropriate supports and appropriate educational and vocational planning and accommodations). Many adolescents and adults with high-functioning ASDs are currently undiagnosed or misdiagnosed with emotional or psychiatric disorders. With the number of children identified with ASDs increasing dramatically over the past 20 years, the number of adolescents and adults with ASDs is also expected to increase in the coming years. In fact, the U.S. Centers for Disease Control and Prevention considers ASDs an urgent public health concern.⁵

The Autism Diagnostic Observation Schedule (ADOS) has recently become part of the gold standard in ASD diagnosis.^6,7 The protocol consists of a series of structured and semistructured tasks that involve communication, social interaction, and play or imaginative use of materials between the examiner (clinician) and the subject (patient). There are four ADOS modules, appropriate for different ages and expressive communication levels. ADOS Module 4 is used with adolescents and adults who are verbally fluent.⁸ Because ASD onset is in childhood, many ADOS administrators regularly see only young children (Modules 1 and 2). In general, there are not many clinicians trained in either the diagnosis or treatment of adults with ASD. Given the number of adults with autism in need and the lack of trained clinicians, assessment services are not available at the level required.

We have developed an integrated remote assessment system in order to address the aforementioned problems. The goal of the development is to make the remote assessment system as close to face-to-face assessment as possible, yet versatile enough to support deployment in low-resource underserved areas. The primary challenge to achieving this goal is that the assessment requires social interaction that appears natural and fluid, while existing telemedicine technologies are often a barrier to fluid natural interaction. An example of technology limitation that breaks the natural flow of interaction is a remote clinician might ask a technician to adjust the video or to present stimuli to the patient. We have developed an innovative solution that allows the clinician to have more control of the remote side in order to facilitate fluid natural interactions. The innovation includes the capability for the clinician to control the display and layout of the display on the remote side, the capability to present and control stimuli remotely, and the capability to control cameras on the patient side. This innovation, combined with the high-resolution video and low-latency interaction that often characterize high-end videoconferencing systems, is intended to achieve natural interaction between clinician and patients. We conducted formative usability studies on clinicians and summative usability studies on patients to evaluate if the goal of the development of an integrated system for remote administration of an autistic assessment has been achieved.

Materials and Methods

The remote assessment system is intended to serve remote areas with limited access to clinicians trained in ADOS, especially rural areas. Therefore, the system needs to make use of off-the-shelf technologies and should work with the broadband Internet bandwidth commonly available in rural clinics or schools, which is defined by the U.S. Federal Communications Commission has having data transmission speeds of at least 4 megabits per second (Mbps) downstream and 1 Mbps upstream.⁹ The system was developed based on the versatile and integrated system for telerehabilitation (VISYTER) platform,¹⁰ which provides versatility to adapt to the bandwidth in rural areas and requires no expensive devices but also has the capability to support integrated services. Typical remote sites will include small rural clinics, primary care doctor's offices, or schools. The remote sites will be connected to the ADOS-certified clinicians located in major medical centers. The system needs to be designed to support all phases of the assessment that go beyond conventional videoconferencing, including stimuli presentation and assessment documentation. In addition to being integrated and extensible, the system should require minimal equipment beyond standard commodity computers to minimize the initial investment cost. Second, the system has to be easy to install and to operate. This is not only to minimize maintenance costs but also to address the fact that the remote facilities usually have no information technology support staff. The system should be able to be operated by a technician, an administrative staff, or clinical assistant at the remote location.

ADOS was developed and validated by Dr. Catherine Lord and colleagues and was first published in 1999. The ADOS is a semistructured, standardized assessment of communication, social interaction, and play or imaginative use of materials for individuals who have been referred because of possible autism or other pervasive developmental disorders.¹² The ADOS Module 4 was designed for adolescents and adults who are verbally fluent (i.e., producing a range of flexible sentence types, providing language beyond the immediate context, and describing logical connections within a sentence). Module 4 consists of 10 standard activities and 5 optional activities. The activities focus on social, communicative, and language behaviors important in the diagnosis of ASD. They combine unstructured conversation with a variety of presses for particular kinds of social and communicative behavior. The general format of the test administration is meant to create an interaction that appears natural, during which preplanned occasions for certain behaviors arise.^12,13 The activities, their in-person requirements, and the remote translation of the requirements are detailed in Table 1.

Table 1.

Autism Diagnostic Observation Schedule Activities and Technology Development Input/Output Demands

ACTIVITY	TECHNOLOGY DEMANDS	SOCIAL AND COMMUNICATION DOMAINS (NOT SPECIFIC TO ACTIVITY)
Construction Task^a	Remote translation impossible	Stereotyped/idiosyncratic use of words or phrases Descriptive, conventional, instrumental gestures Unusual eye contact Facial expressions directed towards others Quality of social overtures Amount of reciprocal social communication Quality of social response Conversation Emphatic or emotional gestures Empathy/comment on others' emotions Responsibility Immediate echoing Speech abnormalities Imagination Mannerisms Unusual sensory behaviors Excessive, specific interests Rituals and compulsive behaviors Overactivity Negative behavior Anxiety
Telling a Story from a Book	Presentation of stimuli on a tablet that is controlled by both the administrator and the patient
Description of a Picture^a	Presentation of stimuli on a tablet that is controlled by the administrator
Conversation and Reporting	Face-to-face videoconferencing; observational camera
Current Work or School^a	Face-to-face videoconferencing; observational camera
Social Difficulties and Annoyance	Face-to-face videoconferencing; observational camera
Emotions	Face-to-face videoconferencing; observational camera
Demonstration Task	Face-to-face videoconferencing; observational camera
Cartoons^a	Presentation of stimuli on a tablet that is controlled by the administrator; Face-to-face videoconferencing; observational camera
Break	Face-to-face videoconferencing; observational camera
Daily Living^a	Face-to-face videoconferencing; observational camera
Friends and Marriage	Face-to-face videoconferencing; observational camera
Loneliness	Face-to-face videoconferencing; observational camera
Plans and Hopes	Face-to-face videoconferencing; observational camera
Creating a Story	Face-to-face videoconferencing; observational camera

Adapted from Lord et al.,¹³ pp. 207 and 209.

Optional activity.

After the 45–60-minute assessment, during which detailed observations are recorded, specific behaviors are coded based on the participant's behavior throughout the entire evaluation. The diagnostic algorithm classifies patients as having autism, ASD (including pervasive developmental disorder and atypical autism), or non-spectrum. We designed the remote assessment system to include an electronic scoring form and algorithm to calculate the diagnostic score as well as the group area (domain) scores.

Development of the Remote Assessment System

We developed an integrated system to address the requirements for a remote assessment discussed in the previous section. The system consists of the following components: videoconferencing, layout control, stimuli presentation, electronic scoring system, and session recording/archiving. Figure 1 shows a user interface on the clinician's station with the components of the integrated remote assessment system.

Fig. 1.

User interface on the clinician's station. It shows two videostreams from the (a) face-to-face and (b) observational cameras from the patient's side, (c) image capture, (d) pan-tilt-zoom on-video remote camera control, (e) electronic scoring system, (f) stimuli presentation, (g) stopwatch, and (h) a quick note.

Videoconferencing

The videoconferencing system is the main component of the system. It supports real-time interactivity between a clinician and a remote patient that resembles face-to-face communication. The system requires high-quality videoconferencing that should be as close to face-to-face as possible. The remote assessment system is designed to have the following capabilities:

• Low-latency and high-resolution audio and video. The system should have a latency of less than 250 milliseconds with synchronized video-audio on both ends and be capable of supporting high-definition (1280×720) or high-quality (800×600) video resolutions.

• Able to support more than one camera. Two cameras are needed on the patient's side: face-to-face and observational cameras. The first camera is primarily to support interviews, and the second camera is to observe nonverbal behaviors such as gesture or body language that cannot be observed by using just a face-to-face camera.

• Remote camera control. We built the capability for the clinician to remotely control the cameras on the patient's side. The unique feature of camera control that we developed allows the clinician to control the cameras (zoom, pan, or tilt) directly on the video windows using a mouse. This capability is especially important for the observational camera in activities such as the demonstration task, which requires the clinician adjust the camera or focus on a certain part of the patient's activities. The observational camera can be controlled by the clinician and is able to capture the client's hands and fingers to view complex mannerisms, gestures, and use of presented objects and materials.

• Image capture from camera. We developed an image capture capability that allows the clinician to take an image snapshot from videostreams that can then be used for clinical reports.

The videoconferencing system is the anchor for other components in the remote assessment system.

Layout control

Having multiple streams of video and stimuli on a screen monitor can be very confusing for the patient. The patient may also be uncomfortable and intimidated by the multiple videos of him- or herself, especially those from the observational camera. Therefore, the presentation on the patient's side needs to be as simple as possible, yet able to be changed to follow the progression of the assessment protocol. To accommodate this need, we developed a layout control system that allows the clinician to control the screen layout on the patient's and clinician's stations (local and remote layout control). One of the innovations of this remote assessment system is the capabilities for the clinician to control screen layout, stimuli, and cameras on the patient's station. Controlling what appears on the patient's screen is important because of the number of windows and streams that can appear on the patient's station that can include the patient's own videostreams, the clinician's videostreams, and stimuli. This includes deciding which videostream is presented on the screen and if more than one videostream will be presented. The clinician can control the layout of the streams using predefined layout settings that include choosing which video is the focus and is presented larger on the screen (e.g., one video large at the center with other videos on the left and right, or one large video on the left with other videos on the right, etc.).

The layout control is available for controlling both local and remote stations, with “R” icons on the menu indicating controls for remote layout. On the clinician's station, the local layout control is important because the clinician's screen is also crowded with many windows, including three or four videostreams (two from the patient's side, one or two from the clinician's side), a stimuli screen, and the electronic scoring system. Figure 2 illustrates how the clinician can change a layout on the patient's station remotely by clicking one of the predefined layouts.

Fig. 2.

Remote control of the layout. In this illustration, the clinician is using (left) remote layout control to change the appearance on the patient's station (upper right) from three parallel videostreams (lower right) to one clinician video as the focus.

Stimuli presentation

We developed a stimuli presentation component that allows the clinician to upload stimuli from the clinician's station and present it on the patient's station. If the stimuli consist of a series of stimuli, the clinician will be able to control which stimulus is presented one at a time, the same way a clinician would present a stimulus from a book page or from a deck of cartoons. The clinician can also control on which display the stimuli should be presented, on the tablet or at a specific location on the patient's display monitor. Because the clinician has full control of the stimuli, the clinician can show a particular stimulus (e.g., a page from a book) without having to ask for the patient to flip pages as in the face-to-face assessment. Thus, the interactivity and fluidity of the remote assessment system can be as good as or better than face-to-face assessment. This system will be capable of providing much broader and richer stimuli than the current document camera system used in telemental health. Any type of stimuli (picture, movie clip, animation, etc.) can be presented and is not limited to a document or still images as in the document camera system.

Electronic scoring system

In a face-to-face assessment, a clinician conducts scoring after the assessment is finished, writing quick notes to record any observations during the assessment. We developed a scoring system that mimics this process. The scoring instrument is implemented as a Web-based system that is integrated into the remote assessment system. The goal of the remote assessment system is to implement all the assessment and scoring activity electronically, with all processes conducted paperlessly. The assessment begins after the session is ended, as in face-to-face sessions. A Web-based evaluation form was developed, and the clinician assigns a score for each of the 31 evaluation items by selecting one of the scores in the combo box. The system automatically calculates the final score, as well as the scores of five areas (language, imagination, etc.) using the algorithms prescribed in the ADOS scoring assessment. Within the remote assessment system, the clinician can write quick notes during the assessment process. Notes are taken during administration of each ADOS module. Overall ratings are completed immediately after administration, even if the session is recorded.

Session archive database

The entire assessment session needs to be archived, including the videostreams and stimuli presentations. A secured session archived database was developed to allow clinicians to record entire sessions in a secure archive database server. In addition to serving as a record, the archive is very useful for scoring and for educational purposes as well. Any session can be replayed by the clinician during the scoring after the assessment to increase the accuracy of the assessment. Interesting cases from the assessment, segments of the session, or the entire session can be used to train new clinicians on how to do assessment or how to spot diagnostic events.

Other components

The clinician often uses paper notes to write observations during assessment that can later be used during scoring. We developed an electronic quick note for the clinician to type observations during assessment to replace the paper note. One of the requirements for the remote assessment system is to be as little intimidating and as similar to face-to-face sessions as possible. Having eye contact is important for this patient population, and this is difficult to achieve with a desktop conferencing environment. We use an inexpensive teleprompter that allows the clinician to see the patient using a periscope or to read verbatim protocol. A stopwatch with noticeable display was also developed for the clinician to measure how long the patient conducts an activity.

Usability Evaluation

We conducted two types of usability studies on the system: formative and summative usability studies. A protocol for usability studies was reviewed and approved by the Institutional Review Board of the University of Pittsburgh, Pittsburgh, PA. Participants in the formative usability studies were experienced ADOS administrators (at least 1 year of experience administering ADOS). Five clinicians were involved in the studies and were recruited using convenience sampling from the Autism Service, Education, Research, and Training network at the Western Psychiatric Institute and Clinic in Pittsburgh. Participants in the summative studies were students currently attending the Hiram G. Andrews Center, a state-operated vocational facility located in Johnstown, PA. The Hiram G. Andrews Center primarily serves consumers of services from Pennsylvania's Office of Vocational Rehabilitation. Students at the Center are individuals 17 years of age and older with a disability. Ten patients participated in the summative usability studies.

Results

Deployment of the remote assessment system

Figure 3 illustrates the deployment of the remote assessment system consisting of clinician's and patient's stations connected over the Internet. The patient's station consists of a computer with a 24-inch LCD monitor, two cameras (face-to-face camera on top of the monitor and observational PTZ camera on the table), a tablet for stimuli presentation, an echo-canceling microphone, and ADOS materials for the “Creating a Story” activity. The clinician's station does not have an observational camera and tablet. The remote assessment system was used to connect patients at a state-operated vocational facility located in Johnstown with the clinician at the University of Pittsburgh. Johnstown is a small town in the Laurel Highlands (Appalachian) region of Western Pennsylvania, a 2-hour drive from Pittsburgh. The patient's station is connected to the Internet via DSL cable with a speed of 768 kilobits per second up/2 Mbps down, whereas the clinician's station is connected to the Internet using fiber optic connection with a speed of above 5 Mbps.

Fig. 3.

(Left) Patient's station and (right) clinician's station.

The assessment process was initiated by the clinician and the remote site operator/technician by launching the VISYTER program, authenticating their identities to the server, entering the clinic room, having the patient in a front of the computer, and then beginning the assessment. After being connected, the clinician is able to control the entire session remotely. Throughout the session, the layout on the patient's side was mostly displaying only the face-to-face videostream ( Fig. 3, left), and hiding other streams such as the patient's own videos was intended to reduce distractions. It is important to note that the clinician was observing gestures throughout the assessment using the observational camera. In the first activity (“Telling a Story from a Book”), the clinician opened visual stimuli and sent it to the tablet on the patient's station ( Fig. 3, left). The clinician and the patient are able to work together to view the visual stimuli, both able to move forward or backward as desired. The next activity also required presentation of visual stimuli. The following five activities consisted of interviews, and the clinician used the face-to-face camera and the observational camera. The stimuli presentation was used again to support the “Cartoon” activity before the Break. The four activities following the Break were interviews using the cameras. The last activity (“Creating a Story”) required the clinician to move his or her camera so the patient could see his or her desk and then his or her face again. The clinician recorded the session, and VISYTER archived the entire session, including all the videoconferencing streams and stimuli presentations. The archives were available to the clinician for later review. After the session ended, the clinician used the electronic scoring system on the right side of the system and could access the archived session for a review.

Evaluation of the remote assessment system

The goal of a formative usability study is to identify usability problems and concerns and improve the usability of systems by addressing their problems. For this study, another goal of the formative study is to identify which aspects of the system can be adjusted to better match the standardized face-to-face administration. The methodology used in this study is the “cognitive walkthrough” usability inspection.¹⁴ In its original form, cognitive walkthroughs involve one or a group of evaluators inspecting a user interface by viewing a set of tasks and evaluating understandability and ease of learning. In this project, experienced clinicians participated in a tele-assessment to evaluate if the remote administration system replicates face-to-face administration conditions. The clinician used the system to perform a remote ADOS assessment on a mock client (i.e., a member of the development team). In this formative evaluation, no formal measurement was used. Instead, any issues encountered by the clinician during the walkthrough were identified, and subsequently the system was revised to address the issues. The results of this formative usability studies through walkthrough inspection were an operational telehealth system for administration of ADOS assessment.

Subsequent to the formative usability study and the resulting improved system, the system was used in a research protocol to remotely assess adults with an ASD diagnosis. The summative usability study was administered to the participants after they completed the remote ADOS administration. Ten patients were involved in the summative usability studies. We used a draft subset of a Telehealth Usability Questionnaire (TUQ) that we are currently developing, primarily to evaluate ease of use and learnability of the system, interface quality, interaction quality, reliability, and satisfaction and future use. Our TUQ adapts and combines questions from three questionnaires relevant to telehealth systems: the Technology Acceptance Model's Perceived Usefulness/Ease of Use,¹⁵ the Computer System Usability Questionnaire,¹⁶ and the Telemedicine Satisfaction Questionnaire.¹⁷ Specifically, the TUQ adapts questions on ease of use aspects from the Technology Acceptance Model's Perceived Usefulness/Ease of Use, adapts questions from the Computer System Usability Questionnaire on learnability, interface quality, reliability, and satisfaction, and adapts questions from the Telemedicine Satisfaction Questionnaire on interaction quality, and future use. This TUQ uses a 7-point Likert scale with 1 representing disagreement and 7 representing agreement. The results are shown in Table 2.

Table 2.

Results of the Usability Studies Using the Telehealth Usability Questionnaire

	QUESTION	MEAN SCORE
1	It was simple to use this system.	6.14
2	It was easy to learn to use the system.	6.14
3	The way I interact with this system (the computer screen and the tablet) is pleasant.	6.29
4	I like using the computer screen and tablet.	6.57
5	The computer screen and tablet are simple and intuitive.	6.57
6	This system is able to do everything I would want it to be able to do during the assessment.	6.57
7	I can easily talk to the clinician in Pittsburgh.	6.57
8	I can hear the clinician in Pittsburgh clearly.	6.50
9	I felt I was able to express myself effectively.	6.14
10	I can see the clinician in Pittsburgh as if we met in person.	6.00
11	I think the assessment provided over the telehealth system (using videoconferencing) is the same as the in-person assessment.	5.86
12	I feel comfortable communicating with the clinician in Pittsburgh.	6.50
13	Telehealth is an acceptable way to receive healthcare services.	5.83
14	I would use telehealth services again.	6.17
15	Overall, I am satisfied with this telehealth system.	6.50
	Total average	6.29

We used the same computers, peripherals, and network settings for the clinician's station in Pittsburgh and for the patient's station in Johnstown for all sessions. We did not encounter technical problems during the sessions. In our previous experience with deploying the telehealth system, the problem was usually caused by setting of the devices, peripherals, and Internet connections. Learning from this experience, we tested all the settings (video resolution, network speed, bit rate, compression, etc.) and peripherals (camera, speakerphone, and tablet) and kept the same optimal settings for all the sessions.

Discussion

Overall, the patients were satisfied with the telehealth system we developed, which received an average overall score of 6.5 out of 7. The system received high marks on ease of use and learnability (average of 6.14 for questions 1 and 2), interface quality (average of 6.5 for questions 3–6), and interaction quality (average of 6.3 for questions 7–10). Patients expressed that they were very comfortable with the use of the telehealth system (6.5 for question 12) and would use the system again (6.17 for question 14). However, the scores for the two questions related to general acceptability of telehealth as a replacement for in-person service were slightly lower (5.86 for question 10 and 5.83 for question 13). It seems that participants were marginally less satisfied (5.85 compared with 6.29 overall) when it came to the questions comparing healthcare received in-person and through telehealth and about whether telehealth is an acceptable way to receive telehealth service. The primary goals of the development of the remote assessment system was to have a system enabling high-quality interaction close to that in face-to-face assessment, a system that is fluid and easy to use, and one with overall good usability. The excellent scores the system received for ease of use and learnability (6.14 out of 7), interface quality (6.5 out of 7), and interaction quality (6.3 out of 7) suggest that these goals have been achieved.

An important contribution of the integrated assessment system is that it can be used for other types of assessments that require fluid interactions between clinicians and patients, such as neuropsychology tests. This type of assessment is very challenging and difficult to implement using existing technologies. To further evaluate the efficacy of the autistic remote assessment system, one of the authors (J.L.S.) is currently conducting research on the reliability and validity of the remote assessment system for autism. The reliability study will compare the remote assessment system with the face-to-face system, while the validity study will estimate the diagnostic accuracy of the remote administration of autistic assessment.

Footnotes

Acknowledgments

We gratefully acknowledge the help of Ashlee Filippone as well as the participants in this research. This project is funded in part by the National Institute on Disability and Rehabilitation Research's Rehabilitation Engineering Research Center on Telerehabilitation (projects number H133E040012 and number H133E980025) and by the Pennsylvania Department of Public Welfare's Autism Service, Education, Research, and Training project.

Disclosure Statement

Authors B.P., I.W.P., and A.S. hold patent pending for the VISYTER system.

References

Antonacci

, Bloch

, Saeed

, Yildirim

, Talley

. Empirical evidence on the use and effectiveness of telepsychiatry via videoconferencing: Implications for forensic and correctional psychiatry. Behav Sci Law, 2008; 26:253–269.

Yellowlees

, Shore

, Roberts

. Practice guidelines for videoconferencing-based telemental health—October 2009. Telemed J E Health, 2010; 16:1074–1089.

Grady

, Singleton

. Telepsychiatry “coverage” to a rural inpatient psychiatric unit. Telemed J E Health, 2011; 17:603–608.

American Psychiatric Association. Diagnostic and statistical manual of mental disorders (4th ed.) text revision. Washington, DC: American Psychiatric Association, 2000.

Autism and Developmental Disabilities Monitoring Network Surveillance Year 2006 Principal Investigators; Centers for Disease Control and Prevention (CDC). Prevalence of autism spectrum disorders—Autism and Developmental Disabilities Monitoring Network, United States, 2006. MMWR Surveill Summ, 2009; 58,10:1–20Erratum in MMWR Surveill Summ 2010; 59(30):956.

Le Couter

, Handen

, Hammal

, McConachie

. Diagnosing autism spectrum disorders in pre-school children using two standardized assessment instruments: The ADI-R and the ADOS. J Autism Dev Disord, 2008; 38:362–372.

Gray

, Tonge

, Sweeney

. Using the Autism Diagnostic Interview-Revised and the Autism Diagnostic Observation Schedule with young children with developmental delay: Evaluating diagnostic validity. J Autism Dev Disord, 2008; 38:657–667.

Bastiaansen

, Meffert

, Hein

, Huizinga

, Ketelaars

, Pijnenborg

, Bartels

, Minderaa

, Keysers

, de Bildt

. Diagnosing autism spectrum disorders in adults: The use of Autism Diagnostic Observation Schedule (ADOS) module 4. J Autism Dev Disord, 2011; 41:1256–1266.

Federal Communications Commission. Sixth Broadband Deployment Report. http://transition.fcc.gov/Daily_Releases/Daily_Business/2010/db0720/FCC-10-129A1.pdf. 2012 January 23.

10.

Parmanto

, Saptono

, Pramana

, Pulantara

, Schein

, Schmeler

, McCue

, Brienza

. VISYTER: Versatile and integrated system for telerehabilitation. Telemed J E Health, 2010; 16:1–6.

11.

Lord

, Rutter

, DiLavore

, Risi

. Autism Diagnostic Observation Schedule—WPS (ADOS-WPS) Los Angeles: Western Psychological Services, 1999.

12.

Lord

, Rutter

, DiLavore

, Risi

. Autism Diagnostic Observation Schedule: ADOS manual. Los Angeles: Western Psychological Services, 2008.

13.

Lord

, Risi

, Lambrecht

, Cook

Jr , Leventhal

, DiLavore

, Pickles

, Rutter

. The Autism Diagnostic Observation Schedule-Generic: A standard measure of social and communication deficits associated with the spectrum of autism. J Autism Dev Disord, 2000; 30:205–223.

14.

Nielsen

, Mack

. Usability inspection methods. New York: John Wiley & Sons, 1994.

15.

Davis

. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q, 1989; 13:319–340.

16.

Lewis

. IBM computer usability satisfaction questionnaires: Pyschometric evaluation and instructions for use. Int J Hum Comput Interact, 1995; 7:57–78.

17.

Yip

, Chang

, Chan

, MacKenzie

. Development of the Telemedicine Satisfaction Questionnaire to evaluate patient satisfaction with telemedicine: A preliminary study. J Telemed Telecare, 2003; 9:46–50.