Why We Need Reliable,Valid,and Appropriate Learning Disability Assessments

Background: The Current Situation

It is generally accepted that the number of students participating in postsecondary education in developed countries, including Canada, and registered with disability service offices because of a diagnosis of learning disability is substantial (Association Québécoise Inter-universitaire des Conseillers aux Ētudiants Ayant des Besoins Spéciaux, 2002-2010; Brinckerhoff, Shaw, & McGuire, 2002; Ebersold, 2008; Ontario Human Rights Commission, 2003; Sénécal, 2007; Vogel, Vogel, Sharoni, & Dahan, 2003). The actual size of the population of students with LD in Canadian universities is unclear. In an environment where education is a provincial responsibility, Pan-Canadian numbers are difficult to come by. However, in 2001-2002, Ontario reported that 8,188 university students received accommodation for a disability, and stated that “the most common type of disability cited by students at post-secondary institutions is a learning disability” (Ontario Human Rights Commission, 2003, p. 45). In a national study of disability service providers at 117 Canadian postsecondary institutions, Wolforth and Harrison (2008b) found that 41% of their respondents reported that 50% to 75% of students registered with their offices were designated as having a learning disability. Only 16% of respondents reported that fewer than 25% of their registered students were designated as having a learning disability. Students with diagnosed LD who also qualify for government financial assistance are accepted under the federal Grant for Services and Equipment for Students With Permanent Disabilities program (http://www.canlearn.ca/eng/postsec/money/grants/gsepd.shtml) and, where available, can also receive additional provincial grants. They are recognized on a par with students who have other types of disability for the purposes of receiving monies to cover disability related academic support costs such as specialized hardware and software and tutors. Quebec is currently considering whether to follow other provinces in the recognition of these students.

To receive disability related services and academic accommodations, a student must present documentation signed by a suitably qualified professional that verifies that they have a disability. In the case of visible disabilities and most medical and mental health conditions, this is quite straight forward. In the case of those diagnosed with a learning disability or attention deficit hyperactivity disorder, not only must the presence of an invisible disability be demonstrated with results from standardized testing but also the disadvantage experienced by the student must be documented. The validity of documentation and whether it illustrates such a disadvantage has become a hot topic among postsecondary disability service providers. Issues which challenge the core mission of colleges and universities include equality of opportunity and fairness, academic integrity, privileging students, the impact of targeted financial aid, potential internal grievances, and external legal challenges. All these are a cause of constant concern in an environment where students presenting documentation attesting to a learning disability make up a substantial portion of the case load of a disability service office.

Mai-Lynn had been a participant in a doctoral research project conducted by a graduate student, who was supervised by a registered psychologist, at her local university. As part of the project, she had taken a few reading tests, some standardized and some not. Through this process she had been identified by the doctoral student as having dyslexia. When she arrived at university, she presented a letter from the doctoral student stating she had dyslexia and requesting she be granted additional time and the use of a computer on exams. The disability services’ LD specialist told Mai-Lynn the testing did not satisfy the requirements of the university because it was not sufficiently comprehensive, and told her she would have to arrange for a full psycho-educational evaluation. The researcher intervened vigorously with the LD specialist on behalf of the student but, on the grounds of consistency and fairness for all students, and out of concern that testing in only one skill area might not have identified the full extent of the student’s difficulties, the specialist could not make an exception for this student. The LD specialist was also concerned that, compared with her peers, the few results presented in the letter did not seem to illustrate a disadvantage, but thought that perhaps more comprehensive testing might do so. Mai-Lynn decided not to seek an additional assessment and completed her year of university without accommodations. Nevertheless she achieved a CGPA of 3.7 out of 4.

With a significant number of students presenting psychoeducational documentation at the postsecondary level, it is probably reasonable to expect that some will arrive with less than perfect assessments. Students arrive from many different regions of the country and some come from overseas. There are no Canadian national standards for what is expected of a psychoeducational assessment, and particular provinces define their school level requirements according to their own needs and beliefs. Whereas colleges may have certain conditions imposed by provincial educational authorities, universities are largely independent entities. As with admission and graduation requirements, course content and program standards, they are able to set their own expectations for what constitutes a valid disability assessment. Many service providers across the country are members of the Canadian Association of Disability Service Providers in Postsecondary Education (CADSPPE), and through this professional organization, some level of informal national consensus for the expected elements that should be included in an acceptable assessment has been established (Wolforth & Harrison, 2008a.) Professionals in some provinces may have guidelines from their professional licensing bodies, though Philpott and Cahill (2008) found no evidence that regulating bodies intervened to ensure consistency of practice or of expertise in LD assessment practice across Canada, and we can therefore only conclude that psychologists are left on their own to decide on a definition of LD and how to test for it. Given a lack of national or provincial regulation, each institution is also left to decide what it will accept as valid evidence of a learning disability, and, depending on the expertise contained within the institution’s disability office, these requirements could vary substantially. This may leave some institutions vulnerable to human rights challenges on grounds of discrimination.

Sophia arrived at university with a recent assessment that clearly stated she had a reading disability. Her reading comprehension scores were within the average range at the 60th percentile, but timed reading comprehension tests showed she was slow reader. Her vocabulary was quite poor and her phonological skills were all weak. Her visual spatial skills were well above average and the performance scores on the IQ test was significantly higher than her verbal scores. Initially the disability service provider read the assessment and anticipated meeting a student with the diagnosis defined by the psychologist in the assessment, a reading disability. However, at their intake interview, it became apparent that Sophia had only lived in the United States for 1 year, and had completed most of her schooling in Jordan, studying in Arabic. She had no problem with reading in her first language (and had never struggled academically in her native language) but clearly had second language problems with both spoken and written English. The psychologist had clearly not observed the norms of the tests used which were not designed for use with students who speak English as a second language, nor had the psychologist ruled out ESL as a possible explanation for her low scores. Sophia needed help with her language skills but she did not qualify for assistance through the disability services office.

Definitional Issues

As discussed elsewhere in this special issue (e.g., Harrison & Holmes, 2012), the picture is also complicated by different jurisdictions failing to employ a common definition of a learning disability. For example the Learning Disabilities Association of Ontario (2001) definition is somewhat different from the Learning Disabilities Association of Canada (2002) definition. The Diagnostic and Statistical Manual of Mental Disorders (4th ed., text revision; DSM-IV-TR; American Psychiatric Association, 2000) offers a consensus-based set of criteria for various learning disorders different from these other two definitions. The National Committee on LD (2001) in the United States puts the definition once again in slightly different terms. Interpreting and applying these definitions in the clinical field has also presented challenges. Clinical practice has tended to develop separately from the findings of researchers, so the operationalization of the construct of “learning disability” has evolved within a framework of controversy between clinicians and researchers (Fletcher, Morris, & Lyon, 2003; Siegel, 1999; Stanovich, 1999; Stanovich & Stanovich, 1997). Different stakeholders (politicians, advocates, educators, researchers) vary in terms of the aspects of the definition they emphasise (Kelman & Lester, 1998; Keogh, 2005). Researchers have focussed primarily on exploring the contribution of higher and lower order processes related to, for example, phonological processing, the contribution of different aspects of memory, and perceptual, motor and visual spatial processing as they apply to various aspects of core academic skills such as reading and mathematics (Fletcher, Denton, & Francis, 2005). Much, but not all, of this research has focussed on children rather than the young adults we see at university.

For researchers, the key to understanding and diagnosis is the identification of a pattern of unexpectedly poor underlying processes which explain a core academic deficit. In contrast, the clinical field adopted a discrepancy definition model and applied a statistical formula which computed a standard deviation or discrepancy between IQ, defined as potential, and levels of academic achievement. The acceptable deviation varied by jurisdiction, with some applying a standard deviation of 1, others using 1.5, and still others as much as 2.5. This variation alone emphasises the lack of empirical support for such a model (Siegel, 1989; Stanovich, 1989, 1991; National Research Center on LD, 2007), with differing discrepancy levels having more to do with the political and economic goals of school boards rather than with defining a true disability (Kelman & Lester, 1998). Yet students still arrive with documentation which includes several pages of computed statistical discrepancies many of them measuring unrelated factors of no apparent relevance to a diagnosis or the purported academic deficit.

Hani arrived at university with an assessment which diagnosed him as having a learning disability (nonspecific type). He had done very well at high school and graduated with excellent grades. His psychologist had administered a full battery of tests including the IQ battery and the Achievement battery. His assessment indicted that the majority of his test scores fell above the 90th percentile, but he had a few lower scores between the 78th and 83rd percentile. The psychologist had used the program provided with the tests to compute cross test discrepancies for a number of factors. Perhaps not surprisingly, some of these computations found standard deviation discrepancies at a significant level. However, the service provider did not agree that Hani had a disability. Many of the factors used to compute discrepancies had no clear relationship with each other, and showed no pattern of weakness. Most importantly, compared with the regular student body, Hani’s scores clearly showed no relative disadvantage.

Why Do University Learning Disability Specialists and Service Providers Need Reliable Assessments?

In postsecondary education, professionals who work with students who have LD are generally the ones who determine if documentation is acceptable. Unlike many school boards, most colleges and universities do not have the resources to carry out their own psychoeducational assessments for students but rely on external psychologists to provide the reports. These are administered at the personal expense of the student or their families, though some students may receive funding for this from the federal Grant for Services and Equipment for Students With Permanent Disabilities. Given that almost universally these reports are expected to be the work of licensed psychologists, institutions have the right to expect that reported findings and recommendations are reliable and justifiable. However, Harrison, Nichols, and Larochette (2008) found that less than 9% of students appeared at Ontario postsecondary institutions with adequate assessment reports that included a proper diagnosis. Similarly, Wolforth and Harrison (2008b) found that 50% of their sample of 111 Canadian disability service providers reported that more than a quarter, and in some instances, up to three quarters, of students requesting LD related accommodations arrived with inadequate documentation. Most of these inadequate reports failed to substantiate the presence of a learning disability as indicated by lower than average levels of performance on administered tests.

Given these findings, it is fair to ask questions about those making these judgements of inadequacy. Deciding on the validity of a report requires the reader to have a background in learning disability theory and assessment. Yet postsecondary institutions vary considerably in terms of the presence of required knowledge and qualifications of staff at disability offices. In their national study, Wolforth and Harrison (2008b) found that 58% of the disability service providers who responded had a master’s degree, with the majority of those degrees being in an education related field, but only 8% had a doctorate. This means that 44% of service providers had at most a bachelor’s degree, something which makes it less likely that they have completed specific training in learning disability theory, assessment or psychometrics. It is likely to be problematic for this group to attempt to interpret scores and understand the implications of test results. Fifteen percent of university service providers and 13% of college respondents did rate their comfort level in reading reports as poor to fair. Sixty-five percent of university respondents and 66% of college respondents reported good to excellent levels of comfort in reading assessment reports though, interestingly, these confidence levels were inversely related to overall reported levels of education. Thus, those with the least education felt more confident in their ability to understand assessments than did their peers with doctorates.

This implies the worst of both worlds. Disability service providers may not have the qualifications or confidence to judge the validity of specific testing conclusions and the related recommendations for accommodations, and yet, the quality of reports being submitted by students is being judged as inadequate. This makes it crucial that universities and colleges be able to rely on the professionalism and credibility of psychologists who carry out assessments. If disability service providers do not feel this confidence, it makes it difficult for them to support the implementation of academic accommodations for a student with faculty, or to find ways to assist the student adequately when this is required.

Why Do Students Need Reliable and Valid Assessments?

Unlike at earlier levels of education, postsecondary level students are recognized as adults, and they are expected to assume responsibility for understanding the nature of their disability, and be able to explain it to others, including their professors, and ultimately their employers. This is the stage when they have to learn to own their strengths and weaknesses and find ways to build on the former and work round or accommodate the latter. Many students who have received poor assessments at the school level arrive at university unprepared for these expectations. Wolforth and Harrison (2008b) found that 35% of postsecondary service providers believed that many students with LD are overaccommodated at high school. They identified key excessive supports as giving students too much time, including unlimited time for exams and assignments, and too much parental involvement and other help including overclarification of questions. This often leads to students not being taught how to manage time, including exam time, and how to apply exam-taking strategies. Compared with their peers, some students have been permitted to cover lesser amounts and levels of material yet have been assigned grades equal to those peers. Students diagnosed (sometimes erroneously) with a reading disability may not have been offered remedial help and arrive at higher levels of education with poor reading, writing, and study strategies. The diagnosis of “learning disability” at the elementary/secondary school level may have become something of a catchall for receiving help but, conversely, may prevent some students from receiving the type of remediation and skill development they need to prepare for college or university.

Christine, a student in a prestigious private school, had had a difficult Grade 9 year. Her parents were going through a divorce, her best friend had moved to another school and her boyfriend had left her. Her school notified the parents that they were concerned because she was not finishing her work and seemed to have difficulty concentrating in class. The school recommended she be assessed for a learning disability and she was diagnosed with ADD and given medication. In addition, her school recommended that she consider another educational option because they had no disability support resources, and she subsequently moved to public school. At her new school, she was given accommodations for ADD throughout Grades 10-12 and did well with unlimited time on all her exams. When she arrived at university the service provider questioned her original assessment. He was concerned that, with so many other things going on in her life in Grade Nine, Christine had been erroneously diagnosed with ADD, and that perhaps she had been depressed instead. He requested that Christine undergo a new assessment. This new assessment did not confirm the original diagnosis. Christine then had to begin to learn how to cope without accommodations for her exams and to realign her self image to one where she did not have a disability. This clearly had both negative and positive impacts on her first year university experience.

Why Do Agencies External to Universities Need Reliable and Valid Assessments?

A number of stakeholders external to colleges and universities and to the student have an interest in the assessment process. The federal government agency, Human Resources and Skills Development Canada (HRSDC), that dispenses costs for assessments and services through the Grant for Services and Equipment for Students With Permanent Disabilities is likely to remain concerned with the constantly rising costs associated with their program (HRSDC personal communication to author, 2009). The Bursary for Students with Disabilities in Ontario (https://osap.gov.on.ca/OSAPPortal/en/A-ZListofAid/UCONT004257.html) and the equipment program overseen by Assistive Technology British Columbia (http://www.at-bc.ca/programs.html) are other programs that rely on valid assessments to disperse funding of several thousand dollars each to eligible students. To maintain the credibility of federal and provincial government funding programs, officials must be assured that assessment practice, and therefore requests for financial assistance to cover disability related service and equipment costs, are reliable and legitimate. Assessments which are not valid can result in students collecting substantial public funding with no justification. This of course goes beyond distortion of the funding program and results in the student committing fraud.

Assessments that are not reliable can also result in the involvement of the internal institutional grievance system or of the external human rights or legal systems. An unreliable assessment can lead students or their parents to believe that their child is entitled to accommodations which the college or university does not consider “reasonable.” Demand for excessive extensions of time for exams or of exceptions from core components of a program are examples of such demands. Legal challenges at whatever level exact a toll on the student, on the service provider, and on the institution. This is not to suggest that if students are truly being denied a justifiable accommodation that they should be discouraged from fighting for it, but psychologists need to understand the consequences of their actions and guard against misleading students into thinking that they have the right to certain interventions when this may not be true. A key component of this process is the defensibility of the assessment. If a college or university can challenge the format and conclusions of an assessment on legitimate grounds, including accepted empirical findings in the field, as well as academic integrity and fairness, it will generally not hesitate to do so.

As Izabela did poorly in her 1st year at university and at the beginning of her 2nd year, she presented the Learning Disability advisor with an assessment which she had completed over the summer. The psychologist had diagnosed Izabela with a reading disability and recommended that she be given triple time for exams, that she not be required to complete all the required assignments for her courses, and that she be provided with a note taker. She was also advised to take a reduced course load. She arrived at the Learning Disability Office expecting to be given these accommodations. She had, after all, paid CAD$2,000 for the assessment. The LD advisor looked at the document and was surprised to see that Izabela’s test scores were well within the average to high average range but that she had attained one low score on a test of reading rate. The diagnosis of reading disability was based on this score alone. The advisor felt that there was insufficient basis for this diagnosis and refused the request for accommodations. Izabela was very upset and filed an internal grievance which, after 6 weeks, was heard at a formal grievance proceeding. The LD advisor was able to call several expert witnesses who explained the prevailing theory behind the diagnosis of reading disability and illustrate that Izabela’s assessment did not portray such a student. Izabela lost her case, but the emotional toll on her was great and she failed her semester.

Professional licensing bodies have also become interested in the reliability and validity of assessments because of the increasing demands of students for accommodations on licensing exams. These include medical associations and colleges, law societies, and business and accountancy licensing bodies. Students also frequently request accommodations on standardized tests such as the Law School Admission Test (LSAT) or the Medical College Admission Test (MCAT), and requests to these American bodies are frequently refused even if students have received accommodations while at university. A very strict interpretation of testing practice, assessment findings, and recommendations is used at such agencies. One is led to wonder if this cautious approach has been fed by concerns about the reliability and therefore the credibility of learning disability assessment.

Why Should Psychologists Ensure Their Assessments Are Reliable and Valid?

The answer to this question should of course be self-evident, but it is worth reiterating some key points. Many disability service offices at the postsecondary level have come to think of LD assessments as potentially unreliable and the diagnoses rendered by clinicians as potentially invalid. Many service providers approach the reading of assessments with great caution because they have become wary of encountering assessments with questionable credibility, and which sometimes seem to veer from a strictly impartial path in interpretation towards a more partial advocacy stance. This is unfortunate, but is within the power of psychologists to change. Most service providers would be delighted if they were relieved of the responsibility of having to challenge findings and recommendations and explain their position to students, and sometimes to their parents, and occasionally to the diagnosing psychologist.

One of the implications for this loss of professional credibility for psychologists is that, sadly, they may be feeding a growing perception that the assessment field is evolving into a conflict of interest situation for the profession. Generally, psychologists choose their field to help people. This is their raison d’être. When students (or their parents or teachers) take the trouble to seek help from a psychologist, they are presumed to be seeking help for a genuine concern about some measure of perceived academic difficulty. Especially at the high school level, this perceived difficulty may have a myriad of causes, often related to adolescent development, and the psychologist may feel pressured to ensure that, to regain good levels of academic performance, the student receives any available support in school. As noted above (Wolforth & Harrison, 2008b), postsecondary disability service providers interpret some of this assistance as “over accommodation,” especially if it is provided in the face of a less than definitive LD diagnosis.

An added potential complication that can feed the perception of conflict of interest is related to payment for assessment services. The assessing psychologist generally receives a significant sum of money from the client and some may feel pressure to produce evidence of any perceived academic “problem.” Poor assessments bolster the belief among disability service providers that a student can pay money to gain a specific diagnosis, and the psychologist is inevitably seen as complicit in gaining advantage for an undeserving student. The potential results for the student, the postsecondary system and external stakeholders, of portraying a phantom “problem” as a disability have been discussed above. Disability service providers are frequently asked by students and parents to recommend assessment psychologists. Presumably they develop a sense of who in their community is reliable and ethical and therefore who to recommend. Although unreliable assessments may generate income for the psychologist for a period of time, one hopes that, eventually, the reliable and impartial assessors will prevail.

Sadly, unreliable assessments undermine the credibility of assessors and the entire learning disability field. As an invisible disability, it has always been one of the most difficult disabilities for college and university disability service providers and their students to explain, and often to defend for the right to accommodation, particularly in professional programs. Research in both cognitive and neurological sciences has added enormously to our knowledge of the underlying causes of various LD. Test development companies have been quite diligent in keeping their instruments in line with this research. This means that a diagnosis of reading disability without evidence of weakness in foundational phonological processes, as measured by up-to-date tests designed for this purpose, cannot be seen as credible. Consideration of comorbidity or differential diagnosis is often missing from learning disability reports. If it is not a reading disability, could slow reading be the result of anxiety, or depression? Being given the wrong label adds to students’ difficulties and undermines attempts by appropriate professionals to treat them successfully.

A well-written, comprehensive, knowledgeable report gives students valuable, reliable, insightful information about themselves. It makes sure that students with valid LD get appropriate help and financial support and allows students with different types of problems to be effectively referred to other services and sources of support. It lets the disability service provider put in place appropriate accommodations and supports. Surely, it is incumbent on psychologists in clinical practice to keep up to date with relevant empirical findings and newer testing instruments so that their own conclusions are defensible and credible and truly help their clients. Why should we expect or accept anything less?