Abstract
The author provides a checklist for educators who are selecting technically adequate tests for identifying and referring students for gifted education services and programs. The checklist includes questions related to how the test was normed, reliability and validity studies as well as questions related to types of scores, administration, and format. The author concludes that selecting technically adequate tests is a critical step in developing an identification procedure that will be inclusive, reliable, and valid for finding students who will benefit from the school’s gifted education services and programs.
Keywords
Schools need to examine how closely their identification procedures follow local, state, and national policies and standards.”
Establishing identification and referral procedures is an important first step in finding students who will benefit from services and programs offered by the school. Most identification procedures will require educators to select and interpret quantitative assessments. This column will address the critical characteristics needed to identify and select technically adequate tests. Subsequent columns will examine these categories in greater depth and how educators should interpret scores in determining a match between the test results and each student’s strengths and needs.
Technically Adequate
What does technically adequate mean? A technically adequate test is one that can reliably (i.e., consistently) and accurately predict a student’s performance related to the construct or content being measured on the test. For example, if a test is supposed to measure a gifted student’s knowledge in math, then the items on the test relate to math content standards. Moreover, performance from one testing to the next testing on the math test, if given sequentially, should be similar or consistent; and students’ higher performance on the test should predict higher performance in math classes than peers.
Technical adequacy is also mentioned in Standard 4, Component 4.2 of the newly revised Initial Practice-Based Professional Preparation Standards for Gifted Educators (K-12): Candidates use norming, reliability and validity data, and information related to minimizing bias in selecting and interpreting assessments to identify and guide all students with gifts and talents, including those from traditionally underrepresented populations.
Gifted educators need to understand the concepts of norm-referencing, reliability, and validity to understand if the instrument is sensitive for identifying gifted students for specific services and programs.
Questions to Ask to Determine if a Test is Technically Adequate
All quantitative tests that are standardized and norm-referenced have technical manuals. These manuals include chapters related to how the test was normed (i.e., the sample of students who were administered the test), reliability (i.e., consistency within the instrument, across scorers, and over time), and validity (i.e., the accuracy of the content or construct that the test is supposed to measure and studies showing how the test minimizes bias, how the test discriminates between different groups, and how well the test predicts future performance). In addition, the technical manual may also have standardized directions in how to administer the test, which are very important in ensuring consistency, and norming tables to interpret the student’s raw scores. These scores are frequently presented as standard or index scores and percentile ranks.
In reviewing the technical manual, a gifted educator should ask these questions to determine the test’s technical adequacy and its usefulness in the identification process (see also Robins & Jolly, 2018):
Purpose of the Assessment
1. What do the test developers state is the purpose of the test? _____ Is it to assess achievement? _____ Is it to assess general intellectual ability? _____ Is it to assess critical thinking? _____ Is it to assess creative thinking? _____ Is it to assess performance in the arts? _____ Other: _________________________________ 2. Do the test developers state that the instrument may be used to identify gifted and talented students? _____ Are gifted students included in the norming sample? _____ Are gifted students included in validity studies? _____ If the purpose of the test is achievement, are items above level? 3. Does the test align with the school’s gifted education services and programs? _____ Is it aligned to the ages/grades where services are provided? _____ Is the alignment to general abilities? _____ Is the alignment to above-level performance in core content areas? ––––– Is the alignment to leadership aptitude? _____ Is the alignment to aptitude in the visual and performing arts? _____ Other: _________________________________
Normative Scores
1. Does the manual provide characteristics of the norming group? _____ Is the size of the norming group specified? _____ Does the norming group contain 100 or more subjects in each chronological age interval or academic grade level? _____ Are there at least 1,000 or more subjects in the total norming group? _____ Are the demographic characteristics of the norming group specified? _____ Are the demographic characteristics similar to the most recent national census data? 2. Was the standardization completed within the last 12 years?
Reliability
1. Do the test developers describe the test’s reliability (i.e., consistency)? ____ Does the test report an internal consistency reliability (i.e., to what degree the items relate to one another) of at least .80 for most ages or grades? ____ Does the test report stability reliability (i.e., test-retest comparisons) of .80 or greater? ____ Does the test report inter-scorer reliability (i.e., examiner variations in scoring) of .90 or greater? –––– If the test has alternate forms, do the scores relate to one another and have a reliability of .80 or greater?
Validity
1. Did the test developers include studies that show the test measures what it is supposed to measure? _____ Is content-description validity included (e.g., how the test relates to standards, curriculum content, theoretical frameworks for intelligence, creativity constructs, and so on)? _____ Did the authors provide studies showing how items discriminate (i.e., each item should relate to one another and to the rest of the items on the test)? _____ Did the authors describe the difficulty of the items and include floors (i.e., the lowest standard score that assesses functioning should be the lowest 3%)? _____ Did the authors describe the difficulty of the items and include ceilings (i.e., the highest standard score that assesses functioning should be the highest 3%)? _____ Did the authors describe the difficulty of the items and include item difficulty gradients (i.e., the number of items a student needs to pass to change the standard score should be more than one third of a standard deviation)? 2. Did the authors include studies related to test bias? _____ Did the authors show that individual items were not biased for different groups (i.e., differential item functioning—items do not function or predict differently for different groups)? ____ Did the authors include studies that showed that different demographic subgroups performed similarly on the test (e.g., male vs. female, White vs. Black/African American, and so on)? 3. Did the authors describe studies that showed how the test predicted a student’s performance on specific activities? ____ Did the authors show how the test correlated to other tests that measure similar content/constructs? ____ Did the authors show how the test predicted future performance (e.g., in a classroom for gifted students, on college level assessments)? 4. Did the authors describe studies that showed the degree to which a test reflects the theoretical model on which the test is based? ____ Did the authors show how the test correctly identified students who are gifted? ____ Did the authors show how performance was related to age since reasoning and academic achievement are developmental in nature? ____ Did the authors show how students who are above average aptitude performed better on the test overall and on different subtests? ____ Did the authors conduct a factor analysis to show how the subtests relate to one another and to a particular model of aptitude and/or achievement? ____ Other: _________________________________
Scores
1. Did the authors describe how the scores interpreted? ____ Does the manual provide standard or index scores that are comparable to other tests? ____ Does the manual provide percentile ranks that show the student’s performance relative to others who took the test? ____ Does the manual provide only grade equivalent or age equivalent scores, which are not comparable? ____ Other scores:____________________________
Along with the above technical qualities, schools will need to consider administration and examiner requirements because of human and material resources. The format of the test may also be important to consider, particularly for students who speak other languages or those with disabilities.
Administration
1. How is the test administered? ____ Is it administered individually? ____ Is it administered in whole group? ____ Is it administered in small group? 2. What is the length of the administration? ____ Is it less than 15 min? ____ Is it less than 30 min? ____ Is it less than 60 min? ____ Is it less than 60 min or longer? 3. Who can administer the test? ____ Can it be administered easily after reading the manual? ____ Does it require special training beyond reading the manual? ____ Is the test restricted to certified examiners such as psychologists, counselors, and other specialists? ____ Is the test administered by computer? 4. Who scores the test? ____ Can it be scored easily after reading the manual? ____ Does it require special training to score the test? ____ Is the test scored by a computer?
Format
1. What does the input require the student to do? _____ Listen _____ Read _____ Look at stimuli _____ Use a computer _____ Other: _________________________________ 2. What does the output require the student to do? _____ Point _____ Mark an answer on a response sheet _____ Indicate an answer on a computer _____ Manipulate objects _____ Draw _____ Speak (one word responses) _____ Speak (phrases, sentences) _____ Write short answers _____ Write longer answers; essays _____ Other required responses: _________________
Conclusion
Schools need to examine how closely their identification procedures follow local, state, and national policies and standards. In following these standards, it is important that all educators understand how to select technically adequate tests for use in the identification process. Selecting tests is a critical step in developing an identification procedure that is inclusive and valid for finding students who will benefit from the school’s gifted education services and programs.
ORCID iD
Susan K. Johnsen https://orcid.org/0000-0002-5425-3498
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Bio
Susan K. Johnsen, PhD, is Professor Emerita of Educational Psychology at Baylor University. She has written three tests for identifying gifted students and more than 300 publications related to gifted education.