Estimating the Reliability of Dichotomous or Trichotomous Scores

Abstract

To meet the requirements of the No Child Left Behind Act, school districts and states must compile summary reports of the levels of student achievement in reading and mathematics. The levels are to be described in broad categories: “basic and below,” “proficient,” or “advanced.” Educational units are given considerable latitude in defining the percentile limits of these categories and the measures to be used in assessing the levels of each student. However, they must defend the validity and reliability of these measures. The question arises: How does one estimate the reliability of a score generated by a test of k items after the score is reduced to three or even two values? A precise solution, based on normality assumptions, requires the application of computer programs inaccessible to most local educators. Tables are provided in this article that enable local educators to deduce the reliability of broad-category scores (ρ_yy′) from the reliability of the original scores (ρ_xx′).

Keywords

reliability coefficient dichotomous score reliability trichotomous score reliability parallel-forms reliability

Get full access to this article

View all access options for this article.

References

Cohen, J. A. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.

Fishman, J. A. (1956). Note in Jenkins’“An improved method for tetrachoric r.” Psychometrika, 21, 305.

Goheen, H. W. , & Kavruck, S. (1948). A worksheet for tetrachoric r and standard error of tetrachoric r using Hayes’ diagram and tables. Psychometrika, 13, 279-280.

Guilford, J. P. , & Fruchter, B. (1973). Fundamental statistics in psychology and education (5th ed.). New York: McGraw-Hill.

Guilford, J. P. , & Lyons, T. C. (1942). On determining the reliability and significance of a tetrachoric coefficient of correlation. Psychometrika, 7, 243-249.

Hamilton, M. (1948). Nomogram for the tetrachoric correlation coefficient. Psychometrika, 13 259-267.

Hayes, S. P. (1943). Tables of the standard error of the tetrachoric correlation coefficient. Psychometrika, 8, 193-203.

Hayes, S. P. (1946). Diagrams for computing tetrachoric correlation coefficients from percentage differences. Psychometrika, 11, 163-172.

Huynh, H. (1976). On the reliability of decisions in domain-referenced testing. Journal of Educational Measurement, 13, 253-264.

10.

Huynh, H. , & Saunders, J. C. (1979). Accuracy of two procedures for estimating reliability of mastery tests (Research Memorandum 79-1). Columbia: University of South Carolina.

11.

Jenkins, W. L. (1955). An improved method for tetrachoric r. Psychometrika, 20, 253-258.

12.

Keats, J. A. , & Lord, F. M. (1962). A theoretical distribution for mental test scores. Psychometrika, 27, 59-72.

13.

Kendall, M. G. , & Stewart, A. (1979). The advanced theory of statistics, Vol. 2 (4th ed.). New York: Macmillan.

14.

Lancaster, H. O. , & Hamdan, M. A. (1964). Estimation of the correlation coefficient in contingency tables with possibly nonmetrical characteristics. Psychometrika, 29, 383-391.

15.

Lord, F. M. , & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.

16.

Pearson, K. (1913). On the probable error of a coefficient of correlation as found from a fourfold table. Biometrika, 9, 22-33.

17.

Peng, C. J. (1979). An investigation of Huynh’s normal approximation procedure for estimating criterion-referenced reliability. Unpublished doctoral dissertation, University of Wisconsin-Madison.

18.

Peters, C. C. , & Van Voorhis, W. R. (1940). Statistical procedures and their mathematical bases. New York: McGraw-Hill.

19.

Subkoviak, M. J. (1976). Estimating reliability from a single administration of a mastery test. Journal of Educational Measurement, 13, 265-276.

20.

Subkoviak, M. J. (1978). Empirical investigation of procedures for estimating reliability for mastery tests. Journal of Educational Measurement, 15, 111-116.