Type I and Type II Errors in Correlations of Various Sample Sizes 1

Abstract

This study documented the effect of sample sizes commonly seen in exercise science research on type I and type II errors in statistical tests of numerous correlations. Data on tennis string testing were used to examine zero-order and partial correlations between six variables for the population (N = 198) and three randomly drawn sub-samples of 99, 50, and 25. Sample size and statistical analysis procedure affected the rates of statistical errors. Reducing sample size increased type II errors 7% to 21% using correlation analysis. Partial correlation analysis of smaller samples increased type II errors 29% to 85%. Correlation studies of small sample sizes are likely vulnerable to type I or type II statistical errors and should be interpreted with caution.

Scientists in kinesiology and exercise science must balance ethical, pragmatic, and theoretical constraints in designing research. Given the complexity of the human body and behavior, as well as the kinds (basic or applied) of research questions, research design and statistical analysis of the data are challenging tasks. Over the years there have been several articles that have pointed out weaknesses in exercise science research and ways to improve research design, statistical analysis, and reporting (Thomas, Salazar, & Landers, 1991; e.g., Bates, Dufek, & Davis, 1992; James & Bates, 1997; Thomas, Lochbaum, Landers, & He, 1997; Atkinson & Nevill, 2001; Chen & Zhu, 2001; Mullineaux, Bartlett, & Bennett, 2001; Hopkins, Marshall, Battermam, & Hanin, 2009; Knudson, 2009; Kordi, Mansournia, Rostami, & Maffulli, 2011). Slowly gaining traction in the social sciences and now in the physical sciences is less emphasis on null hypothesis significance testing, with greater focus on the size of differences or associations (Hopkins, et al., 2009).

A controversial research design and statistics issue has been appropriate sample size. Since most research studies use statistical tests of differences or associations of interest, the design must have an adequate sample size to have a high probability of detecting a meaningful difference with the statistical test assuming the effect or association did exist (e.g., statistical power). The sample size, however, should also not be so large as to waste resources and detect small, meaningless or trivial effects or associations as “statistically significant.” One of the most common design problems in exercise science research may be small sample sizes with inadequate statistical power (Jones & Brewer, 1972; Christensen & Christensen, 1977; Thomas, et al., 1997; Speed & Andersen, 2000). Pyne and Hopkins (2012) recently described the sample sizes in both comparison and correlation studies presented at the 2012 American College of Sports Medicine Annual Meeting as “woefully inadequate.” This problem of small sample sizes is also common in psychology (Maxwell, 2004). Maxwell also noted that a contributing factor to the problem of underpowered studies was the testing of multiple hypotheses from one data set. A study with a small sample could have weak statistical power to detect a real difference for any specific hypothesis, but would be likely to make a type I error if the study tested a large number of dependent variables. Small sample sizes also increase the risk of bias due to sampling error and the influence of outlier observations.

Unfortunately, estimating appropriate sample sizes is difficult for researchers because of variations between designs (Hopkins, 2006), uncertainty in expected statistical error and power rates (Thomas, et al., 1997), and differences between sampling aligned with statistical testing or to determine magnitudes of differences with confidence intervals (Hopkins, 2007). Justifying sample sizes with statistical power calculations (Cohen, 1988; Gatsonis & Sampson, 1989; Bonett & Wright, 2000) can also be a subjective exercise that uses creative adjustments in error rates, variability, and the minimum clinically/pragmatically meaningful difference (Lenth, 2001; Batterham & Atkinson, 2005; Maxwell, Kelly, & Rausch, 2008). Bayesian models that are able to specify all levels of uncertainties, with both conditional and unconditional probabilities, can be solved or simulated to estimate sample sizes. Monte Carlo simulations have recently been employed to determine recommendations for sample sizes in factor analyses (Myers, Ahn, & Jin, 2011).

Correlation designs are also vulnerable to statistical errors in hypothesis testing due to small sample size and multiple comparisons. A study designed to correlate numerous variables from a single data set is more likely to see one “statistically significant” correlation due to chance alone than a traditional study with only two variables. Unfortunately, there are fewer models or rules of thumb for estimating sample sizes in correlation and regression, as there are in comparison designs. Correlation studies are usually recommended to have samples in the hundreds (Pyne & Hopkins, 2012), however some researchers have tried to apply the 10-participants rule of thumb in comparison group studies, to correlation/regression for N = 10 for each predictor or variable (Maxwell, 2004).

The purpose of this study was to document type I and type II errors in correlation studies with small sample sizes and large numbers of dependent variables typically seen in exercise science studies. It was hypothesized that smaller samples, common in exercise science, would have increasing type II errors rates than larger samples. The study took advantage of a unique data set from a whole population to examine the potential change in the inter-correlation of numerous dependent variables and their interpretation given smaller sample sizes common in exercise science.

Method

Data from mechanical testing of virtually all of the polyester tennis strings commercially available in the world from 2012 were analyzed in this study. Tennis Warehouse is a large independent retailer of tennis equipment and performs their own tests on equipment. Tennis Warehouse performs independent tests of samples of all tennis strings produced using a protocol (Cross, Lindsey, & Andruczyk, 2000) to measure mechanical response variables using a load cell and laser timing systems (Knudson & Lindsey, 2013). The results of these tests were performed by one of the authors and are available to the public at the company website² under the string performance database.

Data for all the polyester strings (N = 198) tested in the high speed and 62 pounds of tension condition were analyzed for six variables: gauge, impact duration, string deflection, peak force, string stiffness, and energy return. This essentially represents the entire population of polyester strings commercially available. Polyester strings were selected because that category provided the largest number of individual brands of strings tested in 2012.

The effect of various sample sizes was determined by examining by three different additional sub-samples from the original population. Three random samples of decreasing sample size (99, 50, and 25) were selected using a random number generator. Correlations were performed on the variables for all data sets using SPSS 20. An additional more rigorous partial correlation analysis was also performed, factoring out the one variable with the highest inter-correlations with the other variables. Statistical significance for each correlation was tested and accepted at a type I error rate of p < .05. The number of correlations identified as statistically significant in each data set were compared with correlations from the criterion data on the whole population (N = 198) using 95% confidence intervals. While the use of confidence intervals on population data in this study is not strictly necessary, however it does provide a uniform magnitude of differences in correlations (Hopkins, 2007). Statistical errors observed in the smaller sub-samples were false positives (type I errors: statistically significant sub-sample correlation not observed in the population, and false negatives (type II errors: non-significant sub-sample correlation when the correlation was significant in the population). Descriptive data for the string mechanical variables from the population and three sub-samples (Table 1) indicated normal distributions (skews between −.90 to .96). The consistency of the normative data for each of the random samples and the population supports the effectiveness of the random sampling used.

TABLE 1

Mechanical Variables of Polyester Strings for Various Sample Sizes (Ns = 198, 99, 50, and 25)

Sample	N = 198		N = 99		N = 50		N = 25
Sample	M	SD	M	SD	M	SD	M	SD
Gauge, mm	1.27	0.05	1.27	0.06	1.28	0.06	1.25	0.06
Duration, msec.	35.6	0.94	32.0	0.98	31.9	1.11	32.0	1.06
Deflection, mm	28.9	0.90	28.9	0.94	28.8	1.05	29.0	1.06
Peak Force, lb	34.4	1.04	34.4	1.07	34.5	1.08	34.3	1.07
Stiffness, lb/in.	250.9	23.4	251.3	25.5	252.5	26.0	248.0	27.2
Energy Return, %	90.1	1.50	90.0	1.44	90.0	1.48	90.2	1.74

Results

Fourteen of 15 zero-order correlations were statistically significant in the criterion data set (Table 2). The variables had strong inter-correlations, with stiffness most strongly related to the other variables, so partial correlations were performed controlling for association with stiffness. Partial correlations showed that 7 of 10 associations were statistically significant (Table 3), with strong associations remaining between string deflection and impact duration. Factoring out stiffness made most associations among the variables weaker, with the exception of a weak (r = −.22) inverse association between gauge and energy return becoming statistically significant.

TABLE 2

Zero-Order Correlations Between Polyester String Mechanical Variables For Various Sample Sizes

Variable	N	Duration	Deflection	Peak Force	Stiffness	Energy Return
Gauge	198	−.31 (−.43, −.18)	−.28 (−.41, −.14)	.33 (.20, .45)	.41 (.28, .52)	−.10 (−.24, .04)
	99	−.38	−.35	.36	.46	−.08
	50	−.40	−.37	.36	.47	−.13
	25	−.32	−.32	.11*	.41	−.12
Duration	198		.95 (.93, .96)	−.75 (−.81, −.68)	−.86 (−.89, −.82)	−.44 (−.55, −.32)
	99		.95	−.78	−.87	−.35
	50		.98	−.80	−.92	−.45
	25		.98	−.80	−.93	−.45
Deflection	198			−.60 (−.68, −.50)	−.87 (−.90, −.83)	−.41 (−.52, −.28)
	99			−.64	−.88	−.29
	50			−.71	−.92	−.43
	25			−.69	−.93	−.39 *
Peak Force	198				.79 (.73, .84)	.25 (.11, .38)
	99				.81	.23
	50				.85	.23*
	25				.82	.43
Stiffness	198					.23 (.09, .36)
	99					.15*
	50					.23*
	25					.36*

Note Correlation coefficients under each variable are in descending order of sample size (N = 198, 99, 50, 25) and 95% CI are listed for the largest sample. Statistically significant (p < .05) coefficients are in bold. *Type II error from inadequate sample size.

TABLE 3

Partial Correlations (95% Confidence Intervals) Removing Stiffness For Various Sample Sizes

Variable	N	Duration	Deflection	Peak Force	Energy Return
Gauge	198	.10 (−.04, .24)	.18 (.04, .31)	.01 (−.13, .15)	−.22 (−.35, −.08)
	99	.05	.12*	−.03	−.16*
	50	.08	.16*	−.07	−.27*
	25	.19	.18*	−.43^	−.31*
Duration	198		.81 (.76, .85)	−.21 (−.34, −.07)	−.50 (−.60, −.39)
	99		.81	−.25	−.45
	50		.87	−.11*	−.65
	25		.81	−.16*	−.36*
Deflection	198			.31 (.18, .43)	−.43 (−.54, −.31)
	99			.25	−.33
	50			.29	−.58
	25			.33*	−.15*
Peak Force	198				.11 (−.03, .25)
	99				.19
	50				.07
	25				.24

Note 95% CI are listed for the largest sample. Statistically significant (p < .05) coefficients are in bold. ^Type I and *type II errors.

Reducing the sample size to 99 resulted in zero-order correlations similar to the population data, with the exception of one false negative (type II error) of not identifying a weak association between stiffness and energy return. The more precise partial correlation analysis with this sample size resulted in two type II errors relative to the criterion sample.

The sample size of 50 also had qualitatively similar zero-order correlations to the population data for most variables, but now had two false negative (type II errors) tests. The partial correlation analysis also had false negative statistical errors.

The smallest sample size (n = 25) had greater variation between the zero-order correlation coefficients and the population correlations, and now had three false negative tests. The partial correlation analysis of this smallest data sample had six false negative errors, and also had one false positive (type I error). Partial correlation of this small data set has more statistical errors (one of the seven real associations was detected, along with a false negative association) than associations correctly identified.

Discussion

The present study confirmed theoretical trends of increasing statistical errors with decreasing sample sizes in correlation analyses of multiple dependent variables. Polyester tennis strings have a mix of strong, weak, and no associations between the six mechanical impact variables. Accurately capturing the size of these associations without statistical errors was dependent on sample size and the kind of statistical analysis performed.

Zero order correlations are the most common correlation technique but, unfortunately, conclusions from these tests have several threats to validity. First, this study showed that decreasing sample sizes increased the type II errors of zero-order correlations from 7% to 21%. Second, variables can have statistically significant zero-order correlations with each other so the variables appear to be associated, but they are not because the associations are accidental, erroneous, or spurious (Haig, 2003). Third, correlations are point estimates and a non-linear variable that should normally be interpreted with confidence intervals (Hopkins, 2007), the coefficient of determination (r²) or absolute magnitudes (Zhu, 2012). Adjusted versions of r² are often reported because sampling errors do not accurately model the population (Vacha-Haase & Thompson, 2004).

The more rigorous partial correlation analysis showed even larger negative effects of smaller sample sizes on statistical testing errors. Partial correlations of decreasing sample sizes increased type II errors from 29% to 85% with the smallest sample size also increasing type I errors to 33%. It could be concluded that based on these errors the N = 50 and N = 25 samples sizes were inadequate for an accurate correlation analysis of the six string performance variables studied. These results also call into question the common rule of thumb of having at least 10 observations for every variable of interest in correlation studies. Modern exercise science research equipment is often automated, providing access to numerous variables that increase the temptation for researchers to correlate or compare numerous variables with a modest sample or data set.

Testing numerous hypotheses from a single, small data set is the dominant kind of research design in exercise science, posing a risk to the veracity of the results of these studies (Morrow & Frankiewicz, 1979; Knudson, 2009). In applied biomechanics, e.g., 73% to 81% of original research reports test multiple hypotheses from the same data set (Knudson, 2005, 2009), unfortunately most often without correcting for inflation of the experiment-wise error rate. This presents a serious problem to the development of a coherent body of literature, because even illogical or meaningless effects can appear to be statistically significant from uncorrected testing of multiple hypotheses from the same data set (Austin, Mamdani, Huurlink, & Hux, 2006; Knudson, 2007). Much of the inconsistent results in scientific research reports can be attributed to uncorrected testing of multiple hypotheses in the same data (Maxwell, 2004; Ioannidis, 2005), as well as limited replication and poor review and citation of previous research (Knudson, Elliott, & Ackland, 2012).

The present study was limited in that a modest range of sample sizes were examined and the data represent the mechanical response of fairly uniform sports equipment. It is likely that associations between other mechanical, physiological, or behavioral variables with human subjects could be different, as well as the extent of the negative effects of smaller sample sizes. Future exercise science research should confirm and extend these results to assist researchers in interpreting previous correlation studies and planning well-powered studies.

In summary, the present study reported mechanical data from tennis strings to document the adverse effects of small sample sizes on statistical testing errors in correlation analyses of multiple dependent variables commonly seen in exercise science studies. These statistical errors were also dependent on the kind of correlation analysis performed. Zero-order correlations analyses of small samples (Ns = 25 or 50) resulted in 14% to 21% increases in type II errors, while similar partial correlations had 43% to 85% increases in type II errors. The more precise partial correlation analysis of the smallest sample size also resulted in a type I error. Results of previous studies using correlation analysis of small sample sizes in exercise should be interpreted with caution, given the potentially large percentage of associations not being identified as statistically significant, as well as the more rare association being falsely identified as statistically significant. Greater use of larger sampling, confidence intervals, and more precise partial correlation or regression analyses will likely improve the accuracy and consistency of the results of exercise science research.

Footnotes

2

References

Atkinson

Nevill

A. M.

(2001) Selected issues in the design and analysis of sport performance research. Journal of Sports Sciences, 19, 811–827.

Austin

P. C.

Mamdani

M. M.

Huurlink

D. N.

Hux

J. E.

(2006) Testing multiple statistical hypotheses resulted in spurious associations: A study of astrological signs and health. Journal of Clinical Epidemiology, 59, 964–969.

Bates

B. T.

Dufek

J. S.

Davis

H. P.

(1992) Effect of trail size on statistical power. Medicine & Science in Sports & Exercise, 24, 1059–1065.

Batterham

A. M.

Atkinson

(2005) How big does my sample need to be? A primer on the murky world of sample size estimation. Physical Therapy in Sport, 6, 153–163.

Bonett

D. G.

Wright

T. A.

(2000) Sample size requirements for estimating Pearson, Kendall, and Spearman correlations. Psychometrika, 65, 23–28.

Chen

Zhu

(2001) Revisiting the assumptions for inferential statistical analyses: A conceptual guide. Quest, 53, 418–439.

Christensen

J. E.

Christensen

C. E.

(1977) Statistical power analysis of health, physical education, and recreation research. Research Quarterly, 48, 204–208.

Cohen

(1988) Statistical power analysis for the behavioral sciences. (2nd ed.) Mahwah, NJ: Lawrence Erlbaum.

Cross

Lindsey

Andruczyk

(2000) Laboratory testing of tennis strings. Sports Engineering, 3, 219–230.

10.

Gatsonis

Sampson

A. R.

(1989) Multiple correlation: Exact power and sample size calculations. Psychological Bulletin, 106, 516–524.

11.

Haig

B. D.

(2003) What is a spurious correlation? Understanding Statistics, 2, 125–132.

12.

Hopkins

W. G.

(2006) Estimating sample size for magnitude-based inferences. Sportscience, 10, 63–70.

13.

Hopkins

W. G.

(2007) A spreadsheet for deriving a confidence interval, mechanistic inference and clinical inference from a P value. Sportscience, 11, 16–20 [online] http://sportsci.org/2007/wghinf.htm

14.

Hopkins

W. G.

Marshall

S. W.

Battermam

A. M.

Hanin

(2009) Progressive statistics for studies in sports medicine and exercise science. Medicine & Science in Sports & Exercise, 41, 3–12.

15.

Ioannidis

J. P. A.

(2005) Why most published research findings are false. PLoS Medicine, 2(8), 696–701.

16.

James

C. R.

Bates

B. T.

(1997) Experimental and statistical design issues in human movement research. Measurement in Physical Education and Exercise Science, 1, 55–69.

17.

Jones

B. J.

Brewer

J. K.

(1972) An analysis of the power of statistical tests reporting in the Research Quarterly. Research Quarterly, 43, 23–30.

18.

Knudson

(2005) Statistical and reporting errors in applied biomechanics research. In Wang

(Ed.), Proceedings of XXIII International Symposium on Biomechanics in Sports (Vol. 2). Beijing, China: China Institute of Sport Science. Pp. 811–814.

19.

Knudson

(2007) Analysis of multiple comparison errors using ground reaction force data. In Menzel

H-J.

Chagas

M. H.

(Eds.), Proceedings of the XXVth International Symposium on Biomechanics in Sports. Belo Horizontae: Federal Univer. of Mineas Gerais. Pp. 176–179.

20.

Knudson

(2009) Significant and meaningful effects in sports biomechanics research. Sports Biomechanics, 8, 97–105.

21.

Knudson

Elliott

Ackland

(2012) Citation of evidence for research and application in kinesiology. Kinesiology Review, 1, 129–136.

22.

Knudson

Lindsey

(2013) Change in the mechanical response of nylon tennis strings across tension. Journal of Medicine and Science in Tennis, 18, 59–62.

23.

Kordi

Mansournia

M. A.

Rostami

Maffulli

(2011) Troublesome decimals: A hidden problem in the sports medicine literature. Scandinavian Journal of Medicine and Science in Sports, 21, 335–336.

24.

Lenth

R. V.

(2001) Some practical guidelines for effective sample size determination. American Statistician, 55, 187–193.

25.

Maxwell

S. E.

(2004) The persistence of underpowered studies in psychological research: Causes consequences, and remedies. Psychological Methods, 9, 147–163.

26.

Maxwell

S. E.

Kelly

Rausch

J. R.

(2008) Sample size planning for statistical power and accuracy in parameter estimation. Annual Review of Psychology, 59, 537–563.

27.

Mullineaux

D. R.

Bartlett

R. M.

Bennett

(2001) Research design and statistics in biomechanics and motor control. Journal of Sports Sciences, 19, 739–760.

28.

Morrow

Frankiewicz

R. G.

(1979) Strategies for the analysis of repeated and multiple measures designs. Research Quarterly, 50, 297–304.

29.

Myers

Ahn

Fin

(2011) Sample size and power estimates for a confirmatory factor analytic model in exercise and sport: A Monte Carlo approach. Research Quarterly for Exercise and Sport, 82, 412–423.

30.

Pyne

D. B.

Hopkins

W. G.

(2012) Inadequate sample size in studies of athletic performance at the 2012 ACSM annual meeting. Sportscience, 16, 1–11.

31.

Speed

H. D.

Andersen

M. B.

(2000) What exercise and sport scientists don't understand. Journal of Science and Medicine in Sport, 3, 84–92.

32.

Thomas

J. R.

Lochbaum

M. R.

Landers

D. M.

(1997) Planning significant and meaningful research in exercise science: Estimating sample size. Research Quarterly for Exercise and Sport, 68, 33–43.

33.

Thomas

J. R.

Salazar

Landers

D. M.

(1991) What is missing in p < .05? Effect size. Research Quarterly for Exercise and Sport, 62, 344–348.

34.

Vacha-Haase

Thompson

(2004) How to estimate and interpret various effect sizes. Journal of Counseling Psychology, 51, 473–481.

35.

Zhu

(2012) Sadly, the earth is still round (p < .05). Journal of Sport and Health Science, 1, 9–11.