Depicting variability and uncertainty using intervals and error bars

Variability is inherent in most biological systems due to differences among members of the population. Two types of variation are commonly observed in studies: differences among samples and the “error” in estimating a population parameter (e.g. mean) from a sample. ¹ While these concepts are fundamentally very different, the associated variation is often expressed using similar notation—an interval that represents a range of values with a lower and upper bound.^2,3 In this article we discuss how common intervals are used (and misused).

Consider a study of weight of adult male rats from the hypothetical Z strain. With unlimited resources and error-free equipment, we could weigh the entire population. Doing so, we might find that the weight distribution (Figure 1(a)) is perfectly Normal with mean μ = 680 g (the center) and standard deviation (SD) σ = 10 g (the spread). ⁴ An appropriate summary might be “Normally distributed with μ ± σ = 680 ± 10 g”.

OPEN IN VIEWER

Figure 1.

(a) Normal distribution with mean μ = 680 g and SD σ = 10 g. Of the population, 68.3% falls within the interval (μ ± σ) and 95.4% within (μ ± 2σ). Only 2.3% of values are below 2σ from the mean and 2.3% above. (b) A skewed Normal distribution with μ = 680 g and SD σ = 10 g. Because of skew, >50% (57.3%) of values fall below the mean, there are essentially no values below μ–2σ and values above μ + 2σ are nearly twice as likely as in (a) (4.4% vs. 2.3%). (c) Twenty samples of size n = 10 (black circles) drawn from distribution in (a) shown with their means (magenta circles), 95% CIs (magenta line), 95% PI (light blue line) and 90% tolerance 95% TI (dark blue line). Because samples vary, some samples (yellow highlight) will yield unrepresentative estimates of the population (e.g. CIs that do not capture the population mean). (d) Same as (e) but with samples drawn from skewed Normal in (b). (e) Sampling distribution of means (SDM) of 1000 samples of size n = 3, 10 and 30. Samples were drawn from Normal in (a). (f) Same as (c) but with samples drawn from skewed Normal in (b). Despite the asymmetry of the population, the SDM is essentially Normal when n = 10. CI: confidence interval; PI: prediction interval; TI: tolerance interval; tol: tolerance

Note that ±10 g does not represent uncertainty—there is nothing uncertain about our theoretical scenario. The interval 680 ± 10 g represents a specific proportion of the population—and for a Normal distribution this is 68.3%.

However, many physiological variables are skewed and μ ± σ does not tell the full story (Figure 1(b)). For a skewed Normal distribution with μ = 680 g, σ = 10 g and the skew as a third parameter, the μ ± σ interval captures a proportion close to the Normal proportion (69.1% vs. 68.3%) but 42.4% of the values in the interval fall below the mean and 26.6% are above it. We find essentially no values smaller than μ − 2σ and twice as many as in the Normal (4.4% vs. 2.3%) larger than μ + 2σ. These proportions will differ for other non-Normal distributions.

Practically, we almost never have access to the entire population. By extension, we do not know μ or σ and must estimate these parameters from a sample—a randomly selected set of individuals from the population. And because samples will vary, we will have to account for the fact that these estimates will vary.

For example, suppose we select a sample of n = 10 rats from the Z population (Figure 1(c), sample 1). By weighing each rat, we can calculate a sample mean (denoted by the Roman letter m in contrast to the population mean μ) and sample SD (denoted by Roman letter s in contrast to the population SD σ). We might measure our sample to have m = 672.1 g and s = 8.4 g, but the next n = 10 sample (Figure 1(c), sample 2) would likely be different (m = 683.5 g, s = 6.7 g).

We can gain insight into the relationship between the sample (m) and population (μ) means by generating a histogram of sample means from 1000 samples (Figure 1(f)). This is called the sampling distribution of the mean (SDM)—it comprises sample means and not individual values. The Central Limit Theorem (CLT)—a key pillar of statistical theory—tells us three important facts about the SDM. ¹

First, the mean of the SDM is the population mean. In other words, the histograms in Figure 1(e) and (f) are centered at μ = 680 g, which is the (usually unknown) population mean μ.

Second, the SD of the SDM (often called the standard error (SE)) is related to the population SD σ by SE = σ/√n (here √ indicates the square root). The SDs of the distributions in Figure 1(e) are 5.7, 3.3 and 1.9 for n = 3, 10 and 30, which are very close to σ/√n = 5.8, 3.2 and 1.8, respectively.

And third, the SDM becomes more and more Normally distributed as the sample size increases for all “well-behaved” (e.g. finite variance) population distributions. If the population is Normal, then the SDM is Normal for all sample sizes (even n = 3). For our skewed Normal distribution, the SDM is already very close to Normal for n = 10 samples (Figure 1(f)). Usually n ∼ 30 is sufficient to make the SDM close enough to Normal for practical purposes but this rule-of-thumb does not apply to all distributions—very oddly shaped ones may require larger samples. Every sample summary has a corresponding (but not necessarily Normal) sampling distribution. For example, the sampling distribution of SD is skewed with the longer tail to the right.

These three observations allow us to extrapolate from a single sample to the population. For example, given a sample, where do we predict most of the population values to fall?

To answer this, we might naïvely mimic the μ ± σ interval using m ± s as a guess of where 68.3% of the population would fall. It turns out that, even when the population is Normal, m ± s tends to be too small, because it does not take into account the variation in m and s. For example, the m ± s intervals for our 1000 simulated samples provide average coverage for only 63.2% and 63.6% of their source Normal and skewed Normal populations; lower than expected (68.3% and 69.1%, respectively). Coverage is calculated as the proportion of the population distribution that falls within the interval.

For a Normal population, it is possible, however, to correct the form of m ± s to obtain the so-called prediction interval ³ (PI), with any desired average population coverage (100–α)% (α/2% below, α/2% above). Each sample will have its own PI that may have lower or higher coverage but, over many samples, the coverage of the PIs will average to exactly (100–α)%.

A (100–α)% PI is given by m ± t*√(s² + SE²) = m ± t*s√(1 + 1/n). Here, t* is the 1–α/2 quantile of the Student’s t distribution ⁵ on n–1 degrees of freedom (d.f.). The form of this interval incorporates two important corrections to m ± s.

The first correction factor, t*, takes into account the variation and downward bias in s as an estimate of σ (most samples have s < σ). For example, about 56% of n = 10 samples have s < σ.

The second factor √(s² + SE²) adjusts for both the spread of the population (estimated by s) and the uncertainty in the sample mean (estimated by SE).

As samples grow in size, their PIs approach intervals of the type μ ± z*σ, where μ and σ are known and z* is the Normal quantile. This is a consequence of the fact that for large n, m ≈ μ, s ≈ σ, and 1/n is negligible (1 + 1/n ≈ 1). For a 68.3% PI (α = 31.7%), the 1–α/2 = 84.1% quantile of t* approaches z* = 1, giving us μ ± σ. Similarly, for a 95.4% PI, t* approaches z* = 2, giving us μ ± 2σ.

The intervals m ± s can be interpreted as (100–α)% PI for α such that t*√(1 + 1/n) = 1. For example, for n = 10, we have t* = 1/√(1 + 1/10) = 0.95, which corresponds to α = 63.5% (very close to our simulated 63.2%). Similarly, for m ± 2s, t* = 2/√(1 + 1/10) = 1.91, which corresponds to α = 91.1%. For very small samples (n = 3), these coverages drop substantially to 52.2% and 77.5% respectively.

PIs guarantee coverage—but only on average. For example, the average coverage of the 95% PIs for 1000 samples from our Normal is 94.7%—but only 64% of samples meet the 95% threshold and 5% of samples have PIs with <81.2% coverage. And while these proportions are similar for the skewed Normal, the balance of PI coverage is skewed. While we expect (on average) 2.5% of the population to be below a 95% PI and 2.5% above, we find (on average) 0.2% of the skewed population below and 5.1% above. This has the effect of making large values appear as outliers, even though they are not.

Given this variability in PI coverage, we might like some type of assurance (called the tolerance) that the interval has coverage ≥ (100–α)%. For this, we would use the tolerance interval ³ (TI), which can be tuned to provide any level of assurance independently of its coverage—for example, at least 95% coverage of the population in (on average) 90% samples. The form of the TI is similar to the PI, except now t* is replaced by a larger and more complex quantity.

For our 1000 samples from the Normal, the average 90% tolerance 95% TI coverage is 98.0% with 87% of samples having >95% coverage. And if we collect more samples, we approach 90% tolerance—for example, 90.1% of intervals from 10,000 samples have >95% coverage.

Intervals can also be used to express the accuracy of m as an estimator of μ and give a sense of where the mean of different samples will fall, rather than values from the population. Not surprisingly, these intervals use SE = s/√n as the measure of variability with m ± SE commonly used for error bars. ²

The interval m ± SE can be tuned to have coverage (100–α)% of the SDM by constructing the (100–α)% confidence interval ² (CI) using m ± t*SE. It is important to note that the coverage of the CI interval represents the proportion of samples in which the true mean is in the interval, rather than coverage of a distribution. We expect CIs to vary between samples but if we construct 95% CIs from a large number of samples then 95% of them will capture the population mean μ. Note we do not say that there is a 95% probability of μ falling in an interval—the interval varies while μ is a fixed quantity that either is or is not in the interval. CIs for other data summaries such as the SD, skewness and so on can be obtained using the bootstrap method. ⁶

The 95% CIs and PIs, and the 90% tolerance 95% TI derived from our simulated populations, are shown in Figure 1(c) and (d). While their form can appear quite similar, they have very different uses and interpretations. PIs and TIs provide coverage of the population while CIs are intervals that include the true mean with known probability. And while PIs and TIs are accurate for simple random samples from a Normal population, the CLT tells us that CIs are accurate even if the underlying population is not Normal—as long as the sample size is large enough for the SDM to be close to Normal.

When the population is skewed right, taking a transformation such as logarithm (to any base) or square root makes the data more symmetric. This is particularly effective for log Normal distributions^7,8 where the SD might be large with respect to the mean and the lower bound of intervals is negative. In this scenario, intervals are computed on the transformed data and then the interval bounds are transformed back to the original scale to obtain asymmetric intervals with approximately the correct coverage. Appropriate tolerance bounds are available in many computer packages (e.g. R ⁹ ) for many commonly used distributions such as the Poisson.

Supplemental Material

sj-zip-1-lan-10.1177_00236772241247105 - Supplemental material for Depicting variability and uncertainty using intervals and error bars

Supplemental material, sj-zip-1-lan-10.1177_00236772241247105 for Depicting variability and uncertainty using intervals and error bars by Naomi Altman and Martin Krzywinski in Laboratory Animals