Best practice in statistics: The use of log transformation

Which transformation?

The albumin data are typical of the values reported for many assays. The values are either positive or zero – never negative – and there is a long tail to the right of higher values. One suitable transformation might be the square root. Importantly, this is valid even if there are zero values. When there are no zero values then a reciprocal transformation (1 divided by the value) may be useful, or a logarithmic transformation. The aim of all transformations is to produce a reasonably symmetric distribution. This provides a good basis for further statistical techniques. The transformed distribution need not be totally normal, although if it is, that would enable more confidence in tests based on smaller samples and might simplify statistical modelling of albumin.

If there are many zero values, then methods beyond the scope of this article will be required. If there are just a few (less than 2%), then reciprocal and logarithmic transformations may still be used after adding a small positive number to all values. Logarithm of zero is not well defined (minus infinity) and neither is 1/0 (infinity); adding a small number avoids this embarrassment. An example of such a transformation is the function y = 1/(0.01 + x) where x is the original measurement. Here, 0.01 has been chosen as a small value that makes little difference to the measurement but enables the transformation to be valid, that is, provide sensible values for all measurements.

Natural logarithm or common logarithm

It is possible to use either natural logarithms – to base e – or common logarithms – to base 10. The method is the same. The advantage of common logarithms is that they are more readily ‘interpreted’ or checked. For example, a log10 value of ‘2. xxx’ will lie between 100 and 1000 since log10 (100) = 2 and log10 (1000) = 3.

The transformed distributions, using a log10 transformation, are shown in Figure 2. This includes a fitted curve representing the normal distribution, with the same mean and standard deviation. Although the Shapiro–Wilk test, the most efficient test for normality, fails for both males and females, the fits do not visually appear to be poor. The symmetric shape of the distribution ensures that for these sample sizes, Welch’s t-test comparing the log10 (albumin) values of males and females will be valid. ² This is because the Welch t-test takes advantage of the central limit theorem – it compares means which will be normally distributed. The test statistic is t = 2.087 with 1012.2 degrees of freedom so that the p value is 0.0372; according to this statistical test, there is a significant difference in the mean values of log10 (albumin) (0.933 for males and 0.995 for females).OPEN IN VIEWER

It is not essential to transform albumin values in order to test the difference in the distributions. A log transformation however provides the opportunity to use a Welch t-test which had good validity. Had no transformation been used, then testing might have been instead undertaken with a non-parametric Wilcox rank sum test of the medians which yields a p value of 0.026. Although non-parametric, this test does assume that the distributions of males and females differ only by their median, that is, there is a shift up or down. In contrast, the Welch t-test tests a difference in means while allowing for a difference in variance or spread. This is why log transformation followed by a Welch t-test is preferred.

Back transformation

It is more helpful to think on the original scale (μg/mL) than on a logarithmic scale. A back transformation is therefore needed. For common logarithms, this is achieved by raising to the power 10. So, means transform, respectively, from 0.933 to 10^0.933 = 8.6 μg/mL and 0.995 to 10^0.995 = 9.9 μg/mL. Note that these are not the arithmetic means of the urine albumin measurements (which are 16.5 and 25.9 μg/mL). Instead, they are the geometric means, defined for measurements y₁ … y_n to be the n^th root of the product of them all. Note that zero values will cause problems here, and so need to be handled with care.

Reference intervals

A reference interval might be obtained by (a) transforming measurements so that they are normally distributed, (b) calculating a reference interval on the transformed scale, and then (c) transforming back (see, for example, Shine 2008 ³ ) Since we have shown differences in urine albumin between males and females, two separate reference intervals will have to be derived. This is not a textbook example, but rather an example based on real data with real issues.

The log transformations have not produced normal distributions; although close to normal, there are discrepancies in the upper tails of the distribution which will violate the assumptions necessary for the first method provided here (log transformation). It is included to illustrate the method and to show how the method can induce errors.

For males, the log-transformed variable has a mean of 0.934 and a standard deviation of 0.439. Under the – false – assumption of normal distribution, the reference interval for log10 (albumin) is (0.934 ± [1.96*0.439]) or (0.074–1.794). Transforming back by taking antilogarithms, we get (10^0.074–10^1.794) or (1.2–62.2), that is, a reference interval of 1.2–62.2 μg/mL is suggested. Note that we will have doubts about the upper limit since this is where the distribution of log10 (albumin) appears to vary from normality. The female sample has a mean of 0.995 and standard deviation, 0.495. Following the same procedure produces a reference interval of 1.2–92.2 μg/mL.

Transforming to a normal distribution and using 2.5% and 97.5% centiles of the fitted normal distribution can be useful when the sample size is small (fewer than 50 observations) and the approximation to normality is better, especially in the tails. In the NHANES example used illustratively here, the sample sizes are larger, and a reference interval can be obtained using the 2.5% and 97.5% centiles of the male and female urine albumin distributions. These are (1.1 and 88.3) for males and (1.1 and 155.1) for females. The reference intervals thus produced are 1–88 μg/mL for males and 1–155 μg/mL for females. Note the differences in the upper reference limits compared to those based on log10 transformation. The assumption of a normal distribution by the latter method is violated by the presence of values in the upper tails that are inconsistent with a normal distribution. This might occur if, say, some participants in NHANES have kidney disease and therefore elevated albumin in their urine.

Consider the impact that outliers might have on the dataset. These are often defined as observations that lie 1.5 times the size of the IQR beyond the upper and lower quartiles. Shine recommends their removal. ² For the male sample, this means removing values that lie above (1.5*(15.47–4.55) + 15.47) = 31.85, that is, 32 μg/mL. None are removed at the lower end since (1.5*[15.47–4.55]) is larger than 4.55 and all values are positive. For the female sample, the same process results in removal of values above 40 μg/mL. Removing outliers however does not ‘fix’ the lack of fit (for normality) in the tail area around the 2.5% and 97.5% percentiles in this example. Substantial differences remain between the direct percentile method and the method based on transforming to a ‘normal’ distribution.