Eliciting prior distributions for extra parameters in some generalized linear models

Abstract

To elicit an informative prior distribution for a normal linear model or a gamma generalized linear model (GLM), expert opinion must be quantified about both the regression coefficients and the extra parameters of these models. The latter task has attracted comparatively little attention. In this article, we introduce two elicitation methods that aim to complete the prior structure of the normal and gamma GLMs. First, we develop a method of assessing a conjugate prior distribution for the error variance in normal linear models. The method quantifies an expert's opinions through assessments of a median and conditional medians. Second, we propose a novel method for eliciting a lognormal prior distribution for the scale parameter of gamma GLMs. Given the mean value of a gamma distributed response variable, the method is based on conditional quartile assessments. It can also be used to quantify an expert's opinion about the prior distribution for the shape parameter of any gamma random variable, if the mean of the distribution has been elicited or is assumed to be known. In the context of GLMs, the mean value is determined by the regression coefficients. Interactive graphics is the medium through which assessments for the two proposed methods are elicited. Examples illustrating use of the methods are given. Computer programs that implement both methods are available.

Keywords

Elicitation Subjective prior distribution normal linear model gamma distribution generalized linear model interactive graphical software

1 Introduction

Methods for quantifying expert opinion about a generalized linear model (GLM) have been proposed (Bedrick et al., 1996; Chen and Ibrahim, 2003; Garthwaite et al., 2013). These methods all focus primarily on the task of quantifying opinion about regression coefficients. For some GLMs, such as logistic regression, this determines a complete prior distribution. But with some other common GLMs, such as the normal linear model and gamma GLMs, prior opinion about an extra parameter must also be quantified in order to complete the prior distribution for all model parameters.

A normal linear model is, of course, a form of GLM and the task of quantifying opinion about normal linear models has also been addressed by Kadane et al. (1980) and Garthwaite and Dickey (1988, 1992), amongst others. Some of these elicitation procedures contain a method of assessing opinion about a linear model's error variance, but the methods have drawbacks. For example, the method of Kadane et al. (1980) requires assessments of 0.9375 quantiles of predictive distributions in the part of their procedure that relates to the error variance. Assessing quantiles well into the tails of a distribution is a difficult task that people perform poorly (Alpert and Raiffa, 1969). If a different method were used to quantify opinion about the error variance, then that part of the procedure of Kadane et al. (1980) could be dropped. Garthwaite and Dickey (1988) separate the task of quantifying opinion about regression coefficients from that of quantifying opinion about the error variance. This is potentially beneficial; decomposing a complex assessment problem into a number of smaller problems is recommended (Hogarth, 1975). However the number of assessments that Garthwaite and Dickey (1988) elicit from the expert is the minimum number needed to determine the hyperparameters of the prior distribution for the error variance. A better approach is to elicit enough assessments to give several estimates of the hyperparameters and to then reconcile these estimates in some way (Kadane and Wolfson, 1998). This same criticism applies to methods used to quantify experts’ opinion about the variance of a multivariate normal distribution (Al-Awadhi and Garthwaite, 1998, 2001).

The first task addressed in this article is to assess a prior distribution for the error variance. The method of Garthwaite and Dickey (1988) is extended so as to obtain several estimates of the hyperparameter that is most difficult to assess (a degrees of freedom parameter). Reconciliation of these estimates (perhaps with further input from the expert) yields an overall estimate of it. Although designed to quantify opinion about an error variance, the proposed method may also prove useful in contexts where observations can be paired and their difference is normally distributed with a mean of 0. This is illustrated in an example.

The second task addressed in this article is to assess prior distributions for the shape parameter of a gamma distribution and the scale parameter of gamma GLMs. It is well-known that the scale parameter of a gamma GLM, which is the reciprocal of the dispersion parameter, is also the shape parameter of the gamma distribution. Bayesian methods have been developed for analysing data to estimate these parameters.

Miller (1980) proposed a general conjugate class of priors for the two parameters of the gamma distribution, but he gave no method of eliciting its hyperparameters. Sweeting (1981) introduced some suggestions for the Bayesian estimation of the scale parameters in exponential families. The problem of unknown scale parameters in GLMs was examined by West (1985). In his work, he discussed general ideas concerning scale parameters and variance functions in non-normal models including gamma GLMs, (see also West et al., 1985). However, there does not seem to be a good method of eliciting a prior distribution for such parameters. Ibrahim and Laud (1991) suggested a Jeffreys's prior for the regression coefficients and an independent marginal informative prior on the scale parameter of the gamma GLM, but they did not suggest any family of distributions for this informative prior. The method of Bedrick et al. (1996), which is considered as the first elicitation method of informative prior distributions for GLMs, assumed the scale parameter to be known and elicited priors only for the regression coefficients. Chen and Ibrahim (2003) proposed a novel class of conjugate priors for GLMs. They also discussed elicitation issues and strategies of these conjugate priors. Their proposed prior structure involves the dispersion parameter as well. However, no explicit elicitation method was introduced for this parameter.

Hence, although prior distributions for the shape parameter of a gamma distribution and the scale parameter of gamma GLMs have been proposed in the literature, no prior elicitation method for these parameters has been suggested. To fill this gap, we propose a new method for eliciting lognormal prior distributions for such parameters. It is based on conditional quartile assessments, where the condition is that the mean of the gamma distribution is known or has already been elicited. In the context of GLMs, this mean is given by assessments of the linear predictor.

The two methods proposed in this article are implemented in interactive graphics programmes that could be used as add-ons to any method of quantifying opinion about the regression coefficients of a normal linear model or a gamma GLM. Procedures for quantifying opinion about regression coefficients form an important area of subjective probability assessment (reviews are given in Garthwaite et al., 2005 and O'Hagan et al., 2006), but such procedures are outside the scope of this article, as assessing regression coefficients is separate to the tasks addressed here.

The article is organized as follows. In Section 2 we extend the method of Garthwaite and Dickey (1988) for eliciting the variance of random errors in normal linear models. In Section 3 a novel method for eliciting a lognormal prior distribution for the scale parameter in gamma GLMs is proposed. Implementations of the two methods are described in Section 4 and their use is illustrated through examples in Section 5. Concluding comments are given in Section 6.

2 Eliciting a prior distribution for the error variance in normal linear models

We suppose a dependent variable $Y$ is related to covariates $x_{1}, \dots, x_{m}$ through the normal linear model.

Y = α + β_{1} x_{1} + \dots + β_{m} x_{m} + ε,

(2.1)

where

ε

is assumed to be a normal random error with zero mean and an unknown variance

σ_{ε}^{2}

, i.e.,

ε \sim N (0, σ_{ε}^{2}) .

(2.2)

A conjugate prior distribution for

σ_{ε}^{2}

is the inverse chi-squared distribution [see, for example, Pratt et al. (1995), or Garthwaite and Dickey (1988)]. We assume,

σ_{ε}^{2} \sim inverse gamma (ν / 2, ν w / 2),

(2.3)

so its probability density function (pdf) is:

f (σ_{ε}^{2}; ν, w) = \frac{1}{Γ (ν / 2)} {(\frac{ν w}{2})}^{ν / 2} {(\frac{1}{σ_{ε}^{2}})}^{(ν / 2) + 1} exp \{- \frac{ν w}{2 σ_{ε}^{2}}\}, σ_{ε}^{2} > 0, ν > 0, w > 0 .

(2.4)

The parameters of this distribution (

ν

and

w

) are referred to as ‘hyperparameters’ and are the quantities that must be assessed.

In the method of Garthwaite and Dickey (1988), the expert is asked to suppose that two responses, $Y_{1}$ and $Y_{2}$ say, are observed at the same values of the predictor variables, so that the difference between them is due only to random variation. Let $Z_{1} = Y_{1} - Y_{2}$ . The expert is asked to give her median for their absolute difference, $| Z_{1} |$ , and $q_{0}$ denotes her assessment. She is then asked to suppose that the actual absolute difference between that pair of responses was a value the computer specifies, say $z_{1}$ . Conditional on this hypothetical datum, she is then asked to suppose that a further pair of responses are observed at identical values of the predictor variables. (The actual values of the predictor variables are not specified as these do not matter, as long as they are the same for the two responses.) She gives her median for the absolute difference between this pair of responses and we denote her assessments by $q_{1}$ . The hypothetical sample datum should cause her to revise her opinion; the degree of revision measures her confidence in her original assessment, yielding information about the degrees of freedom parameter, $ν$ . Garthwaite and Dickey (1988) use the assessments $q_{0}$ and $q_{1}$ to estimate $ν$ and $w$ .

As noted in the introduction, using just two assessments to determine two hyperparameters is undesirable. Here we extend this method so that a number of different hypothetical data sets are presented to the expert, the expert repeating the above task after each one. This yields several estimates of $ν$ that can be averaged.

Each hypothetical data set consists of a number of pairs of observations, where the two observations in a pair are taken at the same values of the predictor variables. Focusing on any one hypothetical set, suppose it consists of $k$ pairs of observations and let $Z_{i}$ denote the difference between the $i$ th pair. The expert is asked to suppose that $Z_{1} = z_{1}, \dots, Z_{k} = z_{k}$ and gives her conditional median for the absolute difference between a further pair of observations taken at one set of values of the predictor variables. This median is denoted by $q$ . We next derive formulae for estimating $ν$ and $w$ from $q_{0}$ and $q$ .

Clearly, from (2.1) and (2.2), given $σ_{ε}^{2}$ , the random variables $Z_{1}$ , $\dots$ , $Z_{k}$ are independent and identically distributed normal variates, i.e., for $i = 1, \dots, k$ ,

Z_{i} | σ_{ε}^{2} \sim N (0, 2 σ_{ε}^{2}),

(2.5)

with the joint distribution:

f (z_{1}, \dots, z_{k} | σ_{ε}^{2}) = \frac{1}{{(4 σ_{ε}^{2} π)}^{k / 2}} exp \{- \sum_{i = 1}^{k} z_{i}^{2} / 4 σ_{ε}^{2}\}, - \infty < z_{i} < \infty, σ_{ε}^{2} > 0 .

(2.6)

From (2.4) and (2.6), the joint distribution of

Z_{1}

\dots

Z_{k}

and

σ_{ε}^{2}

is given by:

\begin{matrix} f (z_{1}, \dots, z_{k}, σ_{ε}^{2}; ν, w) = \frac{{(ν w / 2)}^{ν / 2}}{Γ (ν / 2) {(4 π)}^{k / 2}} {(\frac{1}{σ_{ε}^{2}})}^{(ν + k) / 2 + 1} & exp \{- \frac{1}{4 σ_{ε}^{2}} (\sum_{i = 1}^{k} z_{i}^{2} + 2 ν w)\}, \\ - \infty < z_{i} < \infty, σ_{ε}^{2}, ν, w > 0 . \end{matrix}

(2.7)

Integrating

σ_{ε}^{2}

out from the RHS of (2.7), we get:

f (z_{1}, \dots, z_{k}; ν, w) = \frac{Γ ((ν + k) / 2)}{Γ (ν / 2) {[π ν (2 w)]}^{k / 2}} {[1 + \frac{\sum_{i = 1}^{k} z_{i}^{2}}{ν (2 w)}]}^{- (ν + k) / 2}, - \infty < z_{i} < \infty, ν, w > 0,

(2.8)

which is the

k

-variate version of the general three-parameter Student-

t

distribution with

ν

degrees of freedom, zero mean vector and a diagonal scale matrix

2 w I_{k}

, where

I_{k}

is the identity matrix of order

k

, i.e.,

Z_{1}, \dots, Z_{k} \sim MV - t_{ν} (0_{k}, 2 w I_{k}) .

(2.9)

Now, the conditional distribution of

σ_{ε}^{2}

given

Z_{1} = z_{1}, \dots, Z_{k} = z_{k}

, can be obtained by dividing the RHS of (2.7) by that of (2.8) to get:

\begin{matrix} f (σ_{ε}^{2} | Z_{1} = z_{1}, \dots, Z_{k} = z_{k}; ν, w) & = \frac{1}{Γ ((ν + k) / 2)} {[\frac{1}{4} (2 ν w + \sum_{i = 1}^{k} z_{i}^{2})]}^{(ν + k) / 2} \times \\ {(\frac{1}{σ_{ε}^{2}})}^{(ν + k) / 2 + 1} exp \{- \frac{1}{4 σ_{ε}^{2}} [2 ν w + \sum_{i = 1}^{k} z_{i}^{2}]\}, σ_{ε}^{2}, ν, w > 0 . \end{matrix}

(2.10)

Since the inverse gamma distribution is a conjugate prior for

σ_{ε}^{2}

, comparing (2.10) with (2.4), we can write:

(σ_{ε}^{2} | Z_{1} = z_{1}, \dots, Z_{k} = z_{k}) \sim inverse gamma ((ν + k) / 2, (ν + k) w_{k} / 2),

(2.11)

where:

w_{k} = \frac{1}{ν + k} [ν w + \sum_{i = 1}^{k} z_{i}^{2} / 2] .

(2.12)

Z

denotes the difference between two further observations taken at the same design point then

Z | σ_{ε}^{2} \sim N (0, 2 σ_{ε}^{2})

, so the conditional distribution of

(Z | Z_{1} = z_{1}, \dots, Z_{k} = z_{k})

is given by:

f (Z | Z_{1} = z_{1}, \dots, Z_{k (j)} = z_{k (j)}) = \int_{σ_{ε}^{2} = 0}^{\infty} f (Z | σ_{ε}^{2}) \times f (σ_{ε}^{2} | Z_{1} = z_{1}, \dots, Z_{k (j)} = z_{k (j)}) d σ_{ε}^{2} .

(2.13)

The integrand in (2.13) is similar to the RHS of (2.7) with

k

set equal to 1,

ν

replaced by

ν + k

, and

w

replaced by

w_{k}

As in (2.8) and (2.9), integrating $σ_{ε}^{2}$ out from (2.13) gives:

(Z | Z_{1} = z_{1}, \dots, Z_{k} = z_{k}) \sim t_{ν + k} (0, 2 w_{k}) .

(2.14)

Similarly, from (2.4) and (2.5), the marginal unconditional distribution of

Z_{1}

is:

Z_{1} \sim t_{ν} (0, 2 w) .

(2.15)

As $q_{0}$ is the assessed median of $| Z_{1} |$ , it is also the upper quartile assessment of $Z_{1}$ , since the distribution of $Z_{1}$ is symmetric about 0. Similarly, as $q$ is the median of $| (Z | Z_{1} = z_{1}, \dots, Z_{k} = z_{k}) |$ , it is also the upper quartile assessment of $(Z | Z_{1} = z_{1}, \dots, Z_{k} = z_{k})$ . If we denote the upper quartile of a standard Student- $t$ distribution with $n$ degrees of freedom by $Q_{n}$ , then we have

q_{0} = {(2 w)}^{1 / 2} Q_{ν},

(2.16)

and

q = {(2 w_{k})}^{1 / 2} Q_{ν + k} .

(2.17)

The aim now is to solve this pair of equations to obtain

ν

and

w

. By division,

\frac{q_{0}}{q} = \frac{Q_{ν}}{Q_{ν + k}} {[\frac{w}{w_{k}}]}^{1 / 2} .

(2.18)

Using (2.12) and (2.16) we can eliminate

w

from (2.18), giving:

\frac{q_{0}}{q} = \frac{Q_{ν}}{Q_{ν + k}} {[\frac{ν + k}{ν + Q_{ν}^{2} \sum_{i = 1}^{k} {(z_{i} / q_{0})}^{2}}]}^{1 / 2} .

(2.19)

A lower bound, $min (ν)$ , may be placed on the value that $ν$ might take. For example, if the expert's background knowledge is sufficient to give a prior mean for $σ^{2}$ , then $ν$ should exceed 2, as otherwise the prior mean is not defined. The expert's median assessment, $q$ , should satisfy:

q > q_{0} \frac{Q_{min (ν) + k}}{Q_{min (ν)}} {[\frac{min (ν) + Q_{min (ν)}^{2} \sum_{i = 1}^{k} {(z_{i} / q_{0})}^{2}}{min (ν) + k}]}^{1 / 2} .

(2.20)

A simple search procedure is used to find the value of

ν

that solves equation (2.19).

Each hypothetical data set yields a separate estimate of $ν$ . These are combined by taking their geometric mean, rather than their arithmetic mean, because empirical work suggests this is a better way of reconciling several estimates of a degrees of freedom parameter (Al-Awadhi, 1997). With $ν$ set equal to the geometric mean, the other hyperparameter, $w$ , is then obtained from equation (2.16).

3 Eliciting a prior distribution for the scale parameter in gamma GLMs

In this section, we propose a novel method for eliciting a lognormal prior distribution for the scale parameter of a gamma GLM. The method is a viable means of eliciting the shape parameter of any gamma distribution once the distribution's mean has been elicited (or the mean is assumed to be known).

Suppose a GLM has the form,

g [μ (x)] = g [E (Y | X = x)] = α + β_{1} x_{1} + \dots + β_{m} x_{m},

(3.1)

where

Y

is the response variable,

g (.)

is a monotonic link function,

{X = (X_{1}, \dots, X_{m})}^{'}

is the vector of covariates and

{x = (x_{1}, \dots, x_{m})}^{'}

is a design point. If

Y

is a continuous, positive valued and positively skewed distributed response variable, then the sampling distribution may be a gamma distribution, say:

Y \sim gamma (λ, θ),

where

λ

and

θ

depend on

x

. The pdf of

Y

is:

f (y | λ, θ) = \frac{1}{Γ (λ)} θ^{λ} y^{λ - 1} e^{- θ y}, y, λ, θ > 0,

(3.2)

where

λ

is the shape parameter and

θ

is the rate parameter (the inverse of the scale parameter). It is well-known that:

μ = E (Y) = λ / θ, σ_{Y}^{2} = Var (Y) = λ / θ^{2} .

(3.3)

For the gamma GLM in (3.1), with any monotonic increasing link function $g (.)$ , the method of Garthwaite et al. (2013) or the method of Bedrick et al. (1996) can be used to elicit the prior distribution of the regression coefficients

β = (α, β_{1}, \dots, β_{m}) .

(3.4)

The prior distribution of

β

determines the prior distribution of

μ

, i.e., reflects the prior knowledge about the ratio

λ / θ

. The expert's prior opinion about one of the hyperparameters

λ

and

θ

must be quantified to complete the prior structure of the gamma GLM model. In what follows, expert opinion about the scale parameter

λ

is modelled by a lognormal distribution and we propose an assessment method for determining the hyperparameters of this distribution. Specifically, we write the prior distribution for

μ

and

λ

as:

f (μ, λ) = f_{1} (μ) \times f_{2} (λ | μ),

(3.5)

where

ln (λ) | μ \sim N (a, b^{2})

. The task is to find values of

a

and

b

that correspond to the expert's opinions.

We base our method on a gamma distribution with $λ$ as the only unknown parameter, assuming $μ$ to be already assessed or completely known. We take the gamma distributed random variable $Y$ defined in (3.2) and change parameters by putting $θ = λ / μ$ , as in (3.3). This gives:

f (y | λ, μ) = \frac{1}{Γ (λ)} {(\frac{λ}{μ})}^{λ} y^{λ - 1} e^{- (\frac{λ}{μ}) y}, y, λ, μ > 0 .

(3.6)

We let:

W = \frac{Y}{μ},

(3.7)

and then the pdf of

W

will depend only on

λ

, i.e.,

W \sim gamma (λ, λ)

. It has the form:

f (w | λ) = \frac{1}{Γ (λ)} λ^{λ} w^{λ - 1} e^{- λ w}, w, λ > 0 .

(3.8)

We require a meaningful strictly monotonic function in $λ$ , such that the expert can quantify her opinion about this function effectively. The expert cannot be asked questions about $λ$ directly, as a gamma distribution parameter has little meaning to an expert because it is not an observable quantity. Instead, the expert should be asked about an observable quantity that directly relates to the gamma variate, and which can be monotonically transformed to $λ$ . The expert can then be asked about any quantile of the gamma distribution as an observable quantity. In what follows, we quantify the expert's opinion about the lower quartile of the gamma distribution in (3.8) and hence obtain a full prior distribution for $λ$ .

We believe that the expert can efficiently quantify her opinion about quartiles more easily by using the bisection method, see for example Pratt et al. (1995). As shown in Figure 1, the lower quartile of $W$ is a strictly monotonic increasing function for all values of $λ > 0$ , whereas the upper quartile does not have this property. It thus seems reasonable to base our method on the lower quartile of the gamma distribution. Another reason for choosing to ask the expert about the lower quartile rather than the upper quartile is that the lower quartile is more sensitive to changes in the the shape parameter $λ$ . Figure 1 illustrates this fact; it shows values of the lower and upper quartiles as $λ$ changes, and the greater sensitivity of the lower quartile is apparent, especially for $λ$ in the range 0.5 to 15.

Figure 1

Source: Authors' own.

So, we choose to question the expert about the lower quartile, $w_{0.25}$ , which is related to $λ$ through, say, $w_{0.25} = h^{*} (λ)$ . Then, from (3.7), we have:

Q = μ h^{*} (λ) \equiv h (λ),

(3.9)

where

Q

is the lower quartile of

Y

and

h (.)

, like

h^{*} (.)

, is a monotonic increasing function of

λ

The expert will be asked to assess three quartiles of her prior distribution for $Q$ . Then, from the monotonicity of $h (.)$ , these quartiles can be transformed into the corresponding quartiles of $λ$ . To illustrate that quartiles of $Q$ can be elicited through questions that an expert can understand and meaningfully answer, suppose that $Y$ (the variable of interest that has the gamma distribution) is the period of time that a patient with some medical disorder may stay in hospital. Then the expert will be asked to consider the length of time that a hypothetical patient, John, will spend in hospital. She is told,

John has this disorder and will spend a time in hospital. Suppose he is fortunate and does not spend as long as most people in hospital. Specifically, suppose exactly 25% of patients with John's disorder spend a shorter time in hospital than John. Give your median assessment for the length of time, $Q$ , that John spends in hospital. Now give your lower and upper quartiles for this length of time.

The median and quartiles divide the range of $Q$ into four time intervals. One of these intervals contains the time that John actually spends in hospital and the expert should feel that each of these intervals is equally likely to be that interval. If the expert thinks, say, that the third interval is more likely than other intervals to contain the time John spends in hospital, then the median and quartiles should be adjusted by the expert to make that interval smaller.

Let $L_{Q}$ , $M_{Q}$ and $U_{Q}$ denote the quartiles of $Q$ that are assessed by the expert, where the median $M_{Q}$ is a point estimate of $Q$ , and $(U_{Q} - L_{Q})$ is its interquartile range. Then, under the monotonicity of $h (.)$ in (3.9), the three quartiles of $λ | μ$ , say $L_{λ}$ , $M_{λ}$ and $U_{λ}$ , can be obtained from them:

L_{λ} = h^{- 1} (L_{Q}), M_{λ} = h^{- 1} (M_{Q}) and U_{λ} = h^{- 1} (U_{Q}),

(3.10)

where

h^{- 1} (.)

can be implemented by numerically inverting the incomplete gamma function

F (w, λ, λ)

. By assumption, the prior distribution of

λ

given

μ

is the lognormal distribution,

f (λ | μ) = \frac{1}{λ b \sqrt{2 π}} \exp \{\frac{- {[\ln (λ) - a]}^{2}}{2 b^{2}}\}, λ, b > 0 .

(3.11)

Properties of the normal distribution are used to estimate the hyperparameters

a

and

b

from the transformed assessments

L_{λ}

M_{λ}

and

U_{λ}

. From (3.11),

\ln (λ) | μ \sim N (a, b^{2}),

(3.12)

so, using the fact that 1.349 is the interquartile range of a standard normal distribution, we put:

a = \ln (M_{λ}), b = [\ln (U_{λ}) - \ln (L_{λ})] / 1.349 .

(3.13)

4 Implementation of the elicitation methods

The elicitation methods proposed in Sections 2 and 3 have each been implemented as an interactive graphical procedure. We describe these implementations in turn.

4.1 Quantifying opinion about the error variance in normal linear models

So as to frame questions in a way that is meaningful to the expert, first the expert is asked whether the normal linear model relates to an experimental setting, observations on people, or observations on items. In the context of an experiment, a set of values chosen for the predictor variables are referred to as a ‘design point’, so ‘two responses at the same design point’ form each pair of observations. With observations on people (items), the observations are on ‘two people (items) whose covariate values give them identical characteristics (features).’

In a dialogue box, the expert is then asked to assume that two independent experiments have been conducted at the same design point. She assesses her median value of the absolute difference between the observed responses in these two virtual experiments. This assessed median is $q_{0}$ .

Her remaining assessments are conditional assessments after being shown hypothetical data sets. The choice of the conditioning values in these data sets is an important issue. Garthwaite and Dickey (1988) note that hypothetical data should (a) be moderately different from the expert's initial beliefs, so that the data change the expert's beliefs by a measurable amount, but (b) should not be so different that the expert dismisses the data as being false and misleading. The hypothetical data set used by Garthwaite and Dickey (1988) consists of just one hypothetical difference, which they set at $q_{0} / 2$ to try and meet these desiderata. Following Garthwaite and Dickey (1988) we take this as our first data set. This hypothetical datum is presented to the expert visually, using a diagram similar to that in Figure 2. In the figure, the value of the hypothetical datum is 10.0 ( $\times 10^{- 2}$ ) and this is marked by the upward pointing arrow, the expert's original estimate of the median difference ( $q_{0}$ ) is marked by the tall vertical line at 20.0 ( $\times 10^{- 2}$ ). The short lines below the horizontal axis mark the interval within which the expert's conditional median assessment of the difference must lie for statistical coherence. This interval was determined from equation (2.20) with $min (ν)$ set equal to 1 and the initial assessment $q_{0}$ . The expert makes her conditional assessment of the median difference by clicking the mouse-pointer on the horizontal axis within this interval. Let $q_{1}$ denote this assessment. Using equations (2.16) and (2.19), $q_{0}$ and $q_{1}$ yield the first estimates of $ν$ and $w$ , which we denote by $ν_{1}$ and $w_{1}$ .

Figure 2

Source: Authors' own.

Our other hypothetical data sets contain more items. This gives these data sets greater weight and means that they should have greater effect on the expert's opinions, without the expert finding them unbelievable. In our past experience of using the method of Garthwaite and Dickey (1988), occasionally an expert has said that a single datum is too inconsequential to have any impact on his beliefs, which is also a problem that larger data sets should resolve. Kadane et al. (1980) present an expert with a sequence of hypothetical data sets and suggest that an expert should not be asked to forget any hypothetical data after it has been presented. They argue that ‘…asking the experimenter to forget would impose too great a psychological burden’. (Kadane et al., 1980, p. 849). In this spirit, we make our hypothetical data sets steadily bigger, so that each contains all the hypothetical data in its predecessors.

In our implementation we generate five hypothetical data sets. The $j$ th set contains $2^{j - 1}$ data ( $j = 1, \dots, 5$ ). A more flexible implementation would allow the expert to choose the number of data sets and the number of data in each, but it seems unlikely that a subject-matter expert would have knowledge relevant to making these choices. The first $2^{j - 2}$ of these data ( $z_{1}, \dots, z_{2^{j - 2}}$ ) comes from the previous set of hypothetical data. The remaining items ( $z_{2^{j - 2} + 1}, \dots, z_{2^{j - 1}}$ ) are obtained by generating random values from a normal distribution with zero mean and a variance of ${(q_{0} / 1.35)}^{2}$ . The $z_{i}$ are set equal to the absolute values of the generated values. As 1.35 is the interquartile range of a standard normal distribution, the median of the $z_{i}$ should be about $q_{0} / 2$ . However, the hypothetical data must be moderately different from the expert's initial beliefs, so as to give a measurable change in her opinions. In our implementation, for $j > 1$ , $z_{2^{j - 2} + 1}, \dots, z_{2^{j - 1}}$ are resampled from the normal distribution if $\sum_{i = 1}^{k} z_{i}^{2} / k > {(3 q_{0} / 4)}^{2}$ .

Figure 3

Source: Authors' own.

After a suitable hypothetical data set has been generated, it is presented to the expert using the graphical interface. Figure 3 is an example in which a set of 16 hypothetical data is displayed (the arrows pointing up). Their median is the arrow pointing down and is a summary of the data that the expert may find helpful. The expert's original median assessment, $q_{0}$ , is the rightmost (black) tall vertical line. The middle (green) vertical line is the expert's median assessment conditional on the previous hypothetical data set. Taking account of all the hypothetical data, the expert gives her conditional assessment of the median absolute difference by clicking the mouse-pointer in the interval bounded by the short lines below the horizontal axis. One strategy for making this assessment is to consider the original median assessment and the median of the hypothetical data, and then choose a point between them that reflects their relative importance. For example, if the original median assessment is thought to be a better estimate than the median of the data, then the new median assessment should be nearer to the original median assessment than the median of the data. The leftmost (blue) vertical line is the expert's new assessment. We let $q_{j}$ denote the median assessment conditional on the $j$ th set of hypothetical data.

In conjunction with $q_{0}$ , each $q_{j}$ yields estimates of $ν$ and $w$ . Let $ν_{j}$ and $w_{j}$ denote the estimates for the $j$ th data set ( $j = 1, \dots, 5)$ . The estimates are displayed in a table. In principle, the estimates of $ν$ should all be very similar, as should the estimates of $w$ . The expert is invited to revise her median assessment for any hypothetical data set (or sets) she wishes, and should take this option if some estimates appear out of line. When the estimates are acceptable to the expert, the geometric mean of $ν_{1}, \dots, ν_{5}$ , is taken as the value of $ν$ in the prior distribution. Using the assessed value $q_{0}$ , the corresponding value of $w$ is determined from equation (2.16) and taken as the prior value of $w$ .

4.2 Quantifying opinion about the scale parameter in gamma GLMs

In the context of a GLM, it is assumed that one design point has been specified as a reference design point and $Y$ is the response at that point. In other contexts, $Y$ is simply a random variable that follows a gamma distribution. Assessments of the regression coefficients in a GLM may provide the prior mean $μ = E (Y)$ ; otherwise the software asks the expert to specify it, via a dialogue box.

Next, the expert is asked to assess a median value $M_{Q}$ for the lower quartile $Q$ , conditional on the value of $μ$ , giving her assessment in an upper panel of the interactive software (see Figure 4). This panel also shows the allowable range of $Q$ . Outside the permitted range, the pdf curve for $Q$ has undesirable behaviour, as will be discussed below. Once the value of $M_{Q}$ is assessed, the software draws the pdf curve of $Y gamma (M_{Q}, M_{Q} / μ_{0})$ . The expert has the option of changing this value by a mouse-click or a mouse-drag along the allowable range until the corresponding gamma pdf curve becomes the best representation of her opinion. An example of questions that may be addressed to the expert at this stage has been given earlier. See also the example in Section 5.2.

The expert is then asked to assess her uncertainty about $M_{Q}$ by giving her initial assessments $L_{Q}^{0}$ and $U_{Q}^{0}$ for the two quartiles of $Q$ . This is done by mouse-clicking on the lower panel of the interactive graph. Once these two quartiles have been assessed, the software draws the pdf curve of $Q$ on the lower panel. Again, the expert is able to revise these quartile assessments by dragging or clicking on the lower panel.

The initially assessed quartiles $L_{Q}^{0}$ , $M_{Q}^{0}$ and $U_{Q}^{0}$ are used in the RHS of the equation in (3.10) to get the three quartiles $L_{λ}^{0}$ , $M_{λ}$ and $U_{λ}^{0}$ of the parameter $λ$ , respectively. The inversion of $h (.)$ is done by the software through a simple search procedure. As in (3.13), these quartiles are used to compute the two hyperparameters $a$ and $b$ of the assumed lognormal distribution of $λ$ . Using $a$ and $b$ , the mean value of $λ$ , say $μ_{λ}$ , is computed from the lognormal distribution as $μ_{λ} = exp (a + b^{2} / 2)$ . Then $μ_{λ}$ is used with the assessed mean value $μ_{0}$ to draw the pdf graph of the gamma distribution, gamma $(μ_{λ}, μ_{λ} / μ_{0})$ . This is used to update the gamma pdf curve on the upper panel of the interactive software. An example of the main software graph panels presented to the expert is shown in Figure 4. It is based on the expert's assessments in the example in Section 5.2. The thick black line on the upper panel represents the mean value, $μ_{0} = 49$ , and the allowable ranges in both panels are given by the light-blue and red (shaded) areas.

Figure 4

Source: Authors' own.

Under our prior model for $λ$ , the prior pdf of $Q$ will have a local minimum if there is a high probability that $λ$ is small. This is illustrated in Figure 5 where Pr $(λ < 0.5) = 0.07$ . We take the view that such a bimodal distribution will not be a reasonable reflection on an expert's opinion. For this reason, we suggest that the quartile assessments of $Q$ should be within the light-blue region in Figure 4, when Pr $(λ < 0.5) = 0.005$ , and require them to be within the red or light-blue regions, when Pr $(λ < 0.5) = 0.05$ . These choices do not prevent bimodality, but any second mode near $Q = 0$ will be small. The choices are somewhat arbitrary and other choices could be made without otherwise affecting the elicitation method, but some restriction seems necessary. If quartile assessments of $Q$ need to be outside the allowable range in order to represent the expert's opinion, then a lognormal distribution seems an inappropriate choice of prior model, and the elicitation method proposed here should not be used.

Figure 5

Source: Authors' own.

For statistical coherence of the assumed normal distribution of ln $(λ)$ , the two normal quartiles ln $(L_{λ}^{0})$ and ln $(U_{λ}^{0})$ should be symmetrical around the normal mean, $a = \ln (M_{λ})$ , so some reconciliation of assessments is usually needed. We assume that the expert represents her opinion more accurately when assessing the median value than when assessing the other two quartiles, so we treat her original and transformed medians, $M_{Q}$ and $M_{λ}$ , respectively, as being correct. Then we suggest two coherent sets of quartiles $L_{Q}$ , $U_{Q}$ and $L_{λ}$ , $U_{λ}$ to replace the initial assessments $L_{Q}^{0}$ , $U_{Q}^{0}$ and $L_{λ}^{0}$ , $U_{λ}^{0}$ , respectively, as follows. First, $L_{λ}$ , $U_{λ}$ are computed as the actual first and third quartiles, respectively, of a lognormal distribution with the two elicited parameters $a$ and $b$ . Then $L_{Q}$ , $U_{Q}$ are computed from $L_{λ}$ , $U_{λ}$ , respectively, using (3.9). This procedure is also applied whenever the expert separately revises the value of the lower quartile $L_{Q}$ or the upper quartile $U_{Q}$ .

The top group of tabulated values in the right-hand side panel of Figure 4 gives the values of the three suggested coherent quartiles $L_{Q}$ , $M_{Q}$ and $U_{Q}$ . These quartiles are also drawn as the three vertical (blue) lines in the upper and lower panels of Figure 4. The middle group of tabulated values gives the three quartiles of $λ$ : $L_{λ}$ , $M_{λ}$ and $U_{λ}$ . The elicited values of $a$ and $b$ are shown as the lowest group of tabulated values in the same panel.

The lower graph in Figure 4 represents the elicited distribution of the lower quartile $Q$ , with the three vertical (blue) lines representing $L_{Q}$ , $M_{Q}$ and $U_{Q}$ . The graph is intended to help the expert check that the distribution is a reasonable representation of her prior knowledge of $Q$ and she may adjust her assessments in the light of this distribution if she wishes. Although we do not assume any specific family of distributions for $Q$ , the pdf graph is drawn using pointwise numerical derivatives of the cdf of $Q$ . This cdf is obtained from (3.9) using a sufficiently large number of points to smoothly cover the whole range of $Q$ .

As further feedback, the expert should record her assessments of the median, lower and upper quartile of Q. Then she should change the median in the upper graph so that it equals her lower quartile assessment. Both tails of the pdf for $Y$ will become thicker and the range of plausible values for Y will increase. The expert should check that, in her opinion, the pdf for $Y$ looks a bit wide but believable. She should then change the median in the upper graph so that it equals her upper quartile assessment. Now both tails of the pdf for $Y$ will become thinner and the range of plausible values for $Y$ will decrease. The expert should find that the pdf for $Y$ looks somewhat narrow but, again, is believable.

This feedback may indicate that the assessed quartiles do not correspond to the expert's opinions, in which case she should revise them. Some users of the software have found it easier to relate to this feedback, rather than Q, and have formulated their initial quartile assessments by examining the pdf of $Y$ as they varied $Q$ in the upper graph. They found the bounds on $Q$ for which $Y$ had a plausible pdf, and used these to guide their quartile assessments of $Q$ in the lower graph.

When the expert is happy with the quartile values and the corresponding pdf graphs as a reasonable representation of her opinions, she clicks ‘Done’ and obtains the two corresponding hyperparameters $a$ and $b$ of the lognormal distribution as the output of her assessments.

5 Examples

The first example uses the first elicitation method to quantify opinion about the variance of a normal distribution. While that elicitation method requires equations (2.3) and (2.5) to hold, the example illustrates that the method's usefulness is not restricted to quantifying opinion about experimental error. Application of the second elicitation method is illustrated in Section 5.2, where it is used to quantify opinion about the shape parameter of a gamma distribution.

5.1 Measurements along Pine Island Glacier

Pine Island Glacier is a large ice-stream located on the West Antarctic Ice Sheet in Antarctica. It is the fastest shrinking glacier on Earth and, in common with other glaciers that drain into the Amundsen Sea, its progress is accelerating and it is thinning rapidly. These observations have been attributed to the regional oceanography whereby the heat contained in deep water acts to melt the underside of an ice shelf, making it weaker and more likely to crack and fall into the sea.

In the present example, the expert is a polar oceanographer who has done field work on Pine Island Glacier and is in the early stages of planning further work there. He intends to measure water characteristics (temperature, salinity and pressure) at regularly spaced points on a 40 km transect along an ice shelf, having done similar work along the boundary between the sea and the Breidamerkurjökul glacier in Iceland. The spacing between the points where data are collected should be kept as large as possible while meeting scientific needs—polar expeditions must meet multiple objectives, so there is time pressure on all experiments. Here the expert quantifies his opinion about the variance of water temperature along the planned transect; this variance influences the appropriate choice of spacing between data-collection points.

Covariates that affect water temperature include depth, meltwater content, time of year and recent weather. The expert assumed that these covariates had fixed values when quantifying his opinion and, in particular, focused on temperatures at a depth of 60 meters in summer, when there had been no recent surge in meltwater, and no recent heavy precipitation. The difference in temperature between two points depends on the distance between them and it was decided to consider points that are two kilometers apart. Thus we consider pairs of points that are two kilometers apart and we let $Z$ denote the difference in their temperatures. We assume that $Z | σ_{ε}^{2} \sim N (0, 2 σ_{ε}^{2})$ , as in equation (2.5), and further assume that the prior distribution for $σ_{ε}^{2}$ is $σ_{ε}^{2} \sim inverse gamma (ν / 2, ν w / 2)$ , as in equation (2.3).

Using the interactive software, the expert gave $0 . 2^{\circ}$ C as his assessed median for $| Z |$ . He was then asked to suppose that the observed difference in one pair of points was $0 . 1^{\circ}$ C. Conditional on this information, the expert gave $0 . 183^{\circ}$ C as his new median for $| Z |$ . Figure 2 is a screen-shot that shows the expert's original assessment of $0 . 2^{\circ}$ C (given in hundredths of a degree and so displayed as 20), the hypothetical observed difference of $0 . 1^{\circ}$ C, and the revised median assessment. Hypothetical data for 1 2, 4 and 8 further pairs of points were added in turn, after each of which the expert revised his median for $| Z |$ in response to the additional information.

A screen-shot for the last of these assessments is shown in Figure 3.

A table was then displayed that showed the expert the values of the degrees of freedom parameter that each of his assessments had implied. The value from his fourth assessment was above 20—out of line with his other assessments. The figure on which he had given his fourth assessment was re-displayed. One hypothetical datum had a large value of $0 . 25^{\circ}$ C and the expert thought it had influenced his assessment more than it should. He revised that assessment substantially and also made minor adjustment to two other of his assessments. After these adjustments, 5.1, 6.3, 6.0, 8.8 and 9.1 were the five estimates of the degrees of freedom parameter ( $ν$ ) that the expert's assessments had each determined. Their geometric mean, 6.9, was taken as the reconciled estimate of $ν$ and equation (2.16) then gave 0.0395 as the value of $w$ .

5.2 Water-table depth in Wiltshire, UK

Ecologists are often interested in determining how environmental factors influence plant growth, competition and diversity, not least in future scenarios. Soil water availability is a major environmental driver in this respect, with plants demonstrating a high degree of sensitivity to it (see for example Araya et al., 2011). Soil water availability can be measured in a number of ways, the most common one being water-table depth (WTD). It is highly anticipated that any change in the availability of soil water (for example due to global warming) will have significant impact on water-dependent ecosystems such as wetlands.

A team of plant ecologists have recently been studying the levels of the WTD in wet grassland during the growing season (March–September) in Wiltshire, UK. Although data of the current levels of the WTD are now available for this region, experts’ opinions for future levels in 2050 need to be quantified. In this example, a plant ecologist (Dr. Yoseph Araya, University of London, UK) used our proposed method for the gamma distribution to quantify his opinions about the WTD in Wiltshire, UK, during the growing season of 2050. WTD in Wiltshire is a continuous positive valued random variable whose values in Wiltshire currently range from zero, at flooding time, to a maximum that may reach 80 cm at the peak of summer. Over the growing season of 2010, the shape of the WTD distribution was positively skewed with a mean of approximately 45 cm. WTD is often assumed to have a gamma distribution; see for example Yeh and Eltahir (2005).

Here we assume that the WTD in Wiltshire during the growing season of 2050 follows a two-parameter gamma distribution as in (3.2). The WTD is affected by climate variables, particularly rainfall, temperature and wind, whose future values, in turn, depend on the quantities of greenhouse gases that are emitted in years to come. The expert gave assessments of WTD for one ‘Representative Climate Forcing’, called RCP6. (Under RCP6, greenhouse gas emissions continue to rise without strong mitigation.) He assessed the mean value of WTD, $μ_{0}$ , as 49 cm. Then he used the software to elicit the hyperparameters of a lognormal prior distribution for the gamma shape parameter $λ$ . The first question addressed to the expert was:

Suppose that there is a time point in the growing season of 2050 where the depth of water-table in Wiltshire records a value, say Q, that is greater than exactly 25% of all readings of WTD during this season. Given that the true mean value of WTD is $μ_{0} = 49$ cm, please give your median assessment of $Q$ .

Responding to this question, the expert assessed his median $M_{Q}$ on the upper interactive panel as 22.8 cm, see Figure 4.

He was then asked to assess his lower and upper quartile of $Q$ on the lower panel. He gave $L_{Q}^{0} = 19.07$ cm and $U_{Q}^{0} = 25$ cm. The software then suggested the two coherent quartile values $L_{Q} = 19.75$ cm and $U_{Q} = 25.7$ cm, which were reviewed and adjusted again by the expert to $L_{Q} = 19.98$ cm and $U_{Q} = 25.5$ cm. These are shown in Figure 4, where the two corresponding pdf curves of the WTD and $Q$ are also drawn. The expert accepted these two quartile assessments and was happy with the presented two pdf curves as a reasonable representation of his opinion. The median of $λ$ is 1.89 and its lower and upper quartiles are 1.52 and 2.35, respectively. This gave an elicited lognormal distribution of $λ$ with hyperparameter values $a = 0.637$ and $b = 0.323$ .

6 Concluding comments

In this article, two elicitation methods have been proposed. One was designed for assessing a conjugate inverse chi-squared prior distribution for the error variance in normal models, although its usefulness is not limited to that application. The method quantifies an expert's opinions through assessments of a median and conditional medians of the absolute difference between two observations of the response variable that have the same expected value. A number of sets of hypothetical data are used in order to obtain several estimates of the hyperparameter that is most difficult to assess, namely, the degrees of freedom parameter of the inverse chi-squared distribution. Reconciliation of these estimates, using the geometric mean, yields an overall estimate of the number of degrees of freedom. The second hyperparameter of the inverse chi-squared prior distribution is also determined from the same assessments.

The other method elicits a lognormal prior distribution for the scale parameter of a gamma GLM, or the shape parameter of any gamma distribution. The method depends only on quantifying an expert's opinion about the lower quartile of a gamma distributed random variable. This lower quartile is itself a random variable, for which the expert assesses a median value as a point estimate, and an interquartile range. The lower quartile is a monotonic increasing function of the scale parameter so the expert's assessments can be transformed to quartiles of the lognormal distribution, and hence to the hyperparameters of the lognormal distribution. Examples of questions that can be addressed to the expert were given.

The use of interactive graphical software greatly facilitates the tasks that the expert must perform in both these elicitation methods. Computer programmes that implement the methods form part of Prior Elicitation Graphical Software (PEGS) that is freely available and can be downloaded from http://statistics.open.ac.uk/elicitation. PEGS-Normal can be used as an add-on to any other elicitation software for normal linear models. PEGS-Gamma is a stand-alone version that can be used to quantify opinion about the shape parameter of a gamma distribution with a known mean. An important component of PEGS is PEGS-GLM, which is interactive graphical software for quantifying opinion about a generalized linear or piecewise-linear model (Garthwaite et al., 2013). PEGS-GLM now incorporates versions of PEGS-Normal and PEGS-Gamma. Thus, quantifying opinion about error variance can be a natural part of the elicitation process for ordinary linear regression models (as it should be), and assessing the scale parameter $λ$ can be integral to the process of quantifying opinion about a gamma GLM. If a set of data is available, the software offers the option of combining those data with the elicited prior distribution and it will automatically produce a WinBUGS model.

Expert opinion is most commonly quantified when data are absent. For example Garthwaite et al. (2008) report work where PEGS was used in an NHS study aimed at improving the provision of treatment for bowel cancer in England and Wales. In the complex model that was constructed, there were a small number of parameters for which no data were available and the judgements of clinical experts were quantified to fill these gaps. At the same time, when sample data are available, an important question is whether quantifying expert opinion though our methods is still worthwhile. That is, will a better posterior distribution be obtained through using an expert's prior distribution rather than a non-informative prior distribution. Experiments are needed to examine this question and we hope the availability of software will encourage such experiments.

Footnotes

Acknowledgements

The work reported here was supported by a studentship from The Open University, UK. We are very grateful to Dr. Mark Brandon (a polar oceanographer and a reader in the Department of Environment, Earth and Ecosystems, The Open University, UK) whose opinion was quantified in the polar-ice example, and to Dr. Yoseph Araya (a plant ecologist and a lecturer in Environmental Geography at Birkbeck College, University of London, UK) whose opinion was quantified in the water-table depth example. We must also thank a referee for helpful comments that greatly improved this article.

References

Al-Awadhi

(1997). Elicitation of prior distribution for a multivariate normal distribution . PhD thesis, University of Aberdeen, UK.

Al-Awadhi

Garthwaite

(1998). An elicitation method for multivariate normal distributions. Communications in Statistics-Theory and Methods, 27, 1123–42.

Al-Awadhi

Garthwaite

(2001). Prior distribution assessment for a multivariate normal distribution: An experimental study. Journal of Applied Statistics , 28, 5–23.

Alpert

Raiffa

(1969). A progress report on the training of probability assessors. In Kahneman

, Solvic

, Tversky

, eds. Judgement under uncertainty: Heuristics and biases ,. Cambridge: Cambridge University Press, 294–305.

Araya

Silvertown

Gowing

McConway

Linder

Midgley

(2011). A fundamental, eco-hydrological basis for niche segregation in plant communities. New Phytologist , 189, 253–58.

Bedrick

Christensen

Johnson

(1996). A new perspective on priors for generalized linear models. Journal of the American Statistical Association , 91, 1450–60.

Chen

M-H

Ibrahim

(2003). Conjugate priors for generalized linear models. Statistica Sinica , 13, 461–76.

Garthwaite

Dickey

(1988). Quantifying expert opinion in linear regression problems. Journal of the Royal Statistical Society, Series B , 50, 462–74.

Garthwaite

Dickey

(1992). Elicitation of prior distributions for variable selection problems in regression. Annals of Statistics , 20, 1697–1719.

10.

Garthwaite

Kadane

O'Hagan

(2005). Statistical methods for eliciting probability distributions. Journal of the American Statistical Association , 100, 680–701.

11.

Garthwaite

Chilcott

Jenkinson

Tappenden

(2008). Use of expert knowledge in evaluating costs and benefits of alternative service provisions: A case study. International Journal of Technology Assessment in Health Care , 24, 350–57.

12.

Garthwaite

Al-Awadhi

Elfadaly

Jenkinson

(2013). Prior distribution elicitation for generalized linear and piecewise-linear models. Journal of Applied Statistics , 40, 59–75.

13.

Hogarth

(1975). Cognitive processes and the assessment of subjective probability distributions. Journal of the American Statistical Association , 70, 271–89.

14.

Ibrahim

Laud

(1991). On Bayesian analysis of generalized linear models using Jeffreys's prior. Journal of the American Statistical Association , 86, 981–86.

15.

Kadane

Wolfson

(1998). Experiences in elicitation. The Statistician , 47, 3–19.

16.

Kadane

Dickey

Winkler

Smith

Peters

(1980). Interactive elicitation of opinion for a normal linear model. Journal of the American Statistical Association , 75, 845–54.

17.

Miller

(1980). Bayesian analysis of the two-parameter gamma distribution. Technometrics , 22, 65–69.

18.

O'Hagan

Buck

Daneshkhah

Eiser

Garthwaite

Jenkinson

Oakley

Rakow

(2006). Uncertain judgements: Eliciting expert probabilities . Chichester: John Wiley.

19.

Pratt

Raiffa

Schalifer

(1995). Introduction to statistical decision theory . London: The MIT Press.

20.

Sweeting

(1981). Scale parameters: a Bayesian treatment. Journal of the Royal Statistical Society, Series B , 43, 333–38.

21.

West

(1985). Generalized linear models: Scale parameters, outlier accommodation and prior distributions. In Bernardo

DeGroot

Lindley

Smith

AFM

eds. Bayesian Statistics 2 . North-Holland: Elsevier, 531–58.

22.

West

Harrison

Migon

(1985). Dynamic generalized linear models and Bayesian forecasting. Journal of the American Statistical Association , 80, 73–83.

23.

Yeh

PJ-F

Eltahir

EAB

(2005). Representation of water table dynamics in a land surface scheme. Part II: Subgrid variability. American Meteorological Society , 18, 1881–1901.