Forecasts for the Attainment of Major Research Milestones in Parkinson’s Disease

INTRODUCTION

Parkinson’s disease (PD) research and development is progressing along many fronts, including precision medicine, experimental new therapies, and body worn sensors for diagnosis and monitoring. Though major advances in disease modifying therapy have yet to arrive, novel therapeutic strategies like deep brain stimulation or better levodopa infusion have improved the symptomatic management of PD.

Because PD has a protracted course, patients and physicians are often keenly interested in expert impressions of when new treatments and management strategies are expected to emerge and be trialed. Realistic projections of the attainment of major research milestones can help physicians and patients calibrate their expectations. They can also help with research and health system planning. For example, knowing when novel interventions are likely to mature can help healthcare systems determine when to build capacity for their deployment. Knowing which therapeutic strategies are likely to mature first can help funding bodies set priorities or issue calls that are relevant to that milestone.

Forecasting scientific breakthroughs is often not amenable to computational approaches, in part due to the diversity of variables that drive scientific progress [1]. In what follows, we used a “wisdom of the crowd” approach for aggregating expert forecasts about the timing of eleven major research milestones for the future management and diagnosis of PD. Such approaches have been shown to offer greater predictive accuracy than individual judgments by reducing the effect of random variation between expert judgments, while pooling the judgments of individuals who have widely varying knowledge [2–4]. Previously, wisdom of crowds approaches have been used in medicine to improve predictive accuracy in areas like prognosis [5], diagnosis [4, 6], and emergence of diseases [7–9]. Our approach used a method known as coherence weighting, where the aggregation of forecasts is weighted based on the extent to which the forecasts are probabilistically coherent. This approach has demonstrated effectiveness in improving the accuracy of lay forecasts in other domains, where those who provided more coherent forecasts also provided more accurate ones [10].

METHODS

PD experts were recruited from two sources: 1) the Michael J. Fox Foundation for Parkinson’s Research database and 2) an independent sample constructed by identifying doctors specializing in PD at the 25 top ranked neurology departments in the US (as rated by US News and World Report), the top 3 largest hospital systems in Canada, and by identifying corresponding authors of articles on PD from the last 5 years in BMJ, Lancet, PLoS Med, NEJM, JAMA, Annals of Neurology, JAMA Neurology, Lancet Neurology, Neurology and Movement Disorders. Experts were solicited for survey participation three times.

Our survey sought forecasts for 11 major milestones in PD research (this was the largest number of forecast queries that could fit in a 30-min survey). Milestones were generated by our three co-authors with PD expertise (RB, AL, TS) based on their perceptions of what would constitute a significant improvement on the current state of the field. We then helped select milestones that were objectively verifiable, diverse, and thought to be of interest to the patient community. Briefly, each PD expert co-author generated 10–15 potential milestones, and then rated the milestones created by the other co-authors on their importance and verifiability. We selected the highest rated milestones while ensuring that some selected milestones were likely to occur earlier while others were likely to occur later and covered a range of event types including FDA approval and trial launch. Particular milestone events for new treatments (e.g., FDA approval vs. launch of a trial vs. clinical practice guideline recommendations) were selected in order to enable capture of near term as well as late-occurring advances. The survey was created on Qualtrics. Experts provided the probability of milestone attainment in three time bins spanning the next 10 years (within the next 2 years, within the next 2–6 years and within the next 6–10 years), and separately the probability of milestone nonattainment within 10 years. In order to assess probabilistic coherence for the purpose of coherence weighting, we allowed experts to enter probabilities that did not sum to 100% across time bins. A graphical depiction of our survey process is provided in Fig. 1. Milestone questions are provided in Table 1. Milestones were presented in random order.

OPEN IN VIEWER

Fig. 1

A simplified depiction of the interface and flow of our study.

Our survey contained five additional components. First, for milestones entailing launch or completion of a trial, we asked experts to predict the probability the trial would be positive on its primary endpoint. Second, we collected forecasts of whether there would be an FDA approval within the next 10 years for therapies in the following areas: gene therapy, repurposed small molecule therapy, novel small molecule therapy, stem cell therapy or a therapy not falling within the preceding categories. Note that, unlike the 11 milestone questions, these questions were not worded with sufficient precision to enable forecast verification. We additionally asked participants to rate their familiarity with gene therapy, precision medicine, alpha-synuclein targeted treatment and deep brain stimulation on a seven-point Likert scale from extremely unfamiliar to extremely familiar. Third, we asked participants to rate the same four kinds of treatments on their clinical promise on a nine-point Likert scale from minimal promise to maximal promise. None of the above questions were asked using a coherence weighting format. Last, we collected the following demographic items from experts: sex, age, education and number of clinical trials participated in. H-indices of all participants were obtained using Scopus.

Our survey received approval by McGill IRB; experts provided consent online.

Aggregated forecasts were produced in two stages. First, the raw forecasts were all made coherent by ensuring that the forecasts for each time bin jointly summed to 100%. We did this by taking each set of incoherent forecasts, and finding the coherence set of forecasts that was closest to them as measured by the Euclidean distance. For example, if someone provided forecasts of 30%, 40%, 40%, and 30% for the four time bins, their forecasts would be incoherent because they sum to more than 100% percent. These forecasts would be coherentized by finding the closest set of coherent forecasts, in this case 20%, 30%, 30%, 20%. Once the forecasts were coherentized, we averaged them together to produce an aggregate forecast. This average was weighted by how incoherent each forecast was originally, that is how far from 100% the sum was, with lower weight given to more incoherent forecasts [10]. Post-hoc, we tested whether the forecasts for North American experts were different from the forecasts for non-North American experts for the two milestones dealing with FDA approvals using a Kolmogorov-Smirnov test for equality of distributions.

As an exploratory analysis we tested for differences in the deviation of forecasts from the coherence weighted mean as a function of the demographic characteristics of experts. Under the assumption that the coherence weighted mean is the best obtainable prediction for each milestone, this dependent variable serves as a proxy for forecast accuracy. To simplify the analysis, we first dichotomized forecasts by summing each individual’s forecasts for each milestone for the first three time bins to create a forecast of the probability of the milestone occurring in the next 10 years. We performed a similar dichotomization for the coherence weighted means, and then took the absolute value of their difference. We subjected this dependent variable to a linear regression on age, H-index, number of clinical trials participated in, an indicator variable for gender, a set of indicator variables for degrees held and incoherence, along with milestone controls. Based on a preliminary exploration of the data, we also developed a model where we used a log transformation on H-index and number of trials participated in to deal with some of the skew in these variables. When we compared models, we found that the fit was not notably different (R² = 0.029 versus R² = 0.029), so we report the results for the untransformed model only.

RESULTS

Forecasts on milestone attainment

The median incoherence was 14.5% (26.4% standard deviation, range 0% –127%), meaning that forecasts of half of experts summed to either less than 85.5% or more than 114.5%. Forecasts about the timing for milestone attainment are depicted in Fig. 2 (the distribution of individual forecasts for each milestone can be found in the Supplementary Material). Three of the four milestones judged to be least likely to be attained in the next 10 years concerned disease modifying therapies, while those judged most likely to occur are largely refinements in existing therapies. The aggregated predictions of experts for trials being positive on their primary endpoint was 44.5% for the trial specifying eligibility based on GBA mutational status trial (Precision Medicine Therapy), 39.4% for the pluripotent stem cell trial (Cell Therapy), 47.0% for the trial integrating an alpha synuclein imaging agent (Imaging), and 40.5% for the non-cholinesterase inhibiting drug trial (Treatment for PD-MCI). Table 2 contains the mean and standard deviation for each time bin for the FDA related milestones for North American and non-North American experts, as well as the results of a Kolmogorov-Smirnov test for equality of distribution comparing the distributions of predictions. None of the tests were significant, indicating there was no difference in the distribution of these predictions between North American and non-North American samples.

OPEN IN VIEWER

Fig. 2

Coherence weighted mean forecasts for each milestone, arranged in descending order of probability of nonattainment within 10 years. Each bin contains the coherence weighted mean predicted probability of the milestone occurring in that bin.

OPEN IN VIEWER

Table 2

Means and standard deviations of the forecasts for each time bin for the two milestones dealing with the FDA for the North American and non-North American experts, as well as the p-values for a two sided Komogorov-Smirnov test of equality of distributions for the two samples of forecasts for each time bin

Milestone	Time Bin	North American	Non-North American	p-value
		Mean (SD)	Mean (SD)
Gene Therapy	Next 2 years	6.5% (10.9%)	8.0% (9.3%)	0.057
	2–6 years	18.9% (15.7%)	20.9% (14.7%)	0.45
	6–10 years	31.8% (19.2%)	29.9% (19.0%)	0.71
	Not in next 10 years	42.9% (27.4%)	41.1% (28.1%)	0.78
Closed Loop Device	Next 2 years	21.9% (24.0%)	17.0% (20.6%)	0.23
	2–6 years	31.8% (18.9%)	31.2% (17.7%)	0.77
	6–10 years	28.4% (22.8%)	31.2% (20.3%)	0.12
	Not in next 10 years	17.9% (22.1%)	20.6% (22.2%)	0.20

DISCUSSION

We synthesized a large and diverse sample of expert opinions about the timing of attainment of major research milestones in PD. With the exception of precision medicine approaches, experts believed that advances in new treatment modalities are likely still some years away, viewing breakthroughs in disease modifying treatments as having a 30–40% chance of not occurring at all within ten years. Experts were most pessimistic about FDA approval of PD monogenic gene therapy relative to other disease modifying therapies, though this may have more to do with the milestone referring to an FDA approval as opposed to a clinical trial. When asked about broad treatment categories rather than more specific milestones, experts rated the probability of the FDA approving a new treatment within 10 years as less than 50% for all categories. Doubts about near term breakthroughs in disease modification were echoed by forecasts that primary endpoints in trials are more likely to be non-positive than positive, and the fact that experts generally predicted a less than 50% probability of an FDA approval of a new PD therapy across five different treatment platforms. However, on average experts rated the clinical promise of all treatment categories we asked about as above the mid point of our scale, indicating that they believe these lines of research are worth pursuing even though they are less likely to produce concrete gains in the next 10 years.

Experts did, however, believe improvements for existing therapy and improvements in diagnostic techniques are likely in the near future. For example, experts predicted a 2 in 3 chance that a repositioned drug will demonstrate disease-modifying activity in patients within ten years. Experts anticipated that clinical practice guidelines are more likely than not to endorse the use of body worn sensors for PD diagnosis within the next six years.

Our regression analysis of the relationship between expert characteristics and forecasts suggest that simple demographic characteristics shed very little light on “accuracy”. The only significant contributions to model fit were that those individuals with a Master’s degree or less tended to make predictions closest to the coherence weighted mean, and that men tended to make predictions further from the coherence weighted mean than women. However, both effects were small. If the coherence weighted mean is in fact an accurate assessment of the true probabilities, this analysis implies that looking at demographic variables, even those that supposedly track expertise, may not be the optimal way of finding experts with the best predictive abilities. This would suggest that when decisions like funding or priority setting hinge on assessing the timing of scientific advances, granting agencies or policy-makers may be better off randomly recruiting advisors from a list of established experts rather than seeking out rarefied (and often expensive) expertise.

Our study has limitations. The first concerns milestone sampling. The wording of our survey questions was very specific and may not have captured the promise of broader milestone categories. For example, our gene therapy milestone excluded non-viral vector therapies like anti-sense oligonucleotides, and only concerned treatments for monogenic PD. As such, answers to this question should not be viewed as proxies for all gene therapy approaches being tested. The use of specific milestone questions reflected the scientific imperative in forecast research of ensuring that each question is clear, unambiguous, and verifiable. The inclusion of more general questions might mitigate this limitation. Related to this, many milestone questions concerned outcomes in the U.S. drug approval process. As a consequence, forecasts reflected beliefs about both clinical promise and the pacing and standards used for the approval process itself. To the latter point, non-North American based experts may have found the task of forecasting regulatory attainment milestones more difficult, though their forecasts on the FDA related milestones appear similar overall to the those given by the North American based experts. The second set of limitations concern our expert sampling. Despite coherence weighting and selective eligibility criteria, forecasts could have been affected by response bias. Even so, our sample was largely composed of experts who had records of research productivity and who were affiliated with top neurology programs. Our demographic analysis of nonresponders did not suggest striking biases in our sample. The third set of limitations concern the predictions themselves. It remains to be seen whether the wisdom of the crowd approach we used will provide an accurate assessment of timelines. Further, the very availability of predictions reported in our study could change the probability of milestones being achieved. A last set of limitations concerns secondary analyses. A more comprehensive collection of demographic and cognitive features of experts might reveal other important factors that relate to forecast skill. Also, this analysis was conducted after the data had been analyzed for our primary objectives; all p-values should be understood as hypothesis generating.

That experienced experts did not converge more rapidly on aggregated expert opinion should not be interpreted as questioning the value of expertise. More experienced experts are likely to contribute to funding and policy decisions in other important ways, including the identification of factors that need to be considered when making forecasts [11]. Our survey nevertheless provides a meaningful synthesis of state-of-the art expert opinion on the expected timing of several major breakthroughs in PD research. Many patients and caregivers show intense interest in learning about emerging new treatments; their expectations are often buffeted by hyperbolic claims in the press, on the internet or from pharmaceutical companies with a vested interest in particular therapies. Our forecasts provide a more objective representation of how expert communities interpret available evidence about when major advances will occur. They also provide healthcare system planners with an appraisal of the level of optimism about the availability of new therapies, diagnostics, and research techniques in the coming decade. Ultimately, the approach we employed of soliciting expert forecasts, and weighting them using coherence, can also be of use in helping funders, like disease charities or pharmaceutical companies, access accurate expert judgments of where to invest their resources. The approach of elicitation and aggregation is more likely to avoid many of the biases that accrue due to psycho-social dynamics that emerge with committees [12] or other expert elicitation platforms [13].

CONFLICT OF INTEREST

PBK, DMB, TS, and JK declare no competing interests. RAB receives royalties from Springer and Wiley. He provides consultancy services to Living Cell Technologies; Fujifilm Cellular Dynamics Inc, BlueRock therapeutics; Sana Biotherapeutics; Novo Nordisk and UCB. AEL provides consultancies to Abbvie, Acorda, Biogen, Bristol Myers Squibb, Intracellular, Janssen, Jazz, Lilly, Lundbeck, Merck, Ono, Paladin, Roche, Seelos, Syneos, Sun Pharma, Theravance, and Corticobasal Degeneration Solutions, serves on the advisory boards of Jazz Pharma, PhotoPharmics, Sunovion; and has received honoraria from Sun Pharma, AbbVie and Sunovion.