Crowdsourcing utility estimation for spina bifida in the general population

1. Introduction

Cost utility analysis (CUA) is a type of economic analysis that may be useful when more than one acceptable treatment option is available. In these situations, the ultimate treatment decision typically depends on patient preferences that are in turn dependent on the relative value of the possible health states that can occur after any treatment decision is made. This relative value, usually estimated as a “utility” value, is a number between 0 (representing death) and 1(representing perfect health) that quantifies the impact of a health state on the person’s overall quality of life. Utility estimation is a crucial component of any CUA; if the utility estimate is unsound, then the resulting CUA will be irredeemably biased. Perspectives on the impact of disease progression and treatment on quality of life can subsequently be represented as numerical values within this scale [1, 2, 3].

Spina bifida (SB) is an example of such a preference-sensitive condition. SB is the most common permanently disabling birth defect in the U.S., occurring in approximately 3/10,000 live birth [4, 5]. Patients with SB range from minimally symptomatic to severely disabled. Thus, a range of management options exists for SB patients. Improved treatment and management of SB have greatly increased life expectancy and quality of life for affected patients.

With many SB patients living well into adulthood, the need to better understand patient/family perspectives and values as they pertain to treatment options is becoming increasingly important for all physicians involved in their care. Furthermore, demand for SB health state utilities is increasing as more disease-specific cost-effective modeling is undertaken. At the core of SB CUA is the ability to provide an accurate estimation of SB utility.

Estimates for SB utility value via traditional interview approaches have been previously reported to be from 0.74 to 0.84 [6]. However, to date, no SB utility score has been estimated via online surveys or crowdsourcing methods. Our aim was to determine the estimated utility value for a generic SB health state using a previously validated online crowdsourcing tool [7]. Crowdsourcing is an online problem-solving and production model that utilizes the collective intelligence of online communities for specific purposes. One type of crowdsourcing is called Distributed Human Intelligence Tasking (DHIT), where a large scale of information that is too complex to be processed in an automated way can be broken down and processed individually [8]. Our current study utilizes the advantages of DHIT to estimate utility in SB. We hypothesize that the SB utility derived from online surveys will be similar to those estimated via traditional interview approaches.

2. Patients and methods

3. Results

CUA is a useful tool to consider in the management of conditions, such as SB, when multiple treatment options exist. These conditions typically depend to some extent on patient preferences. These patient preferences are typically dependent in turn on health state utility value, or how patients (or families) value the quality of life for a given health state. Spina bifida (SB) is an example of such a preference-sensitive condition. To accurately conduct CUA research, accurate utility values are crucially important. If a baseline utility value incorrectly represents the value of that health state, then the resulting CUA will be irredeemably biased and essentially worthless. However, utility elicitation, which is usually done via lengthy face-to-face interviews, can be time-consuming and expensive. To our knowledge, utility estimation for SB via online survey has not previously been performed.

We aimed to estimate the utility score for a typical 6-year-old child with SB using a validated online platform [3, 7]. We found this annual utility to be 0.85. Surprisingly, factors such as whether respondents/their child/their friend had SB did not significantly change their estimates of utility score. Prior knowledge of neural tube defect and SB were seen as significant factors in utility score determination on bivariate analysis, although this effect did not persist on multivariate analysis. This is consistent with our previous work on utility estimation [3, 7]. This result can be explained by several factors such as the small percentage of respondents with SB/NTD and possible co-linearity of the analyzed variables. Importantly, however, our crowd-sourced, calculated utility value is firmly in line with other estimates derived from more traditional methods.

A recent review by Tarride et al. showed that the utility estimate in children with chronic health conditions to be 0.80 from a utility survey done among parents, patients, and pediatricians. Interestingly, the average utility score derived from pediatricians tend to be higher than those derived from parents [2]. In another study using the National Health Interview Survey, the mean and median SB specific health utility score were 0.72 and 0.84, respectively [6]. Thus, our reported SB utility value developed from online survey is similar to the established values in literature. The consistency of these estimated utility values provides further support in the use of this online platform as a valuable tool to estimate utility for other urological conditions. This significantly lowers the barrier to deriving health state utility estimates for specific health states specific to the various treatments that patients with SB often undergo (e.g., shunted vs. not, continent vs. not, reconstructed vs. not, catheterizing vs. not).

At present, utility estimation in urology has been limited. There have been reported utility estimates for ureteroscopy, shock wave lithotripsy, and congenital ureteropelvic junction obstruction [12, 13]. These mentioned utility estimates have been conducted in the traditional approach via interviews. However, more recently, utility estimation via online platform has gained popularity as a valid and efficient method. Our group previously reported the utility estimation of childhood vesicoureteral reflux (VUR) using this online survey method and found that it was comparable to the utility score obtained via traditional interviews [7]. Methodological limitations of this online platform have also been evaluated recently. We previously reported that utility estimate fluctuates based on whether it was parent vs. child vs. dyad perspective. In addition, utility weight was surprisingly unaffected by anchoring bias, which is a common human tendency where subsequent decisions are made based on the initial piece of information [3]. Therefore, this online crowdsourcing tool not only provides precise utility estimation of VUR, but also more chronic pediatric urological conditions such as SB. The online tool’s success in deriving utility values for these two disease states suggests that it is a valid, reliable, and efficient tool to estimate CUA utilities.

The result of our study should be viewed in the context of its limitations. First, our sampled respondents may not provide a true representation of the U.S. population. The demographics in our study were 48.8% F, 86.8% Caucasian, and 37.6% married. In a study by Dworkin et al., they recruited subjects for family dynamic research. In their study, the demographics of US population were 50.8% F, 74.8% Caucasian, and 52.4% married. Age was not part of their demographics analysis [14]. Compared to national data, we showed similar gender demographics. However, there were less respondents who are married and more Caucasian respondents in our study. Given that the majority of our respondents were Caucasian, it is uncertain how precise our current utility value can estimate other racial ethnicities. However, race was not found to be a significant contributing factor on our multivariate analysis. Furthermore, previous literatures have shown that MTurk is more representative than other commonly used samples (U.S. undergraduates), and the survey result obtained through MTurk is similar to the traditional interview survey [15, 16, 17]. Nonetheless, a wide geographic and racial representation remains a major challenge in utility estimation.

Second, our current study estimates the utility of an overall SB state but does not provide specific distinction for myelomeningocele vs. other forms of SB (e.g., SB occulta or lipomeningocele). We would hypothesize that the utility value for myelomeningocele would be expected to be lower than other “less severe” forms, but we did not explore this point in this study. Furthermore, our analysis was based on trading from a parent/child dyad perspective. A prior study by our group showed that utility estimate was highest from child’s perspective followed by dyad’s perspective, and lowest from a parent’s perspective. Thus, the utility estimated from the dyad perspective likely provides an average from the other two health states. In addition, we estimated utility value based on a generic 6-year-old patient. This may not reflect the true utility estimate in much older or younger patients. However, the purpose of our current study is to validate the effectiveness and feasibility of crowdsourcing methods in estimating utility value in SB in general. This could now be adapted to a nearly infinite array of patient archetypes – preoperative patients with urinary leakage, postop patients with complications, older patients, younger patients, etc.

Third, we limited respondents’ age to older than 18 years. However, we did include a diverse cohort of participants such as parent/non-parent, familiar/unfamiliar with SB, and single/married. It is uncertain how much impact the age limit will affect utility estimation on a mostly pediatric condition. However, due to improvement in the care of SB, majority of patients live well into adulthood, and we would anticipate that the impact of this age limit would likely be minimal.

In addition, our survey respondents may not have a full spectrum of understanding about living with spina bifida. The brief Youtube video and description paragraph serve as a general overview of SB, but they do not necessarily provide an emphasis on some of the specific urological complications related to SB. However, the goal of this project is to determine the feasibility of MTurk for assessing utility in SB health states, but the fundamental structure of crowdsourcing can be applied to other health states as well (mobility, shunt malfunctions, fecal continence, decubiti, etc).

Finally, due to the paucity of literature on SB utility values, it is difficult to draw definitive conclusion on the accuracy of any one published estimate (i.e. “gold standard”). However, the consistency among the currently published SB utility values and our estimate is reassuring. By establishing online crowdsourcing as a valid tool for SB utility estimate, it provides a solid foundation for future studies to better define SB utility values in varies health states.

5. Conclusions

Using a validated online survey platform, we estimated that the utility value for a typical 6-year-old child with SB is 0.85, which is consistent with prior published utility scores obtained via more traditional approaches. Our study further supports the reliability of MTurk and crowdsourcing in general as a viable technique in spina bifida utility research.