Abstract
Decision support systems (DSS) incorporating satellite data can potentially augment the understanding of flood management decision makers in Indonesia. To mitigate mismatches between DSS functionality and stakeholder objectives that have hindered previous efforts, a DSS is developed using a novel stakeholder engagement process. A pilot study of Boston-area participants (n = 20) assessed whether DSS users had improved understanding of historical trends and policy impacts compared to control group participants receiving a briefing. Participants’ understanding and projection of environmental, socioeconomic, and policy trends were assessed via scored questions and compared between groups using a Wilcoxon sign rank test. Higher understanding scores across categories were obtained by DSS users compared to the control group. Improved understanding of environmental, socioeconomic, and policy factors can be gained from a satellite data DSS developed using our novel process. Increased confidence in the DSS was gained ahead of evaluations with Indonesian end-user decisionmakers.
Decision support systems (DSS) incorporating satellite data can potentially augment the understanding of flood management decision-makers in Indonesia. To mitigate mismatches between DSS functionality and stakeholder objectives that have hindered previous efforts, a DSS is developed using a novel stakeholder engagement process. A pilot study of Boston-area participants (n = 20) assessed whether DSS users had improved understanding of historical trends and policy impacts compared to control group participants receiving a briefing. Participants’ understanding and projection of environmental, socioeconomic, and policy trends were assessed via scored questions and compared between groups using a Wilcoxon sign rank test. Higher understanding scores across categories were obtained by DSS users compared to the control group. Improved understanding of environmental, socioeconomic, and policy factors can be gained from a satellite data DSS developed using our novel process. Increased confidence in the DSS was gained ahead of evaluations with Indonesian end-user decisionmakers.
Introduction
The millions of people living on the North coast of Java Island, Indonesia (60% of Indonesia’s population), are vulnerable to land subsidence and related flooding. Land subsidence is defined as the “vertical movement of the earth surface due to the subsurface movement of earth materials” and can be caused by anthropogenic activity (e.g. fluid extraction) or natural causes (e.g. tectonics). Land subsidence is the primary driver of flooding, with rates 10 times greater than sea level rise (Sarah & Soebowo, 2018). Flooding impacts span intersecting societal factors, including coastal ecosystems (Marfai, 2014), public health factors (Yuniarti et al., 2019), and agriculture and aquaculture. These negative impacts are exemplified in Pekalongan, a coastal city of 300,000 people located in Central Java where land subsidence rates range from 4.8 - 10.8 cm/year (Sarah & Soebowo, 2018) and floods routinely inundate large areas of the city (Heru, 2020). Leaders in Pekalongan face decisions on how to implement flood mitigation interventions including mangrove restoration, dikes, and pumping stations. Decisions on how and where to implement these interventions are complex due to gaps in understanding of environmental dynamics and complex system of interrelated socioeconomic impacts, policy interventions, and technological factors. Other challenges include a diverse stakeholder network and data scarcity issues (H. Hendarti, personal communication, 2021). Creating Decision Support Systems (DSS) incorporating sociotechnical factors can lead to improved understanding of the outcomes and impacts.
Incorporating satellite remote sensing data within a DSS can help address these decision-making challenges via software that allows decision makers to examine historical data, explore relationships between disparate factors, and evaluate potential decisions or investments (Shim et al., 2002). Satellite data has key benefits for decision makers including broad spatial coverage, a consistent temporal record, and free availability - all of which have the potential to improve understanding of environmental and socioeconomic trends and address data scarcity issues. DSS incorporating satellite data have been employed in situations related to Pekalongan, including understanding mangrove loss and degradation (Goldberg et al., 2018), mangrove health, ecosystem services, and related policy (Reid & Wood, 2020), and ecological, socioeconomic, and governance factors for marine management areas (Gopal et al., 2015). However, despite the availability and capability of satellite data for flood management decision-making, failures of DSS developers to properly consider end-users’ needs and objectives (Díez & McIntosh, 2009; Edwards et al., 2010; McIntosh et al., 2008) are connected to DSS disuse by end-users. For satellite data specifically, challenges remain with linking the satellite data to the policy realm (Finer et al., 2018), data interpretability (Reid & Wood, 2020), and lack of awareness and accessibility (Ovienmhada et al., 2021).
To address these gaps, a satellite data DSS for Pekalongan was developed using a novel design process. System architecture analysis was used to translate stakeholder interviews into targeted functions and forms for the satellite data DSS, and integrated modeling frameworks were used that incorporated local socioeconomic data to facilitate actionable insights and increased accessibility (Lombardo et al., 2022).
Before providing the DSS prototype to end-users in Pekalongan, an experimental evaluation of the prototype was conducted with Boston-area users. This initial user study assessed the hypothesis that higher understanding scores of historical trends and policy impacts will be achieved by DSS users compared to participants not using the DSS. This experiment supports evaluating the success of our DSS design in overcoming DSS and satellite data accessibility gaps and enables honing the prototype before assessment in Indonesia.
Methods
Decision Support System Prototype
The DSS prototype evaluated in this study was designed through a collaborative process with local decision makers in Pekalongan, Indonesia. Analysis of stakeholder interviews targeted satellite data analyses to local objectives, and integrated modeling frameworks guided the incorporation of local Indonesian datasets for added context. More detail on these initial project stages can be found in Lombardo et al. (Lombardo et al., 2022). The DSS prototype consisted of a web-based dashboard (Fig. 1) containing this integrated data.
DSS prototype evaluated by study participants. The prototype incorporated satellite data analyses and local socioeconomic data in a web-based dashboard interface.
Participants
Study participants (n = 20, 9 female, 11 male) consisted of local Boston and Cambridge area government decision-makers, MIT staff, and MIT graduate students. Inclusion criteria for the study were the ability to read, write, and speak English and to comfortably sit for a 45-minute role-playing experiment. The full experimental protocol was determined exempt from review by the MIT COUHES.
Experimental Design and Procedure
Before engaging in the roleplaying phase, participants completed a pre-study assessment to gather information on participants’ baseline level of experience with data interpretation. The assessment asked questions regarding the interpretation of maps and plots. The maps and plots were not directly related to the DSS prototype, but contained similar categories of information data types (population density, pollution heat maps, and a temporal graph of COVID cases).
The experiment consisted primarily of a simulated roleplaying scenario. Participants were given an overview of the situation in Pekalongan for context, then instructed to play the role of a technical advisor providing recommendations to a local decision-maker in Pekalongan.
Participants received (at random) either a control briefing (an existing presentation from the real-world Indonesian context), an experimental briefing (which incorporates new satellite data), or the experimental DSS (an interactive software incorporating satellite data). A between-subjects design, where the participants in each experimental group were different, was selected due to concerns around learning effects. If participants completed the scenario with the briefing and then completed the same scenario with the DSS, their difference in performance may be attributable to simply gaining more experience with the scenario. If participants were to use the traditional decision making briefing to complete one scenario, then use the DSS to complete a different scenario, issues arise regarding how to ensure the scenarios were different enough to mitigate learning effects, while also remaining similar enough for the results to be comparable for analysis. The inclusion of the experimental briefing enabled assessing whether any improvements in understanding were due to satellite data analyses themselves (which would yield improvements for both the experimental briefing and DSS group), or due to the nature of interacting with the information (in which case the DSS may prove more beneficial than the experimental briefing or vice versa).
For the briefing groups, participants were given several minutes to look through the briefing slides, then were given a verbal briefing using the same slides and were permitted to take notes. After the briefing, participants had 20 minutes to answer the assessment questions for analysis and recommendations as an advisor within the scenario. For the experimental DSS group, a short presentation described how to interact with the software interface and summarized the information in the DSS, and was followed by 5 minutes of DSS exploration for familiarization. Then participants had 20 minutes to utilize the DSS towards answering the question prompts.
Experimental Metrics
The scenario question prompts were designed to assess the hypothesis regarding understanding of historical trends and policy impacts. The questions were delineated in several ways to systematically gauge dimensions of participants’ understanding. The highest-level categorization of questions were environmental phenomena, socioeconomic impacts, and flood mitigation interventions. Within each of these categories, subcategories of understanding questions (gauging understanding of historical trends and present day conditions) vs. projection questions (assessing extrapolation towards insights on future trends) were asked. These sub-categories align with levels of situation awareness defined by Endlsey (Endsley, 1988). The quantity of different question designations were as follows: 3 environmental understanding, 2 socioeconomic understanding, 1 flood mitigation intervention understanding, 2 environmental projection, 1 socioeconomic projection and 1 flood mitigation intervention projection. Each question was in a short-answer format.
One example of an understanding question assessing historical and current phenomena was, “What areas of the city have been impacted by flooding?”. The question, “What areas of city are at high risk of flooding in the future?” exemplifies a projection question assessing extrapolation of future trends.
The answers to the question prompt were scored using a rubric with the following point allocation:
0 – the answer provided is incorrect in its interpretation of the information provided.
1 - the overall answer is incomplete with respect to key content. Some correct information is provided, but key information that is important to the question is missing.
2 - the interpretation is correct, but it is not specific enough to be actionable.
3 - the interpretation is correct, and it contains enough specific information to be actionable.
A final question was asked where participants provided a long-form recommendation in their role as an advisor to capture any emergent behavior that was not assessed via the previous questions. This long-form question was thematically coded into "concerns" and "recommendations" and shortened to key-phrases that summarized the concern or recommendation.
Statistical Analysis
Post-hoc Wilcoxon Sign Rank tests were conducted to examine the differences in scenario question scores between groups. A non-parametric test was used as the data were not normally distributed. These tests were performed to compare each question category (environmental, socioeconomic, and flood mitigation interventions), the understanding and projection question subcategories, and the overall aggregate score. Post-hoc comparisons were corrected using the false discovery rate method (Benjamini & Hochberg, 1995).
Differences in scores for each category that contains more than one question (environmental, socioeconomic, and flood mitigation interventions), the understanding and projection question subcategories, and the overall aggregate score were compared between groups. Scores were summed for each of the categories and subcategories to facilitate comparison.
Results
No significant differences in baseline capabilities were found between groups, with an average score of 98% and all users answering 80% or greater of the pre-study assessment questions correctly.
Before analyzing scores across categories, we can see the significant differences in scores for individual questions (Figure 2). Both the experimental briefing (W = 21, p = 0.0022) and the DSS (W = 21, p = 0.0012) led to an increased understanding score for spatial trends in permanent water compared to the control briefing (Fig. 2a) The DSS led to improved projection of socioeconomic trends (W = 24.5, p = 0.0089, Fig 2b) and flood mitigation intervention efficacy (W = 21, p = 0.0043, Fig. 2c).
Individual question score comparisons across experimental groups. A) Understanding of changes in permanent water, B) projection of socioeconomic impacts, C) projection of flood mitigation intervention efficacy.
Next, we can compare scores summed across question categories and subcategories (Figure 3). Increased environmental understanding scores were achieved via both the experimental briefing (W = 21, p = 0.0022) and DSS (W = 21, p = 0.0012) compared to the control group (Fig. 3a). Similarly, higher overall understanding scores across categories were achieved by DSS users (W = 22.5, p = 0.0035) and experimental briefing users (W = 21.5, p = 0.0043) compared to the control group (Fig 3c).
Category-based score comparisons across experimental groups. A) Understanding environmental phenomena, B) understanding of socioeconomic impacts, C) overall understanding score, D) projection of environmental phenomena, E) overall projection score, F) overall score.
Increased summed projection scores were achieved by DSS users (W = 21, p = 0.0095) compared to the control group (Fig. 3e). Higher overall aggregate scores were achieved by both the experimental briefing (W = 23.5, p = 0.0108) and DSS groups (W = 21, p = 0.0095) when compared to the control group (Fig. 3f).
The final question asked participants to provide a long-form response on their chief concerns and recommendations as an advisor in the scenario (Table 1). Control group users mentioned land subsidence frequently as a chief concern, while this concern was not mentioned by experimental group participants. There was a higher percentage of participants that mentioned threats to agricultural and aquacultural industries from flooding in the experimental groups compared to the control group). Specific flood mitigation interventions (ex: sea walls, mangroves, pumps, etc.) were called out more frequently by the experimental groups. This contrasts with the control group, where several participants specifically called out a need for more research before providing recommendations. The frequency of pumps as a recommendation was higher for experimental briefing users, compared to a high frequency of sea wall recommendations by the DSS group.
Percentage of participants that provided selected responses concerns and recommendations designations.
Discussion
This pilot study of Boston-area participants sought to assess potential augmentations to flood management decision-making obtained via a satellite data DSS designed through a novel stakeholder engagement process ahead of assessment by real under-users in Indonesia. Results indicate that the DSS provided improved projection and understanding of the relevant factors over the control briefing.
Several insights can be drawn from examination of individual questions (Fig. 1). A simple interpretation of improvements on the individual environmental understanding question is that new satellite data analyses included in the DSS and experimental briefing, but not present in the original control briefing, promoted understanding of the state of permanent water in Pekalongan. (Fig. 1a) An interpretation of the higher scores between the control and DSS groups (but not the experimental group) on socioeconomic and flood mitigation intervention projection (Fig. 1b and 1c), is that the format of the combined information present in the DSS promotes unique insights with regards to projecting socioeconomic impacts and flood mitigation infrastructure efficacy. For example, the ability to view integrated information on risk analyses and land use information, or integrated information on risk analyses and flood mitigation interventions (neither of which were present in the control briefing), may promote better projection for these areas. The variability in scores for the experimental briefing was observed to be larger than the DSS, suggesting that some participants in the experimental briefing (where combined information on these quantities was present) were able to achieve better projection scores (Fig. 1b and 1c). However, this contrasts with the uniformly high scores of the DSS users for these questions, suggesting that the format of the DSS more consistently promoted the use of integrated visualizations that led to higher scores (via its interactive, multi-layered web map format), compared to the experimental briefing.
Examining differences in sums scored across categories also yields insights into the benefits of the DSS for user understanding. Significantly higher scores in the environmental subcategory (Fig. 2a), line up with the interpretation of individual environmental questions (Fig. 1a): new information on environmental phenomena present for the experimental group users increases understanding of the present state of these environmental phenomena. The sum of all understanding questions (environmental, socioeconomic, and flood mitigation interventions) (Fig. 2c) was higher for DSS users compared to the control group. Therefore, while not all experimental users achieve the highest scores on socioeconomic understanding the socioeconomic questions (Fig. 2b), we can see via the summed scores that users gain increased understanding across all categories combined (environmental, socioeconomic, and flood mitigation interventions) from the experimental briefing and DSS to make their overall understanding score different from the control group.
The significantly higher scores of the DSS group (Fig. 2e) line up with the previous interpretation of individual socioeconomic and flood mitigation intervention questions (Fig. 1b and 1c): that the integrated information present in the DSS promotes better projection of socioeconomic impacts and flood mitigation efficacy across all participants in that group, compared to a wider distribution for the experimental briefing group. The higher scores of the experimental groups across all categories align with the examples from the literature on the benefits of satellite DSS with respect to understanding complex systems (Fig. 2f).
The frequency of certain coded concerns for the final question varied across groups, providing qualitative insights. The frequency of land subsidence as a concern in the control group over the experimental groups points to 1) the emphasis on land subsidence in the control briefing, but also 2) that there was more information in the DSS and experimental briefings. Given the flexibility to explore that information, those experimental users settled on different concerns. Similarly, the higher frequency of agricultural and aquaculture concerns in the experimental groups compared to the control group, is likely because the experimental decision aids included more information on land use that could then be compared to analyses on flood risk (information that was largely absent in the control briefing).
Likewise, the frequency of certain recommendations also provides insights about the differences between decision aids. The frequent mention of specific flood mitigation interventions in the experimental groups compared with control group, indicates that the information present in the DSS and experimental briefing facilitated insights for participants that gave them confidence to provide specific recommendations rather than feeling that more information was needed. These differences in information were also present between the DSS and experimental briefing as well. Users in the DSS group were more likely to recommend pumps, whereas users in the experimental briefing group were more likely to recommend sea walls. Both interventions are objectively useful for flood mitigation intervention, though the varied nature of how these interventions are presented in each experimental decision aid clearly predisposes some users to certain interventions.
This study had limitations including the inherent challenges in comparing participant scores in a between-subject experimental design, a relatively small number of participants (n = 20), and that participants were drawn from a US participant pool that were not flood management decision-makers, rather than an Indonesian participant pool more familiar with the conditions of the simulated scenario. Future work for this study includes the ongoing analysis of the real end-user assessment conducted local stakeholders in Pekalongan, as well as iteration on the DSS prototype based on user feedback received in this pilot study as well as in Indonesia.
Conclusions
Significantly higher understanding and projection scores from questions and category sums spanning environmental, socioeconomic, and flood mitigation intervention questions provide strong support for the study’s hypothesis: that higher understanding scores of historical trends and policy impacts will be achieved by DSS users compared to participants not using the DSS. These results support the environmental insights that can be derived from the broad spatial coverage and consistent temporal record of satellite remote sensing data. Additionally, the use of integrated modeling frameworks in our novel DSS design process to drive the incorporation of local socioeconomic data alongside satellite data yields benefits in terms of understanding the intersecting socioeconomic and policy factors at play in Pekalongan. Here, we contribute a pilot study that demonstrated the benefits of the prototype DSS developed using our novel design and stakeholder engagement process. The study increased confidence in elements of the experimental protocol and the DSS itself ahead of evaluations of the satellite DSS with real end-user decision makers in Indonesia to aid in the support of flood mitigation efforts.
