Doing Research With a Gamified Survey: Reflections From Smart City Research

Our Case: Your Neighbourhood Your Data

We designed and used a gamified survey in 2018 to explore citizens’ knowledge (technical familiarity) of smart cities, and to assess their privacy concerns and actual privacy-related behaviour in smart cities (for the outcomes of this survey, see Rijshouwer et al.). The first reason to use such a gamified survey that includes a visual representation of a smart city is that there is no actually existing, recognizable ‘smart city’ where we can find all relevant aspects of urban datafication and where we could study respondents’ experiences and interactions with this environment (Pisani, 2015). Secondly, we considered a gamified survey as an effective (accessible, low threshold, inspiring, motivating) means to reach out to a wide audience, both to raise their awareness about smart cities, but also to collect sufficient data to make valid claims on the smart city knowledge, privacy concerns and privacy behaviour of the Dutch population. The third reason to use a gamified survey was that there is ongoing debate about the relevance of research when privacy concerns and behaviour are taken into account: the accuracy of respondents’ answers to privacy related questions and statements are disputed, as they supposedly differ from their actions in real life. Knijnenburg et al. (2017, p. 2) emphasize that, as a lot of privacy research is conducted post-hoc, the retrospective rationalizations of respondents do not necessarily correspond with their actual concerns and behaviours. In the literature this is captured in the term ‘privacy paradox’. ² Through the gamified survey we were able to craft seemingly realistic situations and scenarios to test, at a large scale, how respondents would share their personal data in particular situations rather than asking them about their concerns and their past and potential behaviours in a traditional survey or an interview.

In terms of Adamou’s (2011) qualifications of different research games, our gamified survey was ‘avatar-based’ and took a ‘soft gamification’ approach. The survey included an online visual representation of a smart neighbourhood through which the respondents’ avatar could navigate, whilst answering questions, and solving puzzles. At several points in the game, respondents were asked to share personal data in exchange for financial benefits as well as for convenience, social and safety and security purposes. At two locations in the city, respondents had to identify digital technologies or institutes that collect data, such as public transport gates, the city hall, surveillance cameras or Wi-Fi trackers. At several points in the game, respondents would receive an award or badge. At the end, they were given feedback about where they shared data and where data were collected, as well as a personal profile about their knowledge and data sharing behaviour.

Gamified Surveys

Gamification is ‘the use of game design elements in non-game contexts’ (Deterding et al., 2011, p. 2). These elements are, among others: clearly identifiable rules and objectives; a story or narrative which participants can relate to; challenging tasks; rewards; feedback; and a motivating virtual, visual and often audio environment (Keusch & Zhang, 2017, p. 150). Gamification is considered an effective means to test and train (costly and complex) technical, medical and military situations, and, since the past decade, gamification has increasingly been used to conduct market and consumer research. ‘Research through gaming’, is, according to Adamou (2011) a situation in which organizations and researchers develop games to gain data from the actions respondents take whilst engaging in that game. However, according to Barwick et al. (2018, p. 232), ‘there has been little interest from the academic community in the potential opportunities that gamification presents in the research context’. The few academic reflections on gamified research concern basically two issues: gamification as a means to engage respondents in new and appealing ways, and the challenge to the validity and reliability of the data that gamified research produces. There is even less about the process or the ‘how to’ of designing gamified research and its consequences.

With regard to gamification as an instrument to engage participants in research, and specifically for gamified surveys, the standard argumentation is that respondents of ‘traditional’ surveys often suffer from survey fatigue, which compromises both response rates and the quality of the outcome of surveys. Especially in online surveys, respondents appear to be ‘straight-lining’ (i.e. clicking on the same position on successive similar scales), choosing random answers to speed up the survey, or not finishing the survey at all (cf., Turner et al., 2014, p. 4). Hence, scholarly papers are mostly concerned with identifying and evaluating which game mechanics to deploy that put (potential) respondents in an enjoyable situation that they find ‘both challenging and absorbing’ (Keusch & Zhang, 2017, p. 149), and which motivates them to respond sincerely to questions and challenges. Hoekstra (2020) identifies psychological and behavioural dimensions in legitimating the choice for a gamified survey, with psychological assumptions pertaining to motivation and pleasure of the respondents who fill in the survey, and behavioural assumptions being about higher completion rates and data quality. According to Hoekstra, several studies have found such higher engagement and completion rates. However, he also reports a study that compared completion rates of a standard text-based survey with a gamified text-based survey and found that the gamified version was less often completed than the standard version (Downes-Le Guinn et al., 2012 in Hoekstra, 2020). This study was subject to criticism because visual game elements were lacking and because the game itself had no connection to the survey topic (Keusch & Zhang, 2017). When these two elements of visuality and relevance are present in a gamified survey, both the psychological and the behavioural engagement of participants have been shown to increase.

With respect to the validity and reliability of the data generated by gamified surveys, Keusch & Zhang (2017, p. 151) state that ‘changing the context of a question [to make the survey more game-like] might also alter its meaning and jeopardize the validity of the question’. In other words, adding an image or a narrative scene to gamify a question possibly changes the meaning of the question. Whilst there is a long tradition of designing, testing and adjusting measurement instruments until they are considered ‘validated’ (e.g. Wright, 1997), the challenge of measurement error has not yet been systematically addressed for gamified surveys. Keusch & Zhang (2017), rather ambiguously, write that ‘measurement error can either be increased or decreased by introducing gamification in a survey’. (Keusch & Zhang, 2017, p. 151). This ambiguity is further reflected upon in the different viewpoints of opponents and proponents of gamified surveys. Dan & Lai (2013), for instance, argue that gamification shifts the focus of respondents away from the actual task, which is filling in survey questions, to game play in a ‘blithe and play-like atmosphere’ (Dan & Lai, 2013, p. 4376, quoted in Keusch & Zhang, 2017, p. 152). However, proponents argue that a game environment, provided that it is connected to the survey topic, brings people closer to their real-life experiences than written questions will do, and that it invites a wider variety of viewpoints in answering open questions (Turner et al., 2014).

From these reflections we can conclude that the design of a gamified survey should include the full array of gamification (including visuals, rewards, tasks and narratives that are all perceived as relevant) and that the moments of data collection in the game should be piloted and tested to assess their validity. This still leaves the subsequent question as to how to actually go through the work and decisions of designing a survey game. In other words: how do those involved in designing and producing gamified surveys struggle and deal with challenges of respondents’ engagement and validity of research data in practice? Turner et al. (2014) describe how practical constraints like time and money affected the design of their survey game necessitating decisions about, for instance tradeoffs between engagement and methodological rigour. These are, of course, constraints and consequence that are not unique to designing a gamified survey. Practical constraints also hold for more traditional methods of data collection, but in a gamified survey, like in other design methods, there is at least one other stakeholder who takes part in these discussions and decisions, which is the game designer. In many situations, there is also a third stakeholder in the form of a commercial or societal commissioning party. The design and production of a gamified survey is therefore often a multi-stakeholder process, with all the connected negotiations between actors and their various backgrounds, various concerns, various normative positions and various levels of autonomy and agency operating in the process. Whilst these negotiations have been theoretically well elaborated in science and technology studies (especially in actor network theory) or innovation theory (captured in the concepts of the triple and quadruple helix), their actual and practical manifestations in real-life cases have been much less examined. A noticeable exception, especially for the purpose of this paper, is the thick, historical description of the ideas, conflicts and obstacles surrounding the building of the data dashboards of the municipality of Dublin (Kitchin et al., 2015). Kitchen and his co-researchers followed an almost forensic method to trace all the actors, interactions and motivations that influenced the data and indicators that were included in the final dashboard. They were thus able to come to an analysis of the everyday and office ‘politics’ through which the dashboard eventually took shape. This is the method we will follow in this paper as well, by tracing how the negotiations between the stakeholders affected the design decisions for the gamified survey, articulating, in the end, both the social and epistemological potential of the gamified survey.

Approach and Sources

We analyzed the design and production process of a gamified survey, the various concerns and interests that the stakeholders (researchers, designers, commissioners) brought to the table and their impact on the engagement and knowledge potential of such a research tool. We asked and assessed how they come to their decisions; how they account for them; and how these (might have) affected the design and the possible outcomes of the survey. To answer these questions, we conducted an (auto-)ethnographic study of the design and development process, which took place between May and October 2018. This process was organized according to the so-called ‘agile’ method, meaning that in consecutive ‘sprints’ (i.e. intensive co-creative working sessions), various aspects of the gamified survey were iteratively designed, developed and tested with members of the target audience. The design team consisted of three researchers and the academic director of the Leiden-Delft-Erasmus Centre for BOLD (Big Open and Linked Data) Cities; and the CEO and the CTO, one creative director, one project leader, two UX designers, two illustrators, two developers, and one sound designer/data scientist from the game design agency. We held weekly meetings with the whole team to discuss progress and issues, and irregular meetings in sub-groups on sub-themes. One executive and one project leader of the Dutch Weekend of Science, who commissioned the research, reflected on the design after each sprint. The BOLD Cities researchers functioned as co-designers as well as embedded action-researchers and participant observers. We built an archive of stakeholders’ email correspondence and filed comments, concerns and designs through Google Docs, a designated Slack-channel (#bold-cities-game) and a Trello to-do page. We made a special point to keep notes and logs of the discussions and challenges we considered to be crucial for the actual result of the process. We used this material to reflect on a number of ‘critical moments’ in the collaboration in which different interests collided or conflicted. These are

- our pitch to the Dutch Weekend of Science, which selected our gamified survey as their research showcase for the general public;

In the following section, we describe and analyze decision-making and its (potential) impact for each critical moment of the gamified survey. We then discuss the launch of the survey and its quality as an instrument for data collection.

Making the Gamified Survey

Selection of a Design Agency

As the budget of the project for the design of the gamified survey would exceed €50,000, Dutch procurement law obliged us to invite minimally three agencies to pitch a design, technical specifications and a budget for making the research tool. Given our close connection with the city of Rotterdam, we chose to invite Rotterdam based agencies: a well-known game developer; a successful marketing and communication firm; and a research-driven interaction design company. Although the first company had the most experience in game mechanics and the best ideas how to keep respondents engaged, we did not select them. Their storyline was based on a catching a hacker, which we thought would be helpful in informing respondents about the risks of urban datafication but which also would nudge our respondents beforehand to become protective of their data. Our aim was instead to find out how people handle their data in everyday unreflective situations. The second agency proposed to attract participants via a campaign with comical influencers. We did not select them either, because we did not expect the recruitment of participants to be an issue, based on the fact that previous Weekend of Science projects had drawn well over 10,000 participants. We also felt that more mature respondents would be less well addressed by young influencers, and we feared that their jokes would lead the participants to be less serious in doing the gamified survey.

Their idea of the third agency was that respondents could explore virtual neighbourhoods from a first-person perspective on their laptops and tablets, which would be close to their experiences in real life. The initial proposal was to use the Google Maps API to construct virtual city tours. However, practical considerations (following from the complexity and inflexibility of working with the API) and budget constraints made this agency propose to work with an award-winning illustrator. We were convinced by the vendor’s argument that illustrations would allow us to be in full control of the content and to be able to visualize whatever we required. ³ This helped us also to create a hybrid urban-rural municipality and thus comply with one of the Weekend’s requests.

Transformation of the Research Questions into Game Elements

In a deliberative, iterative, creative process, researchers and designers collaboratively worked to gamify our research on citizens’ knowledge, privacy concerns and behaviour regarding smart cities, both to inform as broad an audience as possible on the topic of smart cities and to motivate them to provide us with as valid and reliable data as possible. We tried to collect socio-demographic independent variables by having respondents construct an avatar with pre-given visuals. However, quite early in the design process we realized that the construction of an avatar could invite all kinds of cross-dressing and identity play, resulting in less valid and reliable data about these characteristics. Moreover, due to financial constraints the avatar could not be made visible in the actual game play of the survey, which made several participants who tested the survey in its early stages respond that they did not see the point of designing the avatar. However, as we figured that avatar-creation would stimulate at least some respondents to immerse themselves in the virtual smart city we designed, we decided to leave it in the game anyway. We added a short sequence of written questions about gender, age, education and residence to motivate them to sincerely provide the socio-demographic data we sought.

Our dependent variables concerned what people know about smart cities, what their concerns about their personal data are, and if and how they actually share their data in real-world situations (privacy behaviour; see Figure 1).OPEN IN VIEWER

Gamifying the Survey

Turning questions and variables into game elements does not itself produce a gamified survey. We also needed a narrative, an element of play and competition and rewards. The narrative was that of a person visiting an unknown city and meeting places and people. The play element was designed by awarding badges for each part of the city they visited. The most important element, both from a respondent engagement and a scientific empirical perspective was the final reward for completing the gamified survey. Having reached the end of the survey, respondents were presented with an overview of the data points they (could have) identified and the instances in which they (could have) shared personal data. From these two dimensions we built a simple algorithm that classified our respondents as one of four types, depending on how they scored on each dimension: a ‘generous data sharer’, being a person identifying more than half of the data points and sharing personal data in more than half of the cases; a ‘frugal data sharer’, being a person identifying more than half of the data points and sharing personal data in less than half of the cases; an ‘unsuspecting data sharer’, being a person identifying less than half of the data points and sharing personal data in more than half of the cases; and a ‘aloof data sharer’, being a person identifying less than half of the data points and sharing personal data in less than half of the cases. The respondents would get an image with text saying, for instance: ‘You are a Generous Data Sharer! You identified quite a few places where data is collected. Although you indicate that you are very concerned when it comes to sharing personal data, in practice you do not seem to be very worried about whether you share your personal data and how it is used’.

Collaboration

The actual collaboration between the commissioning Weekend of Science, researchers and the design agency started in June 2018 immediately after the agency was selected. The project had to be finished within 4 months, by the end of September. The deadline was strict, as the gamified survey had to be launched in the Weekend of Science which takes place during the first weekend of October.

The project was planned in four phases: Research, Conceptualization, Development and Validation. The Development phase in particular turned out problematic. It took place in four so-called ‘sprints’, in each of which a part of the gamified survey was built. In each of these sprints, user-tests were included which would inform the next sprint. Ideally, by the end of each sprint, the team would conclude about the next course of action for all makers involved. In practice, however, this worked differently, as an example illustrates. The team as a whole had decided that having a personal avatar who would walk through the city would be engaging for respondents. Although Chung et al. (2007) found no proof for this assumption, Adamou (2019, p. 61–62) suggests that providing respondents with the opportunity to create an avatar ‘could help participants [in a gamified environment] behave as their true selves’. ⁴ The idea was that respondents could compose a personal avatar with which they would navigate through the streets, parks and squares of our gamified survey. They would be enabled to select a figure’s gender, skin colour, hairdo, wardrobe and accessories. However, the illustrators spent so much time on visualizing the various elements of the avatars and missed, as a result, every deadline to test if respondents understood and liked avatar-creation. Moreover, their focus on detail taxed the budget by making it possible that the avatars would actually move (in isometric projection) with the respondents through the virtual neighbourhood. Finally, the amount of detail made it impossible that the avatars would have worked in the standard browsers of respondents. Whilst we, as research team, did recognize the aesthetic merit in the designs, we were frustrated by the lack of attention to other requirements for the avatar and the neglect for the overall set of requirements that kept expanding based on insights we developed during the process of researching, designing and testing. The designers, on the other hand, became frustrated with what they considered to be an ever-increasing list of demands from the research team. The closer we approached the launch of the project, the more problematic these tensions became until, finally, the illustrators withdrew from communicating with the rest of the team to meet the deadline, as they explained. The agency’s project leader allowed this, and thus cut off the research team from a number of important design decisions. In the end, the first-person perspective in the game and hence the ‘data walking’ character was sacrificed, as was the adaptability to different devices (particularly tablet and mobile) and browsers other than Firefox, Safari and Chrome.

In retrospect, we recognize that all involved lacked experience in collaborating in such a complex, multi-stakeholder project, or underestimated it. Caught between the organizational culture of the design agency and the demands of the Weekend on the one hand, and our pursuit of methodological rigour on the other, we probably spent too long negotiating and trying to find compromise and consensus. We would have done better if we had issued stricter requirements and more definitive responses during the process. However, whilst doing research one inevitably develops insights one could not have had beforehand. The design agency, on the other hand, may have respected the autonomy of their creative staff too much and were thus not able to meet the needs of the research team and the Weekend. Professional experience of the participants who were all relatively young, could have mitigated some of these problems. Underlying these tensions, however, was a somewhat naive assumption that working in collaborative design sprints would automatically produce shared solutions and directions, and does not need clear and (if necessary) authoritative interventions. With the benefit of hindsight, we recognize that the ideology of self-organization and agile development that we all adhered to neglects issues of complexity, leadership and appropriate management – an issue that has been raised in the relevant management literature as well. Veneziano et al. (2014, p. 2), for instance speak of a ‘myth’ of agile working and write that an ‘over-reliance on what has become mainstream practice appears to have impaired critical assessment of whether the specific context (a project, a case study, a problem, an organization etc.) would benefit more from a less commonly used approach’.

Legal and Ethical and Bureaucratic Boundaries

There were a number of ethical and bureaucratic factors that also affected the design of our gamified survey. As standard in academic research with human subjects, we acquired ethical consent from the ethical committee of our university. In addition, Dutch and EU law required us to explicitly inform respondents about the purpose of the research, to ask their consent to use their input for pre-defined research matters, and to take specific measurements for the storage and use of data in order to guarantee respondents’ privacy. Hence, we had to ask participants a series of questions before they entered the gamified survey, such as their age and, for respondents under 16, whether their parents or caretakes gave them permission.

During an early test, respondents mentioned that they would find it invasive when being asked for their postal code, as they felt this could compromise their privacy. Since our purpose to ask for people’s residency was only meant to find out if people coming from urban areas had different privacy concerns and behaviours than people from suburban or rural areas, we chose to let them mention the name of their city or village, leaving us with less detailed location data. In addition, we gave respondents the option not to answer questions for personal information, which meant that we could not include every completed survey in our analysis.

In the wake of the introduction of the GDPR, our university established a privacy office, which informed, assisted and issued procedures to ensure data security. As we contacted the university’s ICT office to host the survey and corresponding data, we received a quote of €10,000 (and €7,000 of yearly recurring costs) for a set of heavily secured servers adding the justification that ‘If you are able to attract 20,000 respondents over the period of a year, then this is ‘only’ €0.50 per respondent’. Besides the fact that we were surprised that we would be charged so heavily for using our own institution’s servers for research purposes, we did not have €10,000 (plus a recurring €7,000) available for this project. As the design and development agency offered a supposedly secure hosting service at 5% of this budget, we gladly accepted this in offer to secure that we could run our survey, that we could contribute to the Weekend of Science’s awareness campaign, and that we could deliver on our academic validation obligations. The data coming from the project have been made publicly available through the national academic data centre DANS.