Evaluating User Engagement with a Reminiscence App Using Cross-Comparative Analysis of User Event Logs and Qualitative Data

Abstract

The aim of this study was to evaluate the usage of a reminiscence app by people living with dementia and their family carers, by comparing event log data generated from app usage alongside the qualitative experience of the process. A cross-comparative analysis of electronic event logging data with qualitative interview data was conducted. Electronic event logging data were obtained for 28 participating dyads (n = 56) and the interview sample comprised 14 people living with dementia and 16 family carers (n = 30). A thematic analysis framework was used in the analysis of interview transcripts and the identification of recurrent themes. The cross-comparison of electronic event log data and qualitative data revealed 25 out of 28 dyads regularly engaged with a reminiscence app, with the analysis of usage patterns revealing four clusters classifying different levels of user engagement. The cross-comparison of data revealed that the nature of the relationship was a significant factor in ongoing user engagement. The comparative analysis of the electronic event logs as “ground truth” in combination with the qualitative lived experience can provide a deeper understanding on the usage of a reminiscence app for those living with dementia and their family carers. This work not only shows the benefits of using automated event log data mining but also shows its clear limitations without using complementary qualitative data analysis. As such, this work also provides key insights into using mixed methods for evaluating human–computer interaction technologies.

Introduction

Reminiscence refers to a range of interventions that prompt memories significant for an individual. Because reminiscence draws primarily on longer term memory, it is widely used as a therapeutic approach for people in the early stages of dementia.^1–3 Reminiscence has become increasingly recognized as a psychosocial intervention that may be as effective as and even preferable to pharmacological interventions, especially to avoid unnecessary medication side effects.^4–7

The use of digital systems to facilitate reminiscing has been shown to be beneficial for people living with dementia.⁸ Technology that facilitates reminiscence increases opportunities for people living with dementia to participate in conversations and to enhance their social interactions.^9,10 Many existing software systems, apps. and social networking Web sites provide opportunities to gather and share multimedia resources.¹¹ However, there is little research into the usability of these systems for reminiscing among people with deteriorating cognitive function.

Lorenz et al.¹¹ conducted a review to explore technology-based services for people living with dementia and their carers and identified the role of technology in supporting therapeutic interventions and home-based reminiscence to reduce caregiver burden.¹² While Lorenz et al.¹¹ acknowledged the significant role technology can play in supporting connection, communication, and independent living, they also highlighted the challenges posed by the ever-changing cognitive status of users. This was supported by other studies recommending that technologies for people living with dementia and their caregivers need to be accessible at the right time, adaptable to changing needs, easy to use, and inexpensive to buy.^13,14 Advances in digital technology have enabled opportunities for supportive interventions, such as reminiscence to be conducted in the home.¹⁴ However, technology-facilitated reminiscence has its challenges, since it relies on the caregiver's willingness to participate and source memorabilia.¹⁵

Many authors have described the learning potential of people living with dementia,^16–18 whereas Riley et al.¹⁹ refer to the inability of people living with dementia to learn new skills. It is noteworthy that in their review on involving people living with dementia in the development of digital applications, Span et al. (2013)²⁰ concluded that cognitive impairment is no reason to exclude people living with dementia from research. Recent research into technology-facilitated reminiscence has shown that it can facilitate opportunities for people living with dementia to retain an empowered role in conversations and relationships.¹ However, as there is a lack of research that evaluates the actual usage and adoption of these technologies among people living with dementia and carers, this study contributes to a deeper understanding of this process.

This research was part of a wider feasibility study designed to investigate the outcomes of a home-based individual-specific reminiscence intervention through the use of a cocreated iPad app known as InspireD for people living with dementia and their family carers, implemented with a paired sample of 30 people living with mild-to-moderate dementia and their family carers. That study is detailed in previously reported work by the authors.²¹ The intervention consisted of reminiscence training, information technology (IT)-based training, and a 12-week period of independent use of the InspireD app to support individual-specific reminiscence in the home.

The objectives of the research were to use machine learning to identify behavioral clusters that typify the different user engagements with the InspireD app; to cross-compare the event log data generated by app usage with quantitative data on previous participant experience; and to contextualize behavioral cluster patterns with the process themes generated from the qualitative interview analysis.

Methods

This study was conducted in a large health and social care trust in the United Kingdom, with recruitment facilitated by trust staff and the U.K. Alzheimer's Society. Event log data were collected from all participating dyads (n = 30) throughout the 12-week period of InspireD app home usage. However, data files for two dyads were corrupted and not used. The reason for the corruptions was identified as technical data communication issues limited to a particular tablet computer used by two of the dyads. Individual interviews were conducted with a volunteer sample of participants at the end of the study (interviews: 16 people living with dementia and 16 family carers).

Table 1 shows the summary of demographic and related information about the participants of the study.

Table 1.

Characteristics of the Participants

Characteristic	Total (N = 60)	Adults living with dementia (N = 30)	Family carers (N = 30)
Age (years)
Range; mean ± SD	31–94; 73 ± 13	61–94; 79 ± 8.9	31–91; 67 ± 14.8
Gender, n (%)
Male	26 (43)	20 (66.7)	6 (20)
Marital status, n (%)
Married	47 (78)	22 (73)	25 (83)
Widowed	9 (15)	8 (27)	1 (3)
Single	3 (5)	0 (0)	3 (10)
Separated	1 (1.7)	0 (0)	1 (3)
Previous IT experience, n (%)
Little or none	35 (58)	24 (80)	11 (37)
Some	21 (35)	5 (17)	16 (53)
A lot	4 (7)	1 (3)	3 (10)
Engagement in hobbies, n (%)
Rarely engaged	12 (20)	11 (37)	1 (3)
Sometimes engaged	38 (63)	17 (57)	21 (70)
Fully engaged	10 (17)	2 (7)	8 (27)
Hobby choices, n (%)
No hobby	5 (8.3)	4 (13)	1 (3)
Social	29 (48)	14 (47)	15 (50)
Creative	7 (12)	4 (13)	3 (10)
Physical fitness	19 (32)	8 (27)	11 (37)

A sample size of 30 dyads was deemed to be sufficient to meet the objectives of the study and represents a significant increase on previous reminiscence studies in the context of dementia. One of the aims of our study is to determine the statistical power across all the parameters to determine the sample sizes that would be required to detect effects in different parts of the analytical model.²² The research team attempted to ensure equity in the recruitment of people living with dementia and their carers in this phase of the study by offering the opportunity of conducting an interview at the end of the 3-month intervention to all participants. Recruitment continued until saturation, where no new themes had been reached at 31 interviews. The research team conducted these interviews no later than 2 weeks after the intervention to maximize the quality of data collected. The total sample size at data saturation was reached at 31 interviews. The total qualitative sample comprised 14 participants living with dementia, 16 carers, and 1 dyad interview. The qualitative data were then matched to the available electronic data of 16 dyads.

Ethical approval for this study was granted from Ulster University's Research Ethics Committee, the regional ORECNI, and the NHS Trust's Research and Development office.

Data collection

Event logging is when each user interaction with an app is automatically logged and stored in a database. This study adopted the health interaction log data analysis pipeline,²³ which involved data cleaning and preparation, as well as the use of exploratory data analysis and K-means clustering, to uncover behavioral patterns of usage by users of the InspireD reminiscence app.

A qualitative approach is an appropriate methodological choice when an understanding of how an individual experienced a phenomenon is necessary.²⁴ In the context of this study, semistructured interviews with participants facilitated the presentation of key trends, through which qualitative findings were compared with the “ground truth” of the multiple strands of electronic data conducted within the wider study.²⁵

Event logging involves storing three variables, namely: (1) the unique user ID to denote who made the interaction, (2) the date and time stamp to denote when the interaction occurred, and (3) the event name that describes the interaction. Individual interviews facilitated a more in-depth exploration of participants' perspectives about key aspects of the study. The researcher used an interview schedule, which included questions on the reminiscence training, IT support, use of the app, and reminiscence activity. The interviews lasted no longer than 45 minutes.

Quantitative data analysis

R studio and the R programming language were used for statistical programming. Hypothesis testing included t-tests and Wilcoxon and chi-square tests where appropriate (p < 0.05 was considered statistically significant). K-means clustering is an unsupervised machine learning technique that was used to uncover the number of archetypal users that existed. Given a series of variables that describe a user, K-means clustering computes clusters of users who have similar characteristics.

Interview questions pertaining to participants' usage patterns provided the team with an invaluable presentation of comparable data from the “ground truth” of the event logging data with participants' previous experience in relation to the intervention. The use of Miles and Huberman's (1994)²⁶ checklist matrix (Table 2) was employed as it not only enabled the exploration of all participants' responses to one key variable but it also enabled the display of an individual participant's responses to a range of variables.

Table 2.

Participant Training and Usage Profiles (PLWD)

Total no. of dyads	Completed reminiscence activity before?	Previous IT experience?	Preferred functionality	Event logging data	Function most commonly used
Total 16 Dyads (n = 30)	14 PLWD-No	12 PLWD-No	3 Dyads enjoyed music/photos	1 Dyad enjoyed music/photos	2 Dyads a.m./early afternoon
Total 16 Dyads (n = 30)	0 PLWD-Yes	2 PLWD-Yes	3 Dyads enjoyed music/photos	1 Dyad enjoyed music/photos	2 Dyads a.m./early afternoon
	13 Carers-No	4 Carers-No	14 Dyads enjoyed photos/music	15 Enjoyed photos/music	15 Dyads late afternoon/p.m.
	3 Carers-Yes	12 Carers-Yes	14 Dyads enjoyed photos/music	15 Enjoyed photos/music	15 Dyads late afternoon/p.m.

PLWD, people living with dementia.

Qualitative thematic analysis

The qualitative stage of analysis was conducted using Braun and Clarke's²⁷ six-phased method of analysis, as this provided the flexibility and responsiveness necessary for a rich and dynamic approach to analysis. Braun and Clarke's^27(p87) six-phased method of analysis comprised the following steps: familiarizing yourself with your data; generating initial codes; searching for themes; reviewing themes; defining and naming themes; and producing the report.

Lincoln and Guba's²⁸ four criteria of credibility, dependability, transferability, and confirmability were used to maximize the credibility of the qualitative data with research team members discussing and agreeing the final themes and subthemes.

Results

Figure 1 shows basic exploratory data analysis of event logs. A total of 71 percent of the interactions from people living with dementia are within the reminiscing screens in the app, which is more than carer interactions (p < 0.001). Figure 1A shows that people living with dementia preferred to interact with photos and reminisce with personalized media. Reminiscing peaked on Thursdays and Fridays but dipped at the weekends. There are peaks of reminiscing at 11 am, 3 pm, and 8 pm.

FIG. 1.

(A) The number of interactions with different types of media for reminiscing, (B) the number of interactions per personal and general media, (C) the number of interactions per hour of the day, and (D) the number of interactions per day.

There is a correlation between the number of days the people living with dementia and carer interacted with the app (r = 0.577, p < 0.001). However, people living with dementia had many more interactions than carers. People living with dementia interacted with the app on 13.73 percent of the days, which is akin to one reminiscence session per week. The median interval between each exclusive day using the app was 7.10 days; hence, if a person living with dementia used the app on a given day, then he or she is not likely to use this again until 1 week later. Figure 2 shows the interaction events over a time line (where the x-axis denotes days). A horizontal jitter was applied to each dot to show the density of interactions per day.

FIG. 2.

Usage patterns of Dyads interacting with the InspireD app-Dyads 1–30 (G = reminiscing using generic media, P = reminiscing using personal media, Admin = carer interactions). For ease of reading, the figure can be viewed online.

Figure 2 shows that each dyad has a unique experience eliciting various usage patterns, where some indicate short bursts of activity. Table 2 illustrates the participants' previous IT and reminiscence training and usage patterns generated from qualitative interviews.

K-means clustering uncovered four clusters or user archetypes. The description of the centroid characteristics of each cluster is provided in Table 3.

Table 3.

A Description of the Four Clusters Using Statistics from the Centroids of Each Cluster, Hence Describing an Average Dyad in That Cluster (PLWD)

Cluster	Description	Cluster size (no. of dyads)	No. of PLWD interactions	No. of carer interactions	No. of PLWD unique day interactions	Mean day interval between interactions	Standard deviation of interval between interactions
1	A PLWD who demonstrates independent and consistent use of the app	1 (4%)	572.00	79.00	46.00	2.00	2.28
2	A PLWD who is reliant on his/her carers for support with the app and exhibits unpredictable usage patterns	12 (43%)	134.83	78.42	12.58	6.61	8.05
3	A PLWD who is highly reliant on his/her carers for engagement with the app. This user has little interaction with extremely unpredictable and inconsistent usage patterns	7 (25%)	38.28	51.14	5.71	19.66	14.71
4	Infrequent user	8 (29%)	127.00	200.25	13.50	6.97	8.99

Figure 3 shows a two-dimensional visualization of the four clusters but does not fully represent the uniqueness of the clusters, given the inability to visualize data points in high-dimensional space.

FIG. 3.

Behavioral usage clusters depicted in two dimensions using principal component analysis.

Figure 4 provides box-plots that show how each individual feature compares across each cluster. While this shows the extent to which each individual feature discriminates between the clusters, one must bear in mind that it is the combination of all features that provides the discrimination in K-means.

FIG. 4.

Each box-plot shows the distribution of each variable in each cluster. This shows the extent to which each individual feature discriminates between the clusters.

Interview findings

The four themes that emerged through analysis were as follows: Cluster 1: “I'm starting to get really good”; Cluster 2: “Able to keep me right”; Cluster 3: “It was a very difficult mountain to climb”; and Cluster 4: “If I wanted to use it…she was here.” These are described in sequence below.

Cluster 1: “I'm starting to get really good”

One person living with dementia repeatedly engaged with the system (3.6 percent or 1 user per 28 users), as he or she had 7.2 times more interactions than his or her carer. While the person living with dementia enthusiastically used the app, the carer showed a normal amount of usage, and hence, the person living with dementia was independently dedicated. The independent person living with dementia and carer used the app for more than half the days in a month (55 percent of days) and, with little variability, used the app every 2 days.

I forgot quite a bit but X (carer) has a good memory and she was able to keep me right. Just the basic things now. I'm starting to get really good. I'm faster as well. (Hugh^*)

Cluster 2: “Able to keep me right”

The majority of people living with dementia (43 percent) fall into this cluster, hence making them the most typical usage profile. These people living with dementia have only 1.7 times more interactions with the app than their carer. This indicates that these people living with dementia have some dependence on the carer for app usage. This dyad uses the app 15 percent of days in a month. This dyad used the app unpredictably, but on average interacts with it every 6.61 days (approximately once per week).

Aye, it was grand. I would be very, very slow in picking a thing up, but I'm getting on to it now. The wife knows a bit about it so she can help me. (Peter)

Cluster 3: “It was a very difficult mountain to climb”

This reflects 25 percent of people living with dementia as they had 25 percent fewer interactions with the app than the carer. These dyads use the app 9 percent of the days in a month and can typically go for 20 days without using it, making them the least consistent users. However, while the people living with dementia had fewer interactions than their carer, they did enjoy using the app.

Well, there's nothing to actually stop me from using it. That's number one. And I like using it. (Denis)

Cluster 4: “If I wanted to use it…she was here”

This reflects 29 percent of people living with dementia—the second largest usage categorization. These people living with dementia had 36 percent fewer interactions with the app than their carers. The carers are very enthusiastic and have more interactions than other carers in all other clusters. Similar to the typical users in Cluster 2, these dyads interact with the app 16 percent of the days in a month and on average use the app every 6.97 days.

He would never, he was always kind of afraid to switch it on himself… He would always say, ‘no I'll wait on X, oh no I can't’… He was afraid, he had this apprehension of … maybe, oh I'd mess things … He thinks he's going to lose the photographs. (Helen)

Discussion

The aim of this article was to evaluate dyad engagement of the InspireD app through a novel cross-comparative analysis of clusters identified from machine learning using k-means clustering, derived from the event logging data generated from app usage, together with the qualitative experience of the process. Event log data for 28 dyads were presented, outlining the behavioral usage of each participant. From this, four clusters of behavioral usage were identified based on five usage characteristics. These data were analyzed with interview data for 16 dyads.

In previous research studies, authors have identified the potential for people with dementia to learn new skills and thereby maintain their quality of life at home. Three of the four clusters derived from the machine learning of the event log data are in concordance with this literature. These three clusters demonstrate that it is the case that people with dementia together with their carers report a sense of gain and self-confidence.

Four themes pertaining to the four clusters of different behavioral usage patterns were identified. The first thematic categorization was Cluster 1: “I'm starting to get really good,” which described the enthusiastic usage of one dyad, particularly the usage patterns of the person living with dementia. Electronic data revealed that the person living with dementia was using the InspireD app 7.2 times more than the carers, which would indicate independent and self-reliant usage.

The second thematic categorization was Cluster 2: “Able to keep me right,” which described the majority of usage patterns for people living with dementia. Event logging data revealed that the people living with dementia engaged with the InspireD app 1.7 times more than their carer, thus showing a level of dependence in their usage. Twelve of the interviewed carers indicated they had some basic previous IT experience. It could therefore be argued that previous carer IT experience was a determinant of InspireD usage. Despite this, all participants found the IT training to be helpful and a significant support within their own usage. This is evident in Figure 2, where electronic usage among Cluster 2 dyads is greatest following the last IT training session and then begins to reduce throughout the duration of home use.

The research team observed that some dyads who were less enthusiastic about the intervention occasionally referred to their difficulty in adapting to a caregiving role. This was confirmed by the event logging data, which thematically categorized three dyads in Cluster 3, also reflected in the theme: “It was a very difficult mountain to climb.” It is interesting to note that none of the Cluster 3 dyads had any previous reminiscence experience, and only two carers had previous IT experience. However, further analysis revealed Cluster 3 people living with dementia found the process of reminiscence training “made me sit up and take notice.” This would indicate that for Cluster 3 people living with dementia engagement was not maintained throughout the intervention.

The event log data indicated some carer disengagement; however, this is not fully captured in the qualitative data. Of the 16 carers interviewed, only three revealed they felt they played a role in poor technology adoption, as “I am not accepting of it” (Ann, Cluster 2). When comparing the qualitative and electronic data, only one carer was categorized in Cluster 3, while the other two carers (Cluster 2) and (Cluster 4) did, in fact, demonstrate considerable support. This could suggest that the carers were either unduly self-critical or did not view their lack of engagement to be a contributing factor for poor usage. This finding resonates with O'Connor et al.'s²⁹ proposition that negatively held carer views on technology were not only a significant limitation on the development of technological programs for people living with dementia but reduced the potential to enhance their relationship through a shared activity as well.

The cross-comparative analysis suggested that those living with dementia felt nervous of the technology and relied on their carer who was “able to keep me right” in Cluster 2 and “If I wanted to use it she was there” in Cluster 4, providing various levels of IT support throughout the duration of home use. As previously proposed,¹⁵ with appropriate carer support, those living with dementia were able to develop a level of independent use as demonstrated in Cluster 1 (one dyad) and Cluster 2 (six dyads). The six dyads in Cluster 4 did require more carer engagement than the other clusters, but this could be indicative of where they are in their own dementia journey.

The main limitation of this work in comparing qualitative interview data with thematic clusters derived from the event log usage of a reminiscing app is that the machine learning algorithms operate on quite a small data set, derived from the small dyad size of the primary study. Another limitation is the loss of data from one dyad, which represents the problems of utilizing real-world data for research purposes.

Conclusions

The novel cross-comparative analysis of clusters derived from machine learning of event log data and qualitative interview data provides significant insights of actual app usage contextualized by the key elements, which facilitated involvement in the intervention and supported engagement for those living with dementia. Carer engagement in the process was vital to support participants living with dementia. However, the cross-comparison of data revealed that the nature of the relationship was a significant factor for dyads less engaged with app usage (Cluster 3).

The article clearly shows the value in future work of combining objective clusters derived from machine learning of event log data and qualitative data to understand user engagement. Both methods are complementary. This work indicates the value of adding qualitative data to enrich the description of clusters elicited by unsupervised machine learning. User event log analysis can inform us of “what patterns exist,” and qualitative interview data can inform us of “why certain patterns exist.” Together, augmenting qualitative data with usage data provides key insights into the benefits of using mixed methods for evaluating human–computer interaction technologies and offers new insights for future research in this area.

Footnotes

Acknowledgments

We acknowledge those living with dementia and their families who participated in the study. We also acknowledge the support of the Reminiscence Network for Northern Ireland, the Alzheimer's Society, and the Community Mental Health Team for Older People in the study site. This work was supported by the Northern Ireland Public Health Agency, Research and Development Division, and The Atlantic Philanthropies (COM/5016/14).

Ethical Approval

The study received ethical approval from the Office for Research Ethics Committees Northern Ireland in February 2016 (16/NI/0035).

Author Disclosure Statement

No competing financial interests exist.

*

All names changed to protect identities

References

Gibson

. (2001) Reminiscence and life story work: a practice guide. London: Jessica Kingsley Press.

Woods

, Orrell

, Bruce

, et al. REMCARE: pragmatic multi-centre randomised trial of reminiscence groups for people with dementia and their family carers: effectiveness and economic analysis. PLoS One, 2016; 11:e0152843.

Subramaniam

, Woods

. The impact of individual reminiscence therapy for people with dementia: systematic review. Expert Review of Neurotherapeutics, 2012; 12:545–555.

Robinson

, Hutchings

, Dickinson

, et al. Effectiveness and acceptability of non-pharmacological interventions to reduce wandering in dementia: a systematic review. International Journal of Geriatric Psychiatry, 2007; 22:9–22.

Fossy

, James

. 2008. Evidence-based approaches for improving dementia care in care homes. London: Alzheimer's Society.

Lawrence

, Fossey

, Ballard

, et al. Improving quality of life for people with dementia in care homes: making psychosocial interventions work. The British Journal of Psycharity, 2012; 201:344–351.

Brooker

, Latham

, Evans

, et al. FITS into practice: translating research into practice in reducing the use of anti-psychotic medication for people with dementia living in care homes. Aging & Mental Health, 2016; 20:709–718.

Lazar

, Thompson

, Demiris

. A systematic review of the use of technology for reminiscence therapy. Health Education and Behaviour, 2014; 41:515–615.

Subramaniam

, Woods

. Digital life storybooks for people with dementia living in care homes. Clinical Interventions in Aging, 2016; 11:1263–1276.

10.

Ryan

, McCauley

, Laird

, et al. 2018. “There is still so much inside”: the impact of home-based, personalised reminiscence, facilitated through an iPad app, on people living with dementia and their family carers. Dementia (London). DOI:10.1177/1471301218795242

11.

Lorenz

, Freddolino

, Comas-Herrera

, et al. Technology-based tools and services for people with dementia and carers: mapping technology onto the dementia care pathway. Dementia (London), 2017; 8:725–741.

12.

O'Shea

, Devane

, Cooney

, et al. The impact of reminiscence on quality of life of residents with Dementia in long stay care. International Journal of Geriatric Psychiatry, 2014; 29:1062–1070.

13.

Mulvenna

, Doyle

, Wright

, et al. Evaluation of card-based versus device-based reminiscing using photographic images. Journal of CyberTherapy and Rehabilitation, 2011; 4:57–66.

14.

Karlsson

, Axelsson

, Zingmark

, et al. “Carpe Diem” Supporting conversations between individuals with dementia and their family members. Journal of Gerontological Nursing, 2014; 40: 38–46.

15.

Sarne-Fleischmann

, Tractinsky

. Development and evaluation of a personalised multimedia system for reminiscence therapy in Alzheimer's patients. International Journal of Social and Humanistic Computing, 2008; 1:8195.

16.

Dechamps

, Fasotti

, Jungheim

, et al. Effects of different learning methods for instrumental activities of daily living in patients with Alzheimer's dementia: a pilot study. Journal of Alzheimer's Disease and Other Dementias. 2011; 26:273–281.

17.

Hanson

, Magnusson

, Arvidsson

, et al. Working together with persons with early stage dementia and their family members to design a user-friendly technology-based support service. Dementia, 2007; 6:411–434.

18.

van Tilborg

, Kessels

RPC

, Hulstijn

. How should we teach everyday skills in dementia? A controlled study comparing and explicit training methods. Clinical Rehabilitation, 2011; 25:638–648.

19.

Riley

, Alm

, Newell

. An interactive tool to promote musical creativity in people with dementia. Computers in Human Behaviour, 2009; 25:599–608.

20.

Span

, Hettinga

, Vernooij-Dassen

, et al. Involving people with dementia in the development of supportive IT applications: a systematic review. Ageing Research Reviews, 2013; 12:535–551.

21.

Laird

LEA

, Ryan

, Mc Cauley

, et al. ‘Using mobile technology to provide personalised reminiscence for people living with dementia and their carers: an appraisal of outcomes from a quasi-experimental study’. JMIR Mental Health, 2018; 5:e57.

22.

Saris

, Satorra

, van der Veld

. Testing structural equation models or detection of misspecifications?. Structural Equation Modelling, 2009; 16:561–582.

23.

Mulvenna

, Bond

, Grigorash

, et al. (2018) Health interaction log data analysis workflow to aid understanding of usage patterns and behaviours. In: The 2nd Symposium on Social Interactions in Complex Intelligent Systems (SICIS) at Artificial Intelligence and Simulation of Behaviour Convention (AISB-2018). Liverpool: Society for the Study of Artificial Intelligence and Simulation of Behaviour.

24.

Creswell

. 2013. Qualitative inquiry and research design: choosing among five approaches. 3rd ed. London: Sage.

25.

Denzin

, Lincoln

. 2013. Strategies of introduction. In Denzin NK, Lincoln YS, eds. Strategies of qualitative inquiry. Thousand Oaks, CA: Sage, pp. 1–35.

26.

Miles

, Huberman

. 1994. Qualitative data analysis: an expanded sourcebook. 2nd ed. Thousand Oaks, CA: Sage Publications.

27.

Braun

, Clarke

. Using thematic analysis in psychology. Qualitative Research in Psychology, 2006; 3:77–101.

28.

Lincoln

, Guba

. 1985. Naturalistic inquiry. Beverly Hills, CA: Sage Publications.

29.

O'Connor

, Bouamrane

, O'Donnell CA Mair

. Barriers to co-designing mobile technology with persons with dementia and their carers. Nursing Informatics 2016. DOI:10.3233/978-1-61499-658-3-1028