Reviewing Health Service and Program Evaluations in Indigenous Contexts: A Systematic Review

Abstract

This study systematically reviewed evidence regarding health program and service evaluations in Indigenous contexts. Following the PRISMA guidelines and combining terms for ‘Indigenous populations’ and ‘health programs and services’. Eight principles emerged: Principle 1: Adopting Indigenous led or co-led approaches is vital to balance power relationships by prioritizing self-determination, Principle 2: Evaluation team should include local Indigenous community members, Principle 3: Indigenous community knowledge and practice should be foundational, Principle 4: Evaluations must be responsive and flexible to meet the needs of the local community, Principle 5: Evaluations should respect and adhere to local Indigenous protocols, culture, wisdom and language, Principle 6: Evaluations should emphasize reciprocity, shared learnings and capacity building, Principle 7: It is important to build strong relationships and trust between and within researcher teams, evaluators and communities, and Principle 8: The evaluation team must acknowledge community capacity and resources by investing in time and relationships.

Keywords

Indigenous peoples Indigenous evaluation cultural responsiveness health systematic review

The evaluation of health services and programs within Indigenous¹ contexts is integral to understanding and improving health policies, programs, and services for Indigenous peoples. As a result of colonial efforts that aimed to eliminate Indigenous governments, ignore historic treaties and land use, and promote assimilation, Indigenous peoples commonly face challenges in achieving recognition as distinct legal, social, and cultural entities with inherent rights. Given this context, evaluations can help improve the responsiveness of a service or program to Indigenous peoples’ needs and inform decisions about policy. Evaluation can also provide insight on the population-specific health impact, relevancy, and sustainability of a particular service or program (Cousins, 2003; Weaver & Cousins, 2007).

The development, implementation, and evaluation of health services and programs are situated within complex systems and influenced by the social, political, historical, and cultural contexts (Chouinard & Cousins, 2007; Davey et al., 2014). Responsiveness to cultural context, as noted by the American Evaluation Association (AEA) Guiding Principles for Evaluators and the Joint Committee on Standards for Educational Evaluation, is thus essential (Chouinard & Cousins, 2007; V. G. Thomas & Parsons, 2017; Ward et al., 2017). Effective evaluation in Indigenous contexts ensures that the local context and Indigenous ways of knowing and being are included within program evaluation and that outcomes reflect the values of the local Indigenous people, communities, and nations (LaFrance & Nichols, 2008).

Due to current and historical evaluation practices, Indigenous communities, clients, participants, and health and program providers often meet for evaluation with contention and suspicion. Externally developed evaluation practices are not always culturally appropriate as they frequently impose Euro-Western worldviews and practices, further oppressing Indigenous ways of knowing and doing (Scott, 2008). For example, transplanted evaluations that are applied to different settings without tailoring or modifying to the community context, or dogmatic adherence to imposed and conventional forms of evaluation, are commonly problematic (Gray et al., 1998; Jan, 1998; Richmond et al., 2008). Overlooking or ignoring the need for contextual modifications can result in faulty theory, restricted scope of measures and methods of data collection (e.g., valuable data source is missed), compromised and unsystematic data collection (e.g., sampling bias), unsuitable language (Gray et al., 1998), and poor response rates and/or biased responses (Bouey & Duran, 2000; Richmond et al., 2008). Any of these factors can undermine the validity and reliability of evaluation in Indigenous contexts (Lowell et al., 2015; McShane et al., 2013; Moore et al., 2014; Patton, 2014).

To contribute to culturally responsive, relevant, useful, and safe evaluation literature in Indigenous contexts, we conducted a systematic review of reviews and evaluations of health service and program in Indigenous contexts to identify evaluation guiding principles. In alignment with the United Nations Declaration on the Rights of Indigenous Peoples (UNDRIP; United Nations, 2007), Article 31 states that “Indigenous peoples have the right to maintain, control, protect and develop their cultural heritage, traditional knowledge and traditional cultural expressions, as well as the manifestations of their sciences, technologies and cultures…,” and the Truth and Reconciliation Commission (TRC) Report of Canada, which includes specific calls to action regarding relevant monitoring and evaluation mechanisms (TRC of Canada, 2015), the aim of this systematic review is to inform and advance program evaluation in Indigenous health services contexts.

Moving toward culturally responsive, relevant, useful, and safe program evaluation in Indigenous contexts is particularly important given the AEA Statement on Cultural Competence in Evaluation, which was approved in April 2011. The Statement informs the AEA Guiding Principles for Evaluators and the significant progress toward culturally safe evaluation that has been made since the Statement’s development. However, cultural competence models, such as the AEA Statement on Cultural Competence in Evaluation, often fail to recognize how Euro-Western evaluation and power structures are constructed in a colonial context and are not culturally safe in Indigenous contexts. Evaluators need to be engaged in working toward cultural safety and critical consciousness. Evaluators frequently draw on methods and theories from disciplines and professions external to Indigenous nations, resulting in the evaluation profession having its own culture and not a culture-free science (AEA & Fairhaven, 2011). Evaluators must be prepared to critique the power structures and to challenge their own culture and cultural systems rather than prioritize becoming “competent” in the cultures of others (Chouinard & Cram, 2019). Further, there are significant differences between cultural safety and concepts such as cultural sensitivity, awareness, and competence (Baskin, 2016; Curtis et al., 2019). Cultural safety moves beyond such concepts and includes reflections on power relations, privilege, status, and oppression and further recognizes that knowledge of language, culture, and being collaborative cannot be equated with culturally safe evaluation (Baskin, 2016). As a result, discussions should not be limited to the cultural competence of the evaluators but must also recognize the culture of evaluation itself (Chouinard & Cram, 2019). Evaluators need to be held accountable for providing culturally safe evaluations, as defined by Indigenous peoples and communities (Freeman, Edwards, et al., 2014; Grover, 2010; Tipene-Leach et al., 2013).

Objective

The objective of this study was to complete a systematic review of the literature on conducting health service and program evaluation in Indigenous contexts. Approaches and methods for undertaking socially, culturally, community-relevant, and scientifically excellent Indigenous health service and program evaluation—or wise practices²—were sought. While these wise practice principles are independent, the AEA Guiding Principles for Evaluators and the AEA Statement on Cultural Competence in Evaluation may supplement the identified principles. Specific research questions guiding the study are as follows:

What are wise practices for evaluating Indigenous health services and programs?

What are wise practice guiding principles for Indigenous health service and program evaluation?

Method

The systematic review was developed and reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (Moher et al., 2009). The PRISMA guidelines are an evidence-based minimum set of items for reporting in systematic reviews. In alignment with the PRISMA guidelines, the systematic review protocol was presented to and vetted by the Indigenous Health Information Knowledge and Evaluation (I-HIKE) Network based in Ontario, Canada. The I-HIKE Network is a regional collaboration of Indigenous health service managers and applied health researchers jointly interested in enhancing Indigenous population health needs assessment data and health program and services evaluation. Further, the Network helped foster an environment for Indigenous community accountability, ensuring that Indigenous peoples led and participated in all stages of the research process. This included formulating the study aims, methods, analysis, and interpretation of results that reflect a uniquely Indigenous perspective of health program and service evaluations in Indigenous contexts.

Sample Selection

Articles were included if they met the following selection criteria. First, articles had to be published or made available in a peer-reviewed journal between 1970 and 2017 as we expected that the majority of the relevant literature was published during this time, including the emergence of evaluations led by Indigenous authors. Second, articles had to report either a majority Indigenous population or have an Indigenous sample that was large enough to produce Indigenous-specific results. The majority of Indigenous population or Indigenous-specific results had to be reported within the article for reviewers to ascertain this information. Third, articles had to be located in health program or service evaluation contexts and include a discussion, reflection, assessment, and/or evaluation of the evaluation. Fourth, articles that described evaluation results were included if a reflection on the evaluation was also described separately (e.g., published in a related article). Fifth, there were no restrictions on the evaluation design or approach. Articles on evaluations using quantitative, qualitative, or mixed methods approaches and case–control, cohort, cross-sectional, experimental, and intervention designs were also included. There were no other geographic, gender, or age restrictions. Articles that were not available in English were excluded as were commentaries, editorials, and letters.

We searched the following databases: Applied Social Sciences Index and Abstracts, Bibliography of Native North Americans, CINAHL, All EBM Reviews (includes ACP Journal Club, Cochrane Central Register of Controlled Trials, Cochrane Database of Systematic Reviews, Cochrane Methodology Register, Database of Abstracts of Reviews of Effects, Health Technology Assessment, National Health Service Economic Evaluation Database), Embase, First Nations Periodical Index, iPortal Indigenous Studies Portal, MEDLINE, Native Health Database, ProQuest Dissertations, PsycINFO, and Social Services Abstracts.

Search terms included a broad range of medical subject headings and key words for “Indigenous” and “health programs and services evaluation” adapted for each database. Forward and backward citation searches were conducted for included articles. The complete MEDLINE search strategy is available upon request.

The databases were initially searched in 2014 and then re-searched in July 2016 and April 2017 by a medical librarian (CZ) to ensure that all relevant literature from 1970 to 2017 was included. Duplicates were removed in EndNote and Mendeley. Titles and abstracts were screened to identify and select relevant articles independently by two authors (RM and RJMM, or RM and GB). Any disagreements between the two reviewing authors were discussed until consensus was reached. Thereafter, full texts of potentially relevant articles were obtained and assessed. Any disagreements between the authors were resolved by consensus and/or discussion with two authors of this article (MF and JS). Mendeley Desktop was used to facilitate the screening of titles, abstracts, and full texts.

Review Strategy

Two authors (RM and MF) independently piloted a data extraction form, and a spreadsheet was created to record and organize information/data from each included article in our systematic review. Data extraction was then completed independently by one of four authors (RM, MF, RJMM, and GB) and cross-checked independently by another author (RM or GB). The following information was extracted from the included articles: publication details (first author, year, title, country of study, funding source), study design, information regarding study methods (aim of study, inclusion/exclusion criteria for study participation, recruitment of participants, randomization procedure, statistical analyses, study limitations), participant characteristics (number of participants, mean/median age, age range, sex, ethnicity), health program or service evaluation details (title, aim, number of intervention/control groups, delivery and setting, underlying theory or theories, evaluation type, community engagement, and integrated iterative community feedback), reflections, discussion, assessment or evaluation, primary and secondary outcomes, and authors’ conclusions.

Quality Assessment

We assessed the quality and noted the limitations and/or bias(es) of the selected articles using the Well Living House quality appraisal tool (Morton Ninomiya et al., 2017; Smylie et al., 2015). As detailed by Morton Ninomiya and colleagues (2017), this tool has three main categories: rigor of evaluation methods, strength of evidence, and relevance to the Indigenous community. Two authors (RM and GB, or RM and RJMM) independently appraised each document, answering the questions on a scale of zero to four for each of the three domains, with a maximum total score of 12. No minimum scores were required. Studies that were rated of lower overall quality on the quality appraisal tool also provided useful information. Limitations are also detailed in the Discussion section of this article.

Analysis

The selected articles were analyzed thematically using a naturalistic, critical, and decolonizing lens (Smylie, 2011). This approach was utilized to ensure a critical examination of power relations between evaluators and Indigenous communities. This decolonizing lens was used to better understand and articulate concepts and frameworks of Indigenous communities who were subject to the evaluations and to attend to historical and ongoing colonial systems. To implement this approach, that is, to critique the power structures and challenge the culture and cultural systems of evaluation, we relied on Indigenous leadership, guidance, and active participation of Indigenous researchers, evaluators, and community health services from the I-HIKE Network. Given that evaluators have generally come from positions of power (Fletcher & Mullett, 2016; Schnarch, 2004; Smylie et al., 2015; United Nations, 2007; West et al., 2016), this approach assisted in fostering an environment for Indigenous peoples to affirm their fundamental right to self-determination and assisted in balancing the power dynamic between Indigenous communities, researchers, and evaluators (United Nations, 2007).

With input from the I-HIKE Network, we identified and categorized key ideas in the reviewed literature, described meaningful patterns and themes, synthesized our findings, and ultimately identified evaluation guiding principles used in Indigenous contexts. Further, in providing direction to ensure that our systematic review met the dual criteria of Indigenous community relevance and scientific excellence, the I-HIKE Network created a pragmatic systematic review that could be used in everyday evaluation practice.

The critical examination and dismantling of individual and systemic power relationship assumptions that drive the suppression of Indigenous knowledges were central to our approach. Our analysis was based on the following assumptions:

The importance of relationality, that is, the interconnection between all things, is foundational. For example, when evaluating health programs, the interconnection between (a) physical, mental, emotional, and spiritual health; (b) all living things; and (c) individual and collective knowledge, practices, and community roles may be central to defining health and well-being. Further, some traditional Indigenous knowledge(s) relate the interconnectedness of all creation; from insects to animals to birds to plants to humans and to the land are interconnected and related as part of the circle of life (Mashford-Pringle & Stewart, 2019; Redvers, 2016). All things and beings that exist on Mother Earth are related as well as connected to each other by virtue of their necessity in the circle of life. This interconnectedness and interdependence are central tenets in Indigenous knowledges and ways of knowing that are taught in many different spaces and places to children, youth, adults, seniors, and Elders. Evaluation takes place within a context of complex relations and interconnections. This context can be fundamentally important; however, such an approach can be in contrast to the modern biomedical scientific traditions which commonly and purposefully decontextualize knowledge from local contexts that have unique relations and interconnections (Smylie, 2011).

Successful Indigenous programs are tailored to, and build upon, local Indigenous values, skills, knowledge, culture, language, and beliefs. This assumption is well aligned with our involvement of Indigenous researchers, evaluators, and health service leaders in the current review and also our application of a critical, naturalistic, and decolonizing lens.

Careful attention to the two previous underlying knowledge assumptions, processes, protocols, and academic community knowledge systems can facilitate innovative and successful Indigenous health policy and program evaluations.

Four authors (RM, AMP, MF, and JS) independently analyzed a subset (n = 14) of included articles. The four authors then met to discuss and reach consensus on key coding themes. Two authors (RM and AMP) independently coded all included studies (n = 132) from 84 identified studies, six sister studies, and eight systematic reviews or some other types of studies. Both authors identified many of the same preliminary key themes and, as such, consensus was reached quickly. Preliminary themes were shared with the larger I-HIKE network to confirm community relevance of the findings. The I-HIKE Network recommended no substantive changes.

Results

Study Selection

A health program and service evaluation study selection decision tree (Figure 1) illustrates that 5,953 articles were identified in the original search, and 132 were included in the study. These 132 studies, which included six sister studies and eight systematic reviews along with some other types of studies, were published between 1990 and 2019. Papers included in the systemic review were predominantly published from Australia, Canada, New Zealand, and the United States.

Figure 1.

Health program and service evaluation study selection decision tree.

Individual Study Characteristics and Results

Study characteristics, including the methods, strengths, and limitations (e.g., methodological limitations), are outlined in the data extraction table. Study characteristics and the respective reviews reflect the diversity of Indigenous communities and Indigenous contexts and provide a range of useful insights to shift toward culturally responsive, relevant, useful, and safe program evaluation.

Synthesis of Results

This systematic review identified many examples of evaluation practices and principles that, when analyzed and synthesized, form eight key guiding principles that need to be considered when conducting evaluation in Indigenous contexts. This section details the eight dynamic and interdependent evaluation guiding principles for Indigenous contexts (Figure 2) that emerged from the thematic analysis of the 132 included articles. As these principles were derived from reviews and evaluations of health program and service evaluation, it is fundamentally important that these principles are not interpreted in isolation, out of context, or as a checklist for evaluation of Indigenous contexts. While acknowledging that the principles should not be interpreted in isolation, the principles are presented in order of priority. Priority was established by considering the crosscutting nature of the principles, weight of evidence, and guidance by the I-HIKE Network to meet the criteria of Indigenous community relevance and scientific excellence in order to help create a pragmatic systematic review for everyday evaluation practice. The literature suggests that a one-size-fits-all approach to evaluation in Indigenous contexts does not exist. Consequently, a range of mechanisms can be used to address the identified principles to various degrees in order to meet the dual criteria of Indigenous community relevance and evaluation excellence, depending on the context of the evaluation as outlined in the guiding principles.

Figure 2.

Evaluation guiding principles for Indigenous contexts.

Principle 1: Adopting Indigenous-Led or Co-Led Approaches is Vital to Balance Power Relationships by Prioritizing Self-Determination

Prioritizing Indigenous self-determination within evaluations can assist in balancing the value of specific knowledge bases, enable the development of common goals and approaches, and identify and reduce prejudices and biases among evaluation team members and within the evaluation process, design, and outcomes (Bouey & Duran, 2000; Kildea et al., 2016; Potvin et al., 1995; Redwood et al., 2016). There are a variety of mechanisms such as governance arrangements, reference and advisory groups, and reporting mechanisms that could be employed to help balance power relationships for Indigenous peoples, communities, service providers, policy and funding bodies, evaluators, and researchers (Fletcher & Mullett, 2016; Schnarch, 2004; Smylie et al., 2015; United Nations, 2007; West et al., 2016). Balancing these relationships and prioritizing local community needs over the requirements and methodologies imposed by external funders are integral to Indigenous self-determination (Barnett & Kendall, 2011; Barwin, 2012; Jacob & Desautels, 2014; Redwood et al., 2016; Tipene-Leach et al., 2013).

There can be significant differences and tensions that arise between (Euro-Western) academic and Indigenous paradigms, epistemologies, ontologies, theories, data collection techniques, contextual analyses and interpretation, and evaluation priorities (Fanian et al., 2015; Jacob & Desautels, 2014; Johnson et al., 2011; McCalman et al., 2015; Pakseresht et al., 2014; Sy et al., 2015). These differences need to be addressed at the conceptual stage of evaluation in order to determine the most suitable approach for the diverse evaluation needs of a particular project (Alvarez et al., 2016; Gray et al., 1998; Tipene-Leach et al., 2013; Voyle & Simmons, 1999).

There is no single approach that will fit with all Indigenous evaluations; however, guidelines and methods can be followed to establish evaluations that prioritize self-determination (Barnett & Kendall, 2011; Jacob & Desautels, 2014; Marley et al., 2014; Redwood et al., 2016). Many authors indicated that evaluation methods built on collaboration, participation, and empowerment are more effective and sustainable than those that position the researcher and members of the Indigenous community as the helper and helpee, respectively. This collaborative approach should be built into the entire evaluation process from initiation through to the implementation and evaluation of recommendations (Broughton et al., 2014; Grover, 2010; Potvin et al., 1995; Tipene-Leach et al., 2013). Collaborative approaches can also help to ensure that the evaluation is practical and improve the trustworthiness throughout the duration of the program or service (Jan, 1998; McShane et al., 2013; Tipene-Leach et al., 2013).

Principle 2: Evaluation Team Should Include Local Indigenous Community Members

Evaluation teams should be comprised primarily of local Indigenous community representatives and ensure that those in community leadership roles are included (Barnett & Kendall, 2011; Barwin, 2012; Brussoni et al., 2012; Pakseresht et al., 2014; Stacy et al., 2014; West et al., 2016). Meaningful community involvement helps ensure that community input and priorities can be fully integrated into the evaluation (Bailie et al., 2015; Janca et al., 2015; Kinchin et al., 2016; McCalman et al., 2015; Pakseresht et al., 2014; Walker et al., 2015). Evaluation must balance Indigenous and non-Indigenous representation, roles, and power within an evaluation team (Bond et al., 2016).

For instance, authors emphasized the importance of recruiting diverse community members who were fluent in Indigenous language(s) and English and who were well known within local networks (Baldwin, 1998; Lowell et al., 2015).

Principle 3: Indigenous Community Knowledge and Practice Should Be Foundational

When evaluation processes are founded in Indigenous communities’ unique and diverse social, cultural, and linguistic characteristics, they are more likely exploring the right evaluation questions. Consequently, responses will be more accurately interpreted, and findings will translate into enhanced services and programs (Broughton et al., 2014; Brussoni et al., 2012; Friesen et al., 2015; McElfish et al., 2015; Mihrshahi et al., 2017; Tingey et al., 2015; Walker et al., 2015). Evaluations should be based on core cultural values that cut across programs and policies (Kim & Driver, 2015; Moran, 1999). This can help mitigate and minimize cultural misunderstandings as well as loss of context and meaning (Friesen et al., 2015; Kypri et al., 2013; Marley et al., 2014; McDermott et al., 2015; Walker et al., 2015).

Several authors argued that Indigenous ways of knowing and doing can be integrated into clinical trials such as randomized controlled trials (RCTs), acknowledging that this can be resource-intensive, including financial resources, time and extensive human resources, and research capacity. RCTs can be challenging within Indigenous communities, partly due to balancing the tension between offering differential access to community resources and Indigenous values around collective sharing of wealth and resources (Kypri et al., 2013; Marley et al., 2014; McDermott et al., 2015; Walker et al., 2015).

Integrating Indigenous community knowledges and practices into the development and testing of data collection tools by ensuring appropriate language, symbols, and communication can positively influence the validity, reliability, and subsequent success of the associated evaluations (Bouey & Duran, 2000; Larzelere-Hinton et al., 2016; Mohatt et al., 2014; Reeve et al., 2016; Townsend et al., 2015). Indigenous and Euro-Western knowledges and ways of being can be reconciled while maintaining intellectual and theoretical rigor using trustworthy and reliable methods of revealing or generating knowledge that is expected of academic research (Baydala et al., 2014). Appropriate evaluation design is critical in balancing Indigenous and Euro-Western knowledges to meet the needs of the community. In this instance, Baydala et al. (2014) used an iterative process to review and alter the focus group guides to capture and build on learning. Solution-focused and strengths-based approaches were cited as more favorable than deficit-based models, and the importance of transparency in selecting evaluation methodologies and methods was cited as being important (Barraza, 2016; Williams et al., 2003).

Finally, evaluation approaches that are not informed by local Indigenous knowledges and practices greatly increase the likelihood of perpetuating inequities in the social distribution of resources and marginalizing or dismissing Indigenous knowledges, voices, and practices (Baydala et al., 2014; Genuis et al., 2015; Gray et al., 1998; Williams et al., 2003).

Principle 4: Evaluations Must Be Responsive and Flexible to Meet the Needs of the Local Community

The historical and sociopolitical context in which a person, community, program, or policy operates has great influence on program and policy outcomes (Askew et al., 2016; Cuesta-Briand et al., 2015; Davey et al., 2014; Helitzer et al., 1999; McShane et al., 2013; Sahota & Kastelic, 2012; Schoen et al., 2010). Evaluations that meet the needs of the local community must include evaluation designs that reflect, respond, and adapt to local Indigenous culture and ways of knowing (Macaulay et al., 1997; McShane et al., 2013; Potvin et al., 1995). This insight was the most common theme in our systematic review (Askew et al., 2016; Cuesta-Briand et al., 2015; M. Davey et al., 2014; Helitzer et al., 1999; Macaulay et al., 1997; McShane et al., 2013; Potvin et al., 1995; Sahota & Kastelic, 2012; Schoen et al., 2010).

Evaluators must not treat Indigenous people as a homogenous group. There are many diverse Indigenous nations, languages, and cultural practices worldwide, both within countries (Baldwin, 1998; Davey et al., 2014; Tipene-Leach et al., 2013; Voyle & Simmons, 1999) and within local Indigenous communities (Baldwin, 1998; Barnett & Kendall, 2011; Grover, 2010, 2016). For example, within Indigenous social systems, respect for individual autonomy of decision making can be an important value, further supporting individual diversity within a community, though it is commonly balanced with a foundational regard for the well-being of the collective (Brant, 1990).

Principle 5: Evaluations Should Respect and Adhere to Local Indigenous Protocols, Culture, Wisdom, and Language

Local Indigenous community ways of knowing and being should be integrated into evaluation logic models, conceptual frameworks, and evaluation designs (Cinelli & Peralta, 2015; Doyle et al., 2016; Freeman, Edwards, et al., 2014; Hayward et al., 2017; Schoen et al., 2010; Sy et al., 2015; Tingey et al., 2015). As part of adhering to local Indigenous protocols, culture, and language, authors also identified the importance of incorporating the local community history and values defined by community members into evaluation work (Brussoni et al., 2012; Gray et al., 1998; Janssen, 2008; Kim & Driver, 2015; Tipene-Leach et al., 2013; Voyle & Simmons, 1999; Ziabakhsh et al., 2016).

The articles cited several common methods and mechanisms for facilitating respect and adherence to local Indigenous protocols, culture, and language. Examples included holding community gatherings such as feasts, sports gatherings, and dances; asking community opinion leaders such as Elders and Knowledge Keepers to share their knowledges and/or practices; and involving community members in developing and sharing evaluation materials in languages and formats that reflect community knowledge sharing systems (Alvarez et al., 2016; Maksimovic et al., 2015; McKenzie, 1997; Towns et al., 2014; Young et al., 2016). Further, some articles suggested involving interpreters and translating evaluation materials into relevant languages (Gray et al., 1998; McShane et al., 2013; Tipene-Leach et al., 2013).

Principle 6: Evaluations Should Emphasize Reciprocity, Shared Learnings, and Capacity Building

Evaluations should equitably distribute the burden and benefits of the research and evaluation, including resources and responsibilities (National Health and Medical Research Council, 2003). Reciprocity highlights the notion of evaluation and research as part of a gift exchange, including the gift of skills, knowledge, and experience (Barraza, 2016). Reciprocity and kinship (relationship to ancestry) are grounded in Indigenous peoples’ traditional values. Reciprocity allows for continued commitment to traditional values and respects self-identification, as it allows Indigenous people to come to know and be accepted by the community (Gunn, 2015). Shared learnings and capacity building are important forms of providing community members with the opportunity to be trained in evaluation and research, while outside evaluators can also gain knowledge about local history, practices, values, and knowledge systems (Arnold, 1991; Buckskin et al., 2013; Castellano, 2000; Cresp et al., 2016; Shah et al., 2015).

Information for and from evaluation must be shared effectively using accessible language and formats that enable the clear communication of theoretical and conceptual materials, study designs, results, and findings to all audiences including community members. Effective communication is essential to shared learnings, capacity building, and research skill development for community health representatives, Indigenous health workers, and researchers alike (Arnold, 1991; Buckskin et al., 2013; Clark et al., 2015; Cresp et al., 2016; Pakseresht et al., 2014; J. Redvers et al., 2015; Shah et al., 2015).

Capacity building within Indigenous communities requires adequate human resources, experience, time, and funding for trained community members to access and meaningfully examine data (Grover, 2010). If evaluators do not have adequate Indigenous community experience and skills to work with communities, evaluators may require their own capacity building to ensure sound communication and community engagement skills are effective. The process of sharing knowledges and capacity building may help facilitate and legitimize the research outcomes within the prevailing Indigenous community (Barnett & Kendall, 2011).

Principle 7: It Is Important to Build Strong Relationships and Trust Between and Within Researcher Teams, Evaluators, and Communities

Good relationships between researchers and community members can help identify common ground, develop shared goals and approaches, and balance tensions and power dynamics (Bouey & Duran, 2000; Eskicioglu et al., 2014; Freeman, Jolley, et al., 2014; Harlow & Clough, 2014; Kildea et al., 2016; Kulis et al., 2015; Kypri et al., 2013; Lowell et al., 2015; Okamoto et al., 2016; Potvin et al., 1995; Redwood et al., 2016; Snijder et al., 2015; Yuskevich, 2010).

Many articles discussed trust, or rather lack of trust, as an evident issue (Barnett & Kendall, 2011; Grover, 2010; McKenzie, 1997; Stacy et al., 2014; Voyle & Simmons, 1999). The words “cynicism,” “distrust,” and “suspicion” were frequently used to describe Indigenous community feelings (Chesterton, 2003; Freeman, Edwards, et al., 2014; Grover, 2010; McShane et al., 2013; Mercer et al., 2013). Factors that triggered or fueled distrust, cynicism, and suspicion by Indigenous community members toward researchers/evaluators included the following: (a) colonial practices; (b) issues of mistrust in government and colonial institutions; (c) government and funders undermining self-determination by controlling/or issuing grants and imposing restrictions on successful programs; (d) resentment of externally imposed processes that delay funding; (e) adhering funding to compliance; (f) inadequate funding; (g) experiences of participating in research that offered no benefits to participants or community; (h) observations of evaluators and researchers who advanced their own academic status and careers while disadvantaging communities; (i) perceptions that bureaucrats and health managers engaged in evaluation with self-serving agendas; (j) lack of transparency around funding and data ownership; and (k) impressions that external stakeholders did not understand, respect, or value community interests and involvement (Chesterton, 2003; Freeman, Edwards, et al., 2014; Grover, 2010; McShane et al., 2013; Voyle & Simmons, 1999).

Grover (2010) argued that evaluators must demonstrate qualities of respect, honesty, and tact. Other authors suggested that evaluators must invest time and resources into building trust, fostering relationships, and capacity building (Brussoni et al., 2012; Maksimovic et al., 2015; Redvers et al., 2015; S. L. Thomas et al., 2015). In some cases, interpersonal relationships and loyalty were valued by community more than efficiency, timeliness, and objectivity (Grover, 2010). Investing in interpersonal relationships, in itself, is a cultural value that made it possible for Indigenous and non-Indigenous people to collaborate and conduct shared decision making (Davey et al., 2014; Grover, 2010; McShane et al., 2013).

Kinship value systems and social structures can assist in balancing power relationships and facilitating the building of trust and respect while developing common goals and approaches to evaluation. Such systems and structures can also assist in holding the evaluation accountable to the community and vice versa in a reciprocal relationship (Freeman, Edwards, et al., 2014; Grover, 2010; Tipene-Leach et al., 2013). Further, if the research team and the community are related as kin, then accountability is intrinsic (Freeman, Edwards, et al., 2014; Grover, 2010; Tipene-Leach et al., 2013).

Principle 8: The Evaluation Team Must Acknowledge Community Capacity and Resources by Investing in Time and Relationships

Indigenous communities and evaluators hold different types of knowledge, expertise, capacity, and resources (Barnett & Kendall, 2011; Barwin, 2012; Brussoni et al., 2012; Tipene-Leach et al., 2013; Voyle & Simmons, 1999). External evaluators may be unfamiliar with the local Indigenous context, so they may not recognize community assets and resources and unwittingly promote dominant Euro-Western knowledges and assess Indigenous communities through an epistemically racist lens (Baydala et al., 2014; Gray et al., 1998; Grover, 2010; Williams et al., 2003). While there are evaluators that practice culturally responsive approaches (Chouinard & Cousins, 2007; V. G. Thomas & Parsons, 2017), evaluation needs to be held accountable for providing culturally safe evaluations as defined by Indigenous communities. Evaluators cannot continue to helicopter in with some phone calls and emails as relationship building and then conduct research that is not culturally relevant and/or culturally safe, thus producing results that do not take into account community(s) knowledges, resources, or ways of analyzing data to provide accurate and relevant findings and recommendations. Further, evaluation guidelines and practices have been developed, for the most part, external to Indigenous communities, and community members may not be familiar with such evaluation guidelines and practices (Baydala et al., 2014; Gray et al., 1998; Grover, 2010; Williams et al., 2003). General models and funding agencies often underestimate the amount of time that is needed to establish and integrate capacity strengthening processes into evaluations communities (Baydala et al., 2014; Cousins, 2003; Grover, 2010; Scott, 2008; Weaver & Cousins, 2007; Williams et al., 2003). However, time must be taken and resources expended to build rapport, seek community advice and input, assess community needs; plan data collection; develop recommendations (Grover, 2010); and decide how best to build an understanding of local Indigenous knowledge, values, and practice among external community evaluators.

Considerable resources are required to build trust, facilitate effective communication between all parties, and promote inclusive working relationships, particularly given the complexity, time, and human resources needed to conduct a rigorous, relevant, and comprehensive evaluation. Commonly, human resource costs to Indigenous communities participating in evaluations are relatively great, and people are often expected to actively participate with minimal or no personal or financial support (Gray et al., 1998; Grover, 2010; Voyle & Simmons, 1999). Evaluators should be aware that communities and community practitioners may not have the time, knowledge, skills, and resources to support a rapid evaluation (Baydala et al., 2014; Gray et al., 1998; Grover, 2010; Williams et al., 2003). Health practitioners working within underfunded health services and programs are often struggling to respond to crises on a day-to-day basis. This can make it difficult to engage in evaluation in a meaningful way (Grover, 2010; Palmater, 2011). Evaluators do not always consider the limited resources facing communities, such as limited infrastructure, food and water insecurity, human resources gaps, crowded substandard housing, limited meaningful employment, the nature of health disparities, and the need for evaluation design and budgeting to challenge and address such inequities.

Discussion

Consistent with the work of LaFrance and Nichols (2008), the eight evaluation guiding principles for Indigenous contexts can help support the planning and implementation of evaluations that have enhanced Indigenous community social value. As outlined, the active fostering of collaborative relationships; the centering of Indigenous knowledges and values; and the recognition of the relationship between culture, context, and health program and service evaluation are of central importance to Indigenous evaluation. Further, the context of Indigenous communities is unique, including imbalanced power dynamics mainly created to serve the central goals of active government assimilation policies (TRC of Canada, 2015). While this certainly distinguishes Indigenous communities from other populations, there is some common alignment with practices identified for other culturally responsive evaluations (Chouinard & Cousins, 2007; V. G. Thomas & Parsons, 2017; TRC of Canada, 2015). In order to assess responsiveness to cultural context, evaluation approaches need to acknowledge and address power between stakeholders, particularly when they are situated within circumstances of power imbalances, dislocation, and dispossession (Chouinard & Cousins, 2007; TRC of Canada, 2015).

The need to build trust is a consistent and crosscutting theme across the eight guiding principles. The principles require funders and evaluators to respect the need for monetary resources to ensure community participation and engagement, such as paying stipends or honorariums, providing food, and respecting and adhering to local protocols, such as gifting practices, as well as the added time to build relationships and trust.

Another crosscutting theme was the need for cultural and academic rigor to meet the dual criteria of Indigenous community relevance and scientific excellence in undertaking evaluations in Indigenous contexts. While scientific excellence is commonly detailed in evaluation principles, the foundational ethical principle of undertaking research and evaluation of Indigenous community relevance is less frequently detailed. This systematic review is precisely aimed at identifying evidence of how evaluation principles in Indigenous communities have been effective.

Our review confirms that within Indigenous health service and program evaluations, evaluation must be culturally safe within Indigenous contexts and considerations must be genuine, account for and address power dynamics, and move beyond sociodemographic health and well-being indicators for communities and programs that reflect local Indigenous concepts of wellness (Senese, 2005; SenGupta et al., 2004). There is also a need to engage in more substantive discussion and actions to understand and address power and politics in Indigenous and cross-cultural evaluation using self-determination and participatory methods (Chouinard & Cousins, 2007; Fanian et al., 2015; LaFrance & Nichols, 2008; Redwood et al., 2016; Tipene-Leach et al., 2013).

Evaluations should be based on the reality of a program or policy and not modeled purely on theory or evaluated within a vacuum (Chouinard & Cousins, 2007; Oser et al., 2013). Further, Indigenous lived experiences in programs and services and their daily lives are not similar to non-Indigenous peoples. As such, it is imperative that evaluations be based on the local Indigenous lived experience and realities that programs and services are not offered based purely on theory or evaluated within a vacuum. For example, a program may have inadequate resources, or the community may be experiencing a crisis that renders a program less safe or relevant at a specific time. Therefore, evaluations of these programs or services need to be grounded in the daily lived experiences of Indigenous peoples who are facing oppressive sociopolitical practices. In the same way, Indigenous-specific evaluation must be grounded in local community contexts, hedged from oppressive sociopolitical practices intended to erase Indigenous ways of knowing and being, to ensure that findings and outcomes are Indigenous-led, accurate, rigorous, relevant, and reliable on a shared set of values (community and evaluators’ values; Dionne et al., 2014; Oser et al., 2013; Pett & Nye, 2015; Steglich et al., 2012).

Indigenous-informed evaluation results can empower health services and communities to exercise self-determination in improving policies, programs, and services. Such approaches are critical for cultural continuity to nurture and advance authentic Indigenous knowledges and knowledge development practices across generations that are authentically grounded in local community. Through Indigenous voice and agency, Indigenous communities can immediately use evaluation knowledge to enhance policies, planning, and programming to better meet community needs in ways that are theoretically aligned and pragmatically accessible to local communities. Evaluation is only as good as the outcomes it achieves, and ultimately, the evaluation of health programs and services in Indigenous contexts should improve the health and well-being of Indigenous populations.

Strengths and Limitations

Key strengths of this study were that Indigenous peoples led and participated in all stages of the research process. This included formulating study aims, methods, analysis, and interpreting results. As such, this research reflects a uniquely Indigenous perspective with strong lived and Indigenous community program/service evaluation experience. In this systematic review, we identified, synthesized, and analyzed evaluations and reflections regarding Indigenous health service and program evaluations on a global scale. “Principle 1: In evaluating health policy and programs, it is vital to balance power relationships by prioritizing self-determination” identified and articulated the importance of self-determination, as outlined in the UNDRIP. The UNDRIP stresses the importance of Indigenous peoples’ right to self-determination, which includes the right “to freely determine their political status and freely pursue their economic, social, and cultural development” and affirms Indigenous peoples’ right “to autonomy or self-government in matters relating to their internal and local affairs.” With this context in mind and as outlined in “Principle 5: Evaluations should respect and adhere to local Indigenous protocols, culture, wisdom, and language,” wise practices must take this context into consideration when evaluating health services and programs in Indigenous contexts (Morton Ninomiya et al., 2017; Ward et al., 2017).

The broad definition of the search terms “health programs” and “health services” enabled us to capture both structured and unstructured reflections on evaluations in Indigenous contexts. However, this also meant that a diversity of studies, including a wide range of methods and Indigenous health topics, met the systematic review inclusion criteria. Not all included studies contained detailed information on specific Indigenous community contexts where the evaluations took place, the evaluation methods, or evaluative feedback regarding the strengths and limitations of the respective evaluations. Further, while the Review Strategy section included evaluation methodological components, given the primary aim of the research as well as the diversity of included evaluations and evaluation contexts, analysis of the methodological components was beyond the scope of this article. There is an acute concern that some of the most exemplary Indigenous health service and program evaluations may not have been reported in the indexed academic literature due to academic publication barriers (McGrail et al., 2006). In addition, the broad definition of health aligns more closely with Indigenous definitions of health. Specifically, health does not just denote the physical well-being of an individual, but rather, health refers to the social, emotional, and cultural well-being of the whole community in which each individual is able to achieve their full potential as a human being, thereby bringing about the total well-being of the community (Greenwood et al., 2015; National Aboriginal Health Strategy Working Party, 1989). As a result, the identified principles can be applied to the design, conduct, and implementation of evaluations with Indigenous peoples but should also include broader well-being outcomes.

In recognition of the barriers to academic publishing, a number of parallel initiatives are aimed at identifying information about health services and program evaluation in Indigenous contexts that are not contained in indexed publications. For example, the authors are conducting a search of the gray literature and online sources, as well as a panel of experts from Canada (Chandna et al., 2019; Well Living House, 2017). Further, Indigenous views and perspectives have been shared in interpreting the findings from included articles. However, this does not equate to Indigenous evaluation principles but evaluation principles derived from Indigenous contexts. Overall, this systematic review raises questions from an evaluative perspective about rigor and relevance, participatory approaches, power dynamics and its effects on relationships, and the perspectives of communities in focus. Further work is recommended to better understand the health service and program evaluation experiences of Indigenous populations including in lower- and middle-income countries.

Conclusion

This systematic review provides useful insight for health service and program evaluation for Indigenous contexts, reflecting a uniquely Indigenous perspective. While there are gaps in the literature, we were able to corroborate findings from the existing literature to identify eight interrelated guiding principles for evaluation for Indigenous contexts. The literature supports the consideration and application of these principles when planning and implementing evaluations with Indigenous communities.

We encourage policy and funding bodies and evaluators to start applying these guiding principles and to track and share the practices and impacts of this application. Working in this way, our understanding of scientifically rigorous and wise practices for Indigenous health service and program evaluation will advance and in turn will improve health programs and policies for Indigenous peoples.

Footnotes

Acknowledgments

We would like to acknowledge and thank the Indigenous Health Information Knowledge and Evaluation (I-HIKE) Network for their constructive and invaluable feedback. The I-HIKE Network is a regional collaboration of Indigenous health service managers and applied health researchers jointly interested in enhancing Indigenous population health needs assessment data and health program and services evaluation based in Ontario, Canada. This systematic review protocol was reported to and vetted by the I-HIKE Network.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by the provincial Ministry of Health and Long Term Care (MOHLTC).

ORCID iDs

Raglan Maddox

Carolyn Ziegler

Notes

References

Alvarez

G. G.

Van Dyk

D. D.

Colquhoun

Moreau

K. A.

Mulpuru

Graham

I. D.

(2016). Developing and field testing a community based youth initiative to increase tuberculosis awareness in remote Arctic Inuit communities. PLoS ONE, 11(7), e0159241.

Arnold

N. L.

(1991). Program evaluation of the Confederated Salish and Kootenai Tribes’ Indian Vocational Rehabilitation Services Project. University of Montana. http://bf4dv7zn3u.search.serialssolutions.com/?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&rfr_id=info:sid/ProQuest+Dissertations+&+Theses+Full+

Askew

D. A.

Togni

S. J.

Schluter

P. J.

Rogers

Egert

Potter

Hayman

N. E.

Cass

Brown

A. D.

(2016). Investigating the feasibility, acceptability and appropriateness of outreach case management in an urban Aboriginal and Torres Strait Islander primary health care service: A mixed methods exploratory study. BMC Health Services Research, 16(1), 1.

Bailie

Schierhout

Laycock

Kelaher

Percival

O’Donoghue

McNeair

Bailie

(2015). Determinants of access to chronic illness care: A mixed-methods evaluation of a national multifaceted chronic disease package for Indigenous Australians. BMJ Open, 5(11), e008103.

Baldwin

J. A.

(1998). Conducting drug abuse prevention research in partnership with Native American communities: Meeting challenges through collaborative approaches. Drugs & Society, 14(1–2), 77–92.

Barnett

Kendall

(2011). Culturally appropriate methods for enhancing the participation of Aboriginal Australians in health-promoting programs. Health Promotion Journal of Australia, 22(1), 27–32. http://ovidsp.ovid.com/ovidweb.cgi?T=JS&CSC=Y&NEWS=N&PAGE=fulltext&D=medl&AN=21717834

Barraza

(2016). Indigenous youth-developed self-assessment: The Personal Balance Tool. American Indian and Alaska Native Mental Health Research(Online), 23(3), 1.

Barwin

(2012). Places of tradition, places of research: The evaluation of traditional medicine workshops using culturally and locally relevant methods. Université d’Ottawa/University of Ottawa.

Baskin

(2016). Strong helpers’ teachings: The value of Indigenous knowledges in the helping professions. Canadian Scholars’ Press.

10.

Baydala

Fletcher

Worrell

Kajner

Letendre

Rasmussen

(2014). Partnership, knowledge translation, and substance abuse prevention with a First Nations community. Progress in Community Health Partnerships: Research, Education, and Action, 8(2), 145–155.

11.

Bond

Foley

Askew

(2016). “It puts a human face on the researched”—A qualitative evaluation of an Indigenous health research governance model. Australian and New Zealand Journal of Public Health, 40(S1), S89–95.

12.

Bouey

P. D.

Duran

B. E.

(2000). The Ahalaya case-management program for HIV-infected American Indians, Alaska Natives, and Native Hawaiians: Quantitative and qualitative evaluation of impacts. American Indian and Alaska Native Mental Health Research(Online), 9(2), 36.

13.

Brant

C. C.

(1990). Native ethics and rules of behaviour. The Canadian Journal of Psychiatry, 35(6), 534–539.

14.

Broughton

Person

Maipi

Cooper-Te

Smith-Wilkinson

Tiakiwai

Kilgour

Berryman

Morgaine

Jamieson

(2014). Ukaipo niho: The place of nurturing for oral health. The New Zealand Dental Journal, 110(1), 18–23.

15.

Brussoni

Olsen

L. L.

Joshi

(2012). Aboriginal community-centered injury surveillance: A community-based participatory process evaluation. Prevention Science, 13(2), 107–117.

16.

Buckskin

Kit

J. A.

Glover

Mitchell

Miller

Weetra

Wiebe

Yelland

J. S.

Newbury

Robinson

(2013). Aboriginal Families Study: A population-based study keeping community and policy goals in mind right from the start. International Journal for Equity in Health, 12(1), 1.

17.

Castellano

M. B.

(2000). Updating Aboriginal traditions of knowledge. University of Toronto Press.

18.

Chandna

Vine

M. M.

Snelling

Harris

Smylie

Manson

(2019). Principles, approaches, and methods for evaluation in indigenous contexts: A grey literature scoping review. Canadian Journal of Program Evaluation, 34(1), 21–47.

19.

Chesterton

(2003). Balancing ethical principles in evaluation: A case study. Canadian Journal of Program Evaluation, 18(1), 49–60. http://ovidsp.ovid.com/ovidweb.cgi?T=JS&CSC=Y&NEWS=N&PAGE=fulltext&D=psyc4&AN=2003-99532-002

20.

Chouinard

Cousins

(2007). Culturally competent evaluation for Aboriginal communities: A review of the empirical literature. Journal of Multidisciplinary Evaluation, 4(8), 40–57.

21.

Chouinard

Cram

(2019). Culturally responsive approaches to evaluation: Empirical implications for theory and practice (Vol. 4). Sage.

22.

Cinelli

Peralta

(2015). “Achievement, pride and inspiration”: Outcomes for volunteer role models in a community outreach program in remote Aboriginal communities. Rural and Remote Health, 15, 3482.

23.

Clark

R. A.

Fredericks

B. L.

Buitendyk

Adams

M. J.

Howie-Esquivel

Dracup

Berry

Atherton

Taylor-Johnson

(2015). Development and feasibility testing of an education program to improve knowledge and self-care among Aboriginal and Torres Strait Islander patients with heart failure. Rural and Remote Health, 15(3231), 1–10.

24.

Cousins

J. B.

(2003). Utilization effects of participatory evaluation. In Kellaghan

Stufflebeam

D. L.

(Eds.), International handbook of educational evaluation (pp. 245–265). Springer.

25.

Cresp

Clarke

McAuley

K. E.

McAullay

Moylan

C. A.

Peter

Chaney

G. M.

Cook

Edmond

K. M.

(2016). Effectiveness of the Koorliny Moort out-of-hospital health care program for Aboriginal and Torres Strait Islander children in Western Australia. The Medical Journal of Australia, 204(5), 197.

26.

Cuesta-Briand

Bessarab

Shahid

Thompson

S. C.

(2015). Addressing unresolved tensions to build effective partnerships: Lessons from an Aboriginal cancer support network. International Journal for Equity in Health, 14(1), 1.

27.

Curtis

Jones

Tipene-Leach

Walker

Loring

Paine

S.-J.

Reid

(2019). Why cultural safety rather than cultural competency is required to achieve health equity: A literature review and recommended definition. International Journal for Equity in Health, 18(1), 174.

28.

Davey

C. J.

McShane

K. E.

Pulver

McPherson

Firestone

, & Ontario Federation of Indian Friendship Centres. (2014). A realist evaluation of a community-based addiction program for urban aboriginal people. Alcoholism Treatment Quarterly, 32(1), 33–57.

29.

Davey

Moore

Walters

(2014). Tasmanian Aborigines step up to health: Evaluation of a cardiopulmonary rehabilitation and secondary prevention program. BMC Health Services Research, 14(1), 349.

30.

Dionne

McKinnon

Squires

Clifford

(2014). Developmental screening in a Canadian First Nation (Mohawk): Psychometric properties and adaptations of ages & stages questionnaires. BMC pediatrics, 14(1), 23.

31.

Doyle

Atkinson-Briggs

Atkinson

Firebrace

Calleja

Reilly

Cargo

Riley

Crumpen

Rowley

(2016). A prospective evaluation of First people’s health promotion program design in the Goulburn-Murray rivers region. BMC Health Services Research, 16(1), 645.

32.

Eskicioglu

Halas

Sénéchal

Wood

McKay

Villeneuve

Shen

G. X.

Dean

McGavock

J. M.

(2014). Peer mentoring for type 2 diabetes prevention in First Nations children. Pediatrics, 133(6), e1624–e1631.

33.

Fanian

Young

S. K.

Mantla

Daniels

Chatwood

(2015). Evaluation of the Kts’ iìhtła (“We Light the Fire”) Project: Building resiliency and connections through strengths-based creative arts programming for Indigenous youth. International Journal of Circumpolar Health, 74(1), 27672.

34.

Fletcher

Mullett

(2016). Digital stories as a tool for health promotion and youth engagement. Canadian Journal of Public Health, 107(2), 183–187.

35.

Freeman

Edwards

Baum

Lawless

Jolley

Javanparast

Francis

(2014). Cultural respect strategies in Australian Aboriginal primary health care services: Beyond education and training of practitioners. Australian and New Zealand Journal of Public Health, 38(4), 355–361. https://doi.org/10.1111/1753-6405.12231

36.

Freeman

Jolley

Baum

Lawless

Javanparast

Labonté

(2014). Community assessment workshops: A group method for gathering client experiences of health services. Health & Social Care in the Community, 22(1), 47–56.

37.

Friesen

B. J.

Cross

T. L.

Jivanjee

Thirstrup

Bandurraga

Gowen

Rountree

(2015). Meeting the transition needs of urban American Indian/Alaska Native youth through culturally based services. The Journal of Behavioral Health Services & Research, 42(2), 191–205.

38.

Genuis

S. K.

Willows

Nation

A. F.

Jardine

C. G.

(2015). Partnering with Indigenous student co-researchers: Improving research processes and outcomes. International Journal of Circumpolar Health, 74(1), 27838.

39.

Gray

Sputore

Walker

(1998). Evaluation of an Aboriginal Health Promotion Program: A case study from Karalundi. Health Promotion Journal of Australia, 8(1), 24–28. http://myaccess.library.utoronto.ca/login?url=http://search.proquest.com/docview/61664883?accountid=14771

40.

Greenwood

De Leeuw

Lindsay

N. M.

Reading

(2015). Determinants of Indigenous peoples’ health. Canadian Scholars’ Press.

41.

Grover

J. G.

(2010). Challenges in applying Indigenous evaluation practices in mainstream grant programs to Indigenous communities. The Canadian Journal of Program Evaluation, 23(2), 33–50.

42.

Gunn

B. L.

(2015). Defining metis people as a people: Moving beyond the Indian/Metis dichotomy. Dalhousie Law Journal, 38, 413.

43.

Harlow

A. F.

Clough

(2014). A systematic review of evaluated suicide prevention programs targeting indigenous youth. Crisis: The Journal of Crisis Intervention and Suicide Prevention, 35(5), 310.

44.

Hayward

Lyndon

Villa

Madell

Elliot-Hohepa

Le Comte

(2017). My Home is My Marae: Kaupapa Māori evaluation of an approach to injury prevention. BMJ Open, 7(3), e013811.

45.

Helitzer

D. L.

Davis

S. M.

Gittelsohn

Going

S. B.

Murray

D. M.

Snyder

Steckler

A. B.

(1999). Process evaluation in a multisite, primary obesity-prevention trial in American Indian schoolchildren. The American Journal of Clinical Nutrition, 69(4), 816s–824s.

46.

Jacob

Desautels

(2014). Assessing the quality of Aboriginal program evaluations. The Canadian Journal of Program Evaluation, 29(1), 62.

47.

Jan

(1998). A holistic approach to the economic evaluation of health programs using institutionalist methodology. Social Science & Medicine, 47(10), 1565–1572.

48.

Janca

Lyons

Balaratnasingam

Parfitt

Davison

Laugharne

(2015). Here and Now Aboriginal Assessment: Background, development and preliminary evaluation of a culturally appropriate screening tool. Australasian Psychiatry, 23(3), 287–292.

49.

Janssen

J. M. E.

(2008). Meeting the needs of Maori with diabetes: An evaluation of a nurse-led service. Victoria University of Wellington.

50.

Johnson

Morales

Leavitt

Carry

Kinnon

Rideout

Clarida

(2011). Pan-Arctic TV Series on Inuit wellness: A northern model of communication for social change? International Journal of Circumpolar Health, 70(3), 236–244. http://ovidsp.ovid.com/ovidweb.cgi?T=JS&CSC=Y&NEWS=N&PAGE=fulltext&D=medl&AN=21703132

51.

Kildea

Gao

Rolfe

Josif

C. M.

Bar-Zeev

S. J.

Steenkamp

Kruske

Williams

Dunbar

Barclay

L. M.

(2016). Remote links: Redesigning maternity care for Aboriginal women from remote communities in Northern Australia—A comparative cohort study. Midwifery, 34, 47–57.

52.

Kim

Driver

D. D.

(2015). Teleophthalmology for First Nations clients at risk of diabetic retinopathy: A mixed methods evaluation. JMIR Medical Informatics, 3(1), e10.

53.

Kinchin

Jacups

Hunter

Rogerson

(2016). Economic evaluation of “Return to Country”: A remote Australian initiative to address indigenous homelessness. Evaluation and Program Planning, 56, 69–75.

54.

Kulis

Ayers

S. L.

Baker

(2015). Parenting in 2 worlds: Pilot results from a culturally adapted parenting program for urban American Indians. The Journal of Primary Prevention, 36(1), 65–70.

55.

Kypri

McCambridge

Vater

Bowe

S. J.

Saunders

J. B.

Cunningham

J. A.

Horton

N. J.

(2013). Web-based alcohol intervention for Māori university students: Double-blind, multi-site randomized controlled trial. Addiction, 108(2), 331–338.

56.

LaFrance

Nichols

(2008). Reframing evaluation: Defining an Indigenous evaluation framework. The Canadian Journal of Program Evaluation, 23(2), 13.

57.

Larzelere-Hinton

Goklish

Ingalls

Craft

Sprengeler

McGuire

(2016). Entrepreneurship education: A strength-based approach to substance use and suicide prevention for American Indian adolescents. American Indian and Alaska Native Mental Health Research(Online), 23(3), 248.

58.

Lowell

Kildea

Liddle

Cox

Paterson

(2015). Supporting Aboriginal knowledge and practice in health care: Lessons from a qualitative evaluation of the strong women, strong babies, strong culture program. BMC Pregnancy and Childbirth, 15(1), 1.

59.

Macaulay

A. C.

Paradis

Potvin

Cross

E. J.

Saad-Haddad

McComber

Desrosiers

Kirby

Montour

L. T.

Lamping

D. L.

(1997). The Kahnawake schools diabetes prevention project: Intervention, evaluation, and baseline results of a diabetes primary prevention program with a native community in Canada. Preventive Medicine, 26(6), 779–790.

60.

Maksimovic

Shen

Bandick

Ettridge

Eckert

(2015). Evaluation of the pilot phase of the “Give up smokes for good” social marketing campaign. Health Promotion Journal of Australia, 26(1), 16–23.

61.

Marley

J. V.

Atkinson

Kitaura

Nelson

Gray

Metcalf

Maguire

G. P.

(2014). The Be Our Ally Beat Smoking (BOABS) study, a randomised controlled trial of an intensive smoking cessation intervention in a remote Aboriginal Australian health care setting. BMC Public health, 14(1), 1.

62.

Mashford-Pringle

Stewart

S. L.

(2019). Akiikaa (it is the land): Exploring land-based experiences with university students in Ontario. Global Health Promotion, 26(3_suppl), 64–72.

63.

McCalman

Searles

Bainbridge

Ham

Mein

Neville

Campbell

Tsey

(2015). Empowering families by engaging and relating Murri way: A grounded theory study of the implementation of the Cape York Baby Basket program. BMC Pregnancy & Childbirth, 15(1), 1.

64.

McDermott

R. A.

Schmidt

Preece

Owens

Taylor

Esterman

(2015). Community health workers improve diabetes care in remote Australian Indigenous communities: Results of a pragmatic cluster randomized controlled trial. BMC Health Services Research, 15(1), 1.

65.

McElfish

P. A.

Bridges

M. D.

Hudson

J. S.

Purvis

R. S.

Bursac

Kohler

P. O.

Goulden

P. A.

(2015). Family model of diabetes education with a Pacific Islander community. The Diabetes Educator, 41(6), 706–715.

66.

McGrail

M. R.

Rickard

C. M.

Jones

(2006). Publish or perish: A systematic review of interventions to increase academic publication rates. Higher Education Research & Development, 25(1), 19–35.

67.

McKenzie

(1997). Developing First Nations child welfare standards: Using evaluation research within a participatory framework. The Canadian Journal of Program Evaluation/La Revue canadienne d’evaluation de programme, 12(1), 133–148.

68.

McShane

K. E.

Smylie

J. K.

Hastings

P. D.

Prince

Siedule

(2013). Evaluation of the acceptability of a CD-Rom as a health promotion tool for Inuit in Ottawa. International Journal of Circumpolar Health, 72, 20573–20573. http://ovidsp.ovid.com/ovidweb.cgi?T=JS&CSC=Y&NEWS=N&PAGE=fulltext&D=medl&AN=23717816

69.

Mercer

Riini

Hamerton

Morrison

McPherson

(2013). Evaluating a healthy eating, healthy action program in small Maori communities in Aotearoa, New Zealand. Australian journal of primary health, 19(1), 74–80.

70.

Mihrshahi

Vaughan

Fa’avale

Weliange

S. D. S.

Manu-Sione

Schubert

(2017). Evaluation of the good start program: A healthy eating and physical activity intervention for Maori and Pacific Islander children living in Queensland, Australia. BMC Public Health, 17(1), 77.

71.

Mohatt

G. V.

Fok

C. C. T.

Henry

Allen

Team

P. A.

(2014). Feasibility of a community intervention for the prevention of suicide and alcohol abuse with Yup’ik Alaska native youth: The Elluam Tungiinun and Yupiucimta Asvairtuumallerkaa Studies. American Journal of Community Psychology, 54(1–2), 153–169.

72.

Moher

Liberati

Tetzlaff

Altman

D. G.

, & The PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine, 6(7), e1000097. https://doi.org/10.1371/journal.pmed.1000097

73.

Moore

Jiang

Manson

S. M.

Beals

Henderson

Pratte

Acton

K. J.

Roubideaux

(2014). Case management to reduce cardiovascular disease risk in American Indians and Alaska Natives with diabetes: Results from the Special Diabetes Program for Indians Healthy Heart Demonstration Project. American Journal of Public Health, 104(11), e158–e164.

74.

Moran

J. R.

(1999). Preventing alcohol use among urban American Indian youth: The seventh generation program. Journal of Human Behavior in the Social Environment, 2(1-2), 51–67. http://ovidsp.ovid.com/ovidweb.cgi?T=JS&CSC=Y&NEWS=N&PAGE=fulltext&D=psyc3&AN=2000-02470-004

75.

Morton Ninomiya

M. E.

Atkinson

Brascoupé

Firestone

Robinson

Reading

Ziegler

C. P.

Maddox

Smylie

J. K.

(2017). Effective knowledge translation approaches and practices in Indigenous health research: A systematic review protocol. Systematic Reviews, 6(1), 34. https://doi.org/10.1186/s13643-017-0430-x

76.

National Aboriginal Health Strategy Working Party. (1989). National aboriginal health strategy. Aboriginal and Islander Health Worker Journal, 13(4), 7.

77.

National Health and Medical Research Council. (2003). Values and ethics: Guidelines for ethical conduct in Aboriginal and Torres Strait Islander health research. https://www.nhmrc.gov.au/_files_nhmrc/publications/attachments/e52.pdf

78.

Okamoto

S. K.

Kulis

Helm

Lauricella

Valdez

J. K.

(2016). An evaluation of the Ho’ouna Pono curriculum: A pilot study of culturally grounded substance abuse prevention for Rural Hawaiian youth. Journal of Health Care for the Poor and Underserved, 27(2), 815–833.

79.

Oser

Gohdes

Fogle

Tadios

Doore

Bell

Helgerson

(2013). Cooperative strategies to develop effective stroke and heart attack awareness messages in rural American Indian communities, 2009-2010. Preventing Chronic Disease, 10, E80.

80.

Pakseresht

Kolahdooz

Gittelsohn

Roache

Corriveau

Sharma

(2014). Improving vitamin A and D intake among Inuit and Inuvialuit in Arctic Canada: Evidence from the healthy foods north study. Journal of Epidemiology and Community Health, 69, 453.

81.

Palmater

P. D.

(2011). Stretched beyond human limits: Death by poverty in First Nations. Canadian Review of Social Policy, (65/66), 112–127.

82.

Patton

M. Q.

(2014). Qualitative research & evaluation methods: Integrating theory and practice. Sage publications.

83.

Pett

R. G.

Nye

(2015). Evaluation of a pharmacist-managed asthma clinic in an Indian Health Service clinic. Journal of the American Pharmacists Association: JAPhA, 56(3), 237–241.

84.

Potvin

Desrosiers

Leduc

Rivard

Trifonopoulos

(1995). The Kahnawake Schools Diabetes Prevention Project: Evaluation. Diabetes and Indigenous Peoples, 1, 80–83.

85.

Redvers

Bjerregaard

Eriksen

Fanian

Healey

Hiratsuka

Jong

Larsen

C. V. L.

Linton

Pollock

Silviken

Stoor

Chatwood

(2015). A scoping review of Indigenous suicide prevention in circumpolar regions. International Journal of Circumpolar Health, 74, 27509.

86.

Redvers

J. M.

(2016). Land-based practice for indigenous health and wellness in Yukon, Nunavut, and the Northwest Territories Environmental Design. Master's thesis, Environmental Design.

87.

Redwood

Provost

Lopez

E. D.

Skewes

Johnson

Christensen

Sacco

Haverkamp

(2016). A process evaluation of the Alaska native colorectal cancer family outreach program. Health Education & Behavior, 43(1), 35–42.

88.

Reeve

Banfield

Thomas

Reeve

Davis

(2016). Community outreach midwifery-led model improves antenatal access in a disadvantaged population. Australian Journal of Rural Health, 24(3), 200–206.

89.

Richmond

L. S.

Peterson

D. J.

Betts

S. C.

(2008). The evolution of an evaluation: A case study using the tribal participatory research model. Health Promotion Practice, 9(4), 368–377. http://ovidsp.ovid.com/ovidweb.cgi?T=JS&CSC=Y&NEWS=N&PAGE=fulltext&D=med5&AN=16803933

90.

Sahota

P. C.

Kastelic

(2012). Culturally appropriate evaluation of tribally based suicide prevention programs: a review of current approaches. Wicazo Sa Review, 27(2), 99–127.

91.

Schnarch

(2004). Ownership, control, access, and possession (OCAP) or self-determination applied to research. Journal of Aboriginal Health, 1(1), 80–95.

92.

Schoen

Balchin

Thompson

(2010). Health promotion resources for Aboriginal people: Lessons learned from consultation and evaluation of diabetes foot care resources. Health Promotion Journal of Australia, 21(1), 64–69. http://ovidsp.ovid.com/ovidweb.cgi?T=JS&CSC=Y&NEWS=N&PAGE=fulltext&D=medl&AN=20406155

93.

Scott

(2008). Drawing on indigenous ways of knowing: Reflections from a community evaluator. Canadian Journal of Program Evaluation, 23(2), 73–92.

94.

Senese

(2005). The PENAL project: Program evaluation and Native American liability (Chapter 8). In Hood

Frierson

Hopson

(Eds.), The role of culture and cultural context: A mandate for inclusion, the discovery of truth, and understanding in evaluative theory and practice (pp. 129–147). Information Age Publishing.

95.

SenGupta

Hopson

Thompson-Robinson

(2004). Cultural competence in evaluation: An overview. New Directions for Evaluation, 2004(102), 5–19.

96.

Shah

V. O.

Carroll

Mals

Ghahate

Bobelu

Sandy

Colleran

Schrader

Faber

Burge

M. R.

(2015). A home-based educational intervention improves patient activation measures and diabetes health indicators among Zuni Indians. PLoS ONE, 10(5), e0125820.

97.

Smylie

(2011). Knowledge translation and Indigenous communities: A decolonizing perspective. In Banister

E. M.

Leadbeater

B. J.

Marshall

(Eds.), Knowledge translation in context: Indigenous, policy, and community settings (pp. 181–200). University of Toronto Press.

98.

Smylie

Kirst

McShane

Firestone

Wolfe

O’Campo

(2015). Understanding the role of Indigenous community participation in Indigenous prenatal and infant-toddler health promotion programs in Canada: A realist review. Social Science & Medicine, 150. https://doi.org/10.1016/j.socscimed.2015.12.019

99.

Snijder

Shakeshaft

Wagemakers

Stephens

Calabria

(2015). A systematic review of studies evaluating Australian indigenous community development projects: The extent of community participation, their methodological quality and their outcomes. BMC Public Health, 15(1), 1.

100.

Stacy

Wisener

Liman

Beznosova

Lauscher

H. N.

Jarvis-Selinger

(2014). Community-based research in action: Tales from the Ktunaxa community learning centres project. Progress in Community Health Partnerships: Research, Education, and Action, 8(3), 281–290.

101.

Steglich

Sinclair

Holliday

Moore

(2012). Actor-based analysis of peer influence in a Stop Smoking In Schools Trial (ASSIST). Social Networks, 34(3), 359–369. https://doi.org/10.1016/j.socnet.2010.07.001

102.

Greaney

Nigg

Hirose-Wong

(2015). Developing a measure to evaluate a positive youth development program for native Hawaiians: The Hui Mālama o ke Kai Rubrics of Hawaiian Values. Asia-Pacific Journal of Public Health, 27(2), NP1517–NP1528.

103.

Thomas

S. L.

Williams

Ritchie

Zwi

(2015). Improving paediatric outreach services for urban Aboriginal children through partnerships: Views of community-based service providers. Child: Care, Health and Development, 41(6), 836–842.

104.

Thomas

V. G.

Parsons

B. A.

(2017). Culturally responsive evaluation meets systems-oriented evaluation. American Journal of Evaluation, 38(1), 7–28

105.

Tingey

Mullany

Chambers

Hastings

Lee

Parker

Barlow

Rompalo

(2015). Respecting the circle of life: One year outcomes from a randomized controlled comparison of an HIV risk reduction intervention for American Indian adolescents. AIDS Care, 27(9), 1087–1097.

106.

Tipene-Leach

D. C.

Coppell

K. J.

Abel

Pāhau

H. L.

Ehau

Mann

J. I.

(2013). Ngāti and healthy: Translating diabetes prevention evidence into community action. Ethnicity & Health, 18(4), 402–414.

107.

Towns

Cooke

Rysdale

Wilk

(2014). Healthy weights interventions in Aboriginal children and youth: A review of the literature. Canadian Journal of Dietetic Practice and Research, 75(3), 125–131.

108.

Townsend

C. K.

Dillard

Hosoda

K. K.

Maskarinec

G. G.

Maunakea

A. K.

Yoshimura

S. R.

Hughes

Palakiko

D.-M.

Kehauoha

B. P.

Kaholokula

J. K. a.

(2015). Community-based participatory research integrates behavioral and biological research to achieve health equity for Native Hawaiians. International Journal of Environmental Research and Public Health, 13(1), 4.

109.

Truth and Reconciliation Commission of Canada. (2015). Honouring the truth, reconciling for the future: Summary of the final report of the Truth and Reconciliation Commission of Canada. http://www.trc.ca/websites/trcinstitution/index.php?p=890

110.

United Nations. General Assembly. (2007). United Nations Declaration on the Rights of Indigenous Peoples.

111.

Voyle

Simmons

(1999). Community development through partnership: Promoting health in an urban indigenous community in New Zealand. Social Science & Medicine, 49(8), 1035–1050.

112.

Walker

Johnston

Glover

Bullen

Trenholme

Chang

Morris

Segan

Brown

Fenton

(2015). Effect of a family-centered, secondhand smoke intervention to reduce respiratory illness in Indigenous infants in Australia and New Zealand: A randomized controlled trial. Nicotine & Tobacco Research, 17(1), 48–57.

113.

Ward

A. F.

Smylie

Firestone

(2017). Evidence brief: Wise practices for Indigenous-specific cultural safety training. Indigenous Primary Heath Care Council.

114.

Weaver

Cousins

J. B.

(2007). Unpacking the participatory process. Journal of Multidisciplinary Evaluation, 1(1), 19–40.

115.

Well Living House. (2017). Emergent principles and protocols for Indigenous Health Service evaluation: Summary report of a provincial “three ribbon” expert consensus panel. Well Living House, Centre for Urban Health Solutions, St. Michael’s Hospital.

116.

West

Gamble

Kelly

Milne

Duffy

Sidebotham

(2016). Culturally capable and culturally safe: Caseload care for Indigenous women by Indigenous midwifery students. Women and Birth, 29(6), 524–530.

117.

Williams

Labonte

O’Brien

(2003). Empowering social action through narratives of identity and culture. Health Promotion International, 18(1), 33–40.

118.

Young

Gunasekera

Kong

Purcell

Muthayya

Vincent

Wright

Gordon

Bell

Gillor

(2016). A case study of enhanced clinical care enabled by Aboriginal health research: The Hearing, EAr Health and Language Services (HEALS) project. Australian and New Zealand Journal of Public Health, 40(6), 523–528.

119.

Yuskevich

(2010). Telling a story about Indigenous evaluation: Insights of practitioners from Australia, Canada, NZ and United States. University of Hawaii at Manoa.

120.

Ziabakhsh

Pederson

Prodan-Bhalla

Middagh

Jinkerson-Brass

(2016). Women-centered and culturally responsive heart health promotion among Indigenous women in Canada. Health Promotion Practice, 17, 814–826.