Sovereignty,Reciprocity,and Evidence: Practical Recommendations From a State-Funded Tribal Evaluation

Culturally relevant program evaluation with Indigenous communities raises important challenges for mainstream program evaluation. The unique legal relationship between Tribal governments and granting agencies establishes requirements about data ownership, research practice, and the goals of research on Tribal lands that go beyond standard evaluation practice (Bowman et al., 2015). Furthermore, the fraught historical relationship between Tribal communities and non-Tribal agencies demands ethical scrutiny that goes beyond usual practice. This includes the State of California, which committed genocidal violence and forced displacement against California Native communities in the mid-19th century (Madley, 2017). Recent efforts toward apology, land conservation, and collaboration with Tribal governments (Cowan, 2019; State of California, Office of the Governor, 2022) underscore the need for research designs that respect and incorporate Indigenous ways of knowing.

As an applied research approach, program evaluation has the potential to bridge Indigenous and mainstream scientific approaches by creating tailored, local, and actionable knowledge (Tuhiwai Smith, 2023). Yet evaluating programs using Indigenous approaches challenges mainstream methodological assumptions. In Kaupapa Māori evaluation, for example, whakawhanaungatanga centers relationship-building between evaluators and communities as a core methodological practice, shaping a ground-up, community-driven approach that does not necessarily conform to dominant paradigms of evaluation science (Cram et al., 2018). Indigenous evaluation practices also rely on ancestral values, core teachings, and intergenerational mentorship, while mainstream approaches often emphasize experimental design, quantitative data collection, and statistical analysis (Patton, 2014). From an Indigenous perspective, evaluation should contribute to survival, capacity building, and reciprocity with the community (Anderson et al., 2012; Brayboy, 2005; LaFrance & Nichols, 2010). These differing assumptions about what counts as evidence, whose knowledge is valued, and how results are used and disseminated have historically produced tensions between Indigenous and mainstream science.

We confronted these tensions often as the California American Indian Tobacco Initiative Evaluation (CAITIE) research team tasked with assessing the impact of the first initiative by the California Department of Public Health's California Tobacco Prevention Program (CDPH/CTPP) to directly fund Tribal community projects for commercial tobacco prevention. We had to make practical decisions that honored both our funder's expectations and our commitments to ethical and culturally relevant evaluation with Tribal partners. This article draws lessons from four challenges posed to CAITIE in the 2019 to 2024 grant cycle: (1) the lack of American Indian and Alaska Native (AI/AN) staff on the initial evaluation team, (2) negotiating the line between outreach and sensitivity, (3) the decision to aggregate data in reporting, and (4) disseminating the results to ensure cultural reciprocity. These tensions reflect core issues in Indigenous evaluation science related to Tribal sovereignty and data ownership. We describe our decisions, strategy, and process for addressing these tensions in this article.

We grounded our approach to writing this article in these lessons learned. This article draws on meeting notes, project documents, and interviews with team members whose storytelling provided important detail and mirrored the collaborative and relationship-centered approach used throughout CAITIE. We offer these reflections as a resource for others navigating similar tensions and as a contribution to the growing body of scholarship bridging Indigenous ways of knowing with mainstream evaluation practice.

Mainstream and Indigenous Ways of Knowing

Historically, mainstream scientific institutions have treated Indigenous methodologies as inferior (Williams & Shipley, 2023). Institutions that shape knowledge production, including universities, publishers, and funding agencies, systemically reinforce colonialism and racism, which perpetuate the marginalization of Indigenous methods (Bowman-Farrell, 2021; Williams & Shipley, 2023). Although this landscape is changing, major gaps remain between mainstream standards and Indigenous approaches to evaluation.

Organizations such as the National Science Foundation (NSF), the National Institutes of Health (NIH), and the Institute of Education Sciences (IES) provide widely used guidelines and frameworks that shape dominant approaches to evaluation and research. These standards for study designs, such as those created by the IES’ What Works Clearinghouse (WWC), assume that evaluators exercise control over participant assignments and can establish and maintain baseline equivalence between study conditions. These types of study designs have clearly defined outcome measures, avoid confounding factors, and manage changes in the composition of intervention and comparison groups through attrition or the late addition of participants. While standards vary by funding agencies, these mainstream expectations do not acknowledge Indigenous ways of knowing and can easily lead to tensions when the evaluation standards do not allow room for Indigenous perspectives.

Indigenous scholars and practitioners have recently codified Indigenous evaluation approaches within mainstream academia (Anderson et al., 2012, p. 567; see also Cajete, 2014). The scholarship of LaFrance and their colleagues (LaFrance, 2004; LaFrance et al., 2010; LaFrance et al., 2012), Brayboy (2005), Bowman et al. (2015), Waapalaneexkweew and Dodge-Francis (2018), Bowman-Farrell (2019, 2020), and Wehipeihana (2019), among others, has established a rigorous standard of work that ought to guide evaluation design, data collection, and analysis with Indigenous communities. While emphasizing different aspects of the work, the frameworks developed by these Indigenous scholars share certain fundamental tenets. These include a respect for Tribal sovereignty (Bowman et al., 2015; Brayboy, 2005), recognition of Tribal diversity (Caldwell et al., 2005; Chino & DeBruyn, 2006; LaFrance, 2004), an emphasis on participatory research practices (Bowman-Farrell, 2021; Clarke et al., 2022; Simonds & Christopher, 2013), a strengths-based approach (LaFrance & Nichols, 2010; Martinez et al., 2018), centering Indigenous ways of knowing (Cajete, 2016; Christensen, 2003; Kimmerer, 2015; LaFrance et al., 2012), and advocating context-specific solutions on behalf of Tribal partners (Caldwell et al., 2005; Treiber, 2011).

Indigenous scholars have increasingly examined the relationship between Indigenous and mainstream evaluation paradigms alongside the growing incorporation of Indigenous evaluation methods into mainstream scholarship. The work by LaFrance (2004), for example, presents an approach to evaluation that incorporates Indigenous ways of knowing into mainstream scientific approaches. They call for “culturally competent evaluation” with a focus on research that supports the longevity of the local community, whose purpose and results are accessible, and that includes participatory research practices and capacity building (2004). Waapalaneexkweew (Bowman-Farrell) and coauthors call for a “culturally responsive Indigenous evaluation” that respects Tribal sovereignty by challenging our assumptions of what is considered “good” data or analysis and calls for evaluators to relinquish privilege and power in the partnership in favor of local self-determination (Bowman-Farrell, 2021; Simonds & Christopher, 2013). Going further, scholars like Wehipeihana (2019) challenge the field by arguing that local Indigenous people should conduct all research with Indigenous communities and do so for local Indigenous purposes.

CAITIE Project Background

In 2017, the CDPH/CTPP held consultation meetings with AI/AN Tribes from throughout California to identify appropriate approaches to reduce commercial tobacco use disparities. These discussions included policy topics, cessation programs, Tribal-specific media campaigns, capacity building, respect for Tribal sovereignty and culture, and potential funding opportunities. These consultations, along with a follow-up online survey, formed the basis of the California Tribal Grants to Reduce Tobacco-Related Disparities, also called the American Indian Initiative. In the first 5 years, the initiative provided three waves of funding directly to 13 Tribal Governments and four American Indian Serving Community Organizations (AISCOs). Through collaboration and support from the CDPH/CTPP, the Tribal Community Coordinating Center (a project resource also funded by the initiative), and other local Tribal organizations, the initiative provided resources for projects to create and implement culturally appropriate commercial tobacco prevention interventions for California Tribal communities to work towards eliminating commercial tobacco-related disparities.

The purpose of CAITIE was to assess the impact and outcomes of this American Indian Initiative. The CDPH/CTPP tasked the evaluation team with assessing this impact through interviews, talking circles, and surveys with the staff of funded projects and community members affiliated with the Tribal communities that they serve. The evaluation results led to a final report outlining a series of recommendations regarding the success of the programs, the receptivity of the communities they serve, and the barriers to success that the CDPH/CTPP may need to address in future grant structures. The CDPH/CTPP would then use the results to guide future recommendations to support commercial tobacco prevention initiatives among AI/AN populations in California. Finally, the evaluation team would share the results with communities through educational webinars, case study reports, and peer-reviewed publications.

The research conducted by the evaluation team was bound by Data Use and Sharing Agreements (DUA) signed by the CDPH/CTPP, The Regents of the University of California on behalf of UCSD, and by the participating Tribes and AISCOs. These agreements established the responsibilities and protocols of the signees regarding data collection, analysis, and reporting for the evaluation research. These included that the Tribe or AISCO would be responsible for nominating and recruiting research participants, that neither the CDPH/CTPP nor UCSD would publicly identify individuals or Tribes in the dataset without written approval, that the results would be presented and reported in aggregate form, and that the CDPH/CTPP would share de-identified data with the Tribe. Negotiating these agreements respects Tribal sovereignty by explicitly stating the ways the signatories will use and store the data, and for what purposes. Evaluators should therefore carefully consider the consequences of the language in these agreements, as they affect the extent to which their research design and approach can incorporate best practices in Indigenous evaluation.

Indigenous Evaluation by Indigenous Evaluators

The development of the CAITIE team was a stepwise process, shaped by the expertise of the external advisory committee recruited for the project. The initial meeting with the advisory committee, however, was fraught with tension. The committee included five expert evaluators familiar with Indigenous tobacco research, three of whom were AI/AN. The AI/AN advisors posed a series of pointed questions that challenged the team's structure and goals. They included why non-AI/AN evaluators were conducting this work, why the CDPH/CTPP only provided 13 Tribes funding, and why the CDPH/CTPP did not consult with community leaders in the funding decisions. In short, the advisors saw the project as a State-led initiative that ignored Indigenous ways of knowing and reproduced existing inequalities and settler-colonial tendencies in research. At the core of this discussion was a lack of trust in the research team hired by CDPH/CTPP.

The advisors’ candidness in raising these issues led to a redesign of the team. One key concern of the advisory board was the absence of AI/AN evaluators conducting this work, which perpetuates the lack of trained AI/AN evaluators. The CAITIE team sought to address this by hiring AI/AN staff. Funding was allocated to hire a part-time AI/AN intern but, due to the lack of built institutional capacity for AI/AN evaluation, identifying suitable and interested candidates took a significant amount of time and effort. The evaluation team hired the first intern after 1 year, and their contributions proved vital, leading to a restructuring of the evaluation that ensured its cultural relevance. The evaluation team recruited three additional AI/AN interns throughout the 5-year project, and their contributions fundamentally shaped the work.

When the Administrative Director left their position, the CAITIE team sought to hire a candidate with significant experience working in and with AI/AN communities. The selected candidate was an AI/AN community leader, activist, and artist from Northern California. His contributions transformed the project as he became the ambassador of the CAITIE evaluation team in the partnered Tribal communities. His cultural knowledge and his commitment to advocacy through evaluation allowed for the project to grow in nuance and impact. He also served as a co-mentor with the PI for the interns who came through the program. The role that the new Administrative Director played in CAITIE emphasizes the vital importance for the field to continue to train and support AI/AN evaluation professionals.

The CAITIE team's diverse staff, including both AI/AN and non-AI/AN members, fostered an environment that valued and incorporated multiple perspectives. The weekly staff meetings served the procedural requirements of a complex project but were also key moments for mutual education and capacity building across different backgrounds and expertise. In these meetings and throughout the project, the PI encouraged AI/AN staff members to freely and openly share their insights and knowledge, providing a grounded understanding of the implications of the research decisions and assumptions. Furthermore, the weekly meetings often began and ended with photographs, news stories, or personal stories about events and histories related to the topics of discussion. These grounded the discussions in personal experience and current events. This inclusive approach facilitated a deeper understanding of the initiative's implications and provided a nuanced evaluation that honored the unique context of each project.

Prioritizing Relationships Over Data

Two AI/AN members of the CAITIE team led outreach to American Indian Initiative projects and Tribal community members. One team member is descended from multiple Northern California Tribal cultures, while the other is from a Tribe in Southern California. Both team members had relationships with Tribal communities they could rely on to facilitate communication within Tribal families and governments. These relationships, whether directly personal or rooted in social or cultural ties, helped the CAITIE team conduct outreach in ways that respected shared histories and, where possible, respected the specific contexts of participating Tribes.

Several challenges affected outreach despite the AI/AN background of some CAITIE members. The foremost was the unprecedented social and governmental response to the COVID-19 pandemic, which severely impacted the team's ability to reach Tribal community members. Tribal communities shut down in distinct ways during the pandemic. For example, there were funerals held at several Tribal facilities during the pandemic. These communities would close the Tribal government for days, sometimes up to 2 weeks, to mourn. The close-knit nature and often small numbers of people within Tribal communities meant that COVID-19-related struggles often impacted much of the community. These conditions caused long delays in the project. Many AI/AN community members prioritized survival during this ordeal and could not set aside time to think about and discuss this project. The CAITIE outreach team proceeded with extreme compassion and patience when working with community members as they sought to complete research tasks, prioritizing respect and relationships over data collection. This often meant giving people more time to recruit participants or schedule interviews, sacrificing methodological rigor for respect and understanding.

The widespread and frequent project staffing issues and turnover that occurred over the course of the funding period also created critical barriers for outreach. Under the DUAs between the CDPH/CTPP, UCSD, and participating Tribes, project staff provided the names of key informants and talking circle participants rather than having the CAITIE team recruit participants directly. When project staff left their positions, the CAITIE team did not have a direct link to the community, and new project coordinators were often unaware of CAITIE and their obligation to participate in the evaluation. Internal politics, priorities, and decision-making processes added layers of complexity in connecting with key informants. In some of the participating Tribes, for example, Tribal Council members had tenures shorter than the study period. If a council member who was especially supportive of a project left, that could lead to changes in project staffing, resources, and project support. The CAITIE team was able to maintain contact with most communities despite these challenges primarily through the work and experience of AI/AN staff. Knowledge of these political dynamics, for example, as well as the personal relationships that the Native team members had with members of some of the participating Tribes, helped CAITIE stay connected or reconnect with project staff when projects experienced turnover.

Outreach to community members was further complicated by the absence of any obligation to participate and, in some cases, outright hostility to participation. This hesitancy to participate in research, especially research associated with the State of California, is not surprising given the historical trauma and exploitation that AI/AN communities have experienced at the hands of researchers and governments. Reminding participants that involvement in this evaluation of the State-funded project was a way for them to have their voice and concerns heard was one way to reduce participants’ hesitancy. But failure to ease concerns could mean having to put pressure on project staff or community members to participate. Striking the right balance required taking historical context, present experience, and future relationships into account, and at times required prioritizing participant sovereignty over research goals.

Navigating Aggregation and Local Relevance

The aim of the research was to evaluate the impact and outcomes of the American Indian Initiative. The CDPH/CTPP would use the results to guide future recommendations on commercial tobacco prevention initiatives and serve to identify effective methods and protocols that they could share with California Tribal governments and AISCOs. This meant that the work would have two audiences: the CDPH/CTPP and the Tribes. While there was a responsibility to frame and report findings in a way that was helpful to the CDPH/CTPP, there was also a need to make sure that the findings would be locally relevant and useful to Tribal community partners, in accordance with best practices in Indigenous evaluation methodology (Caldwell et al., 2005). These responsibilities sometimes came into conflict, particularly in how we would aggregate the data while maintaining local relevance.

In the DUA between UCSD, the CDPH/CTPP, and Tribal governments and AISCOs, the partners agreed to aggregate data rather than analyze participating communities individually. This would protect the Tribes from repercussions regarding their success with the program and allow participants to express their honest opinions about the initiative. The logic of this agreement clearly follows standard scientific methodology and ethical practices. The confidentiality of participants is a cornerstone of ethical research practices and one that ethics committees and institutional review boards closely scrutinize. The requirement for aggregation, however, made it difficult for the results to be relevant to community partners. Aggregation comes into tension with a core principle of Indigenous evaluation: Tribes are diverse in their culture, practices, and legal and political relationships to the United States, as well as in their existing public health programming and infrastructure. Recognizing this diversity is not only ethical but also methodologically necessary if we are to correctly understand the consequences of Indigenous programs (Caldwell et al., 2005; Chino & DeBruyn, 2006; LaFrance, 2004). Aggregating communities with divergent historical relationships with the granting agency runs the risk of obfuscating the nature and origin of that diversity.

The stated requirement for aggregation did, however, allow some flexibility for discussing overall patterns. For example, the different initial capacities of Tribes were a central theme in the final report presented to CDPH/CTPP. The final report argues that to develop and implement more effective public health initiatives in Tribal communities, CDPH/CTPP needs to account for these differences in capacity through different expectations and definitions of success. While one community may define success as passing a policy that limits the sale of commercial tobacco products on Tribal land, another may define success as building enough trust and support to staff the project and participate in Tribal events. The difference between Tribes that can accomplish one or the other is not one of ability, but of initial capacity. Tribes that receive the grant within an existing infrastructure of public health or education programming, and with strong and existing relationships with Tribal leadership, are far more likely to achieve the stated aims of commercial tobacco use reduction than Tribal communities without such existing infrastructure. Funding Tribes without that infrastructure, however, may ultimately be more important for reducing commercial tobacco usage than funding Tribes with existing structures and similar grants, as it would allow for the development of that institutional capacity. While the evaluation team could report these trends in a theoretical way, the local differences in capacity and their origins remained unspoken and unaddressed.

Cultural Reciprocity in Dissemination

Tribal ownership of evaluation data and findings is a best practice for respecting sovereignty and healing the historic injustices of colonization, particularly as they have manifested in research (Burhansstipanov et al., 2005; Caldwell et al., 2005; Wehipeihana, 2019). This approach allows the participating Tribes to do what they determine is best with the findings, and it moves the important work of evaluation away from the extractive practices that fomented historic resentment and distrust of outside researchers in Tribal communities. However, the DUA between UCSD, the CDPH/CTPP, and the Tribal governments and Tribal community organizations that received American Indian Initiative funding placed ownership of the data and findings with the CDPH/CTPP. Although the same agreement requires the CDPH/CTPP to disseminate evaluation findings to all grantees, CDPH/CTPP approval ultimately determines which findings the evaluation team shares, how it frames them, and when it disseminates them.

To disseminate the results of the research in a timely manner and to gather feedback on our initial findings and analysis, the project included a series of “member check” presentations during talking circles. Member checks are a common, if somewhat controversial, research tool meant to improve the validity of a study's results (Birt et al., 2016; Motulsky, 2021; Thomas, 2017). They involve the research team sharing their findings and analysis with participants to assess resonance, surface misinterpretations, and co-interpret findings (Guba & Lincoln, 1989). Researchers often use this method in psychology (Motulsky, 2021) and sociology (e.g., Duneier et al., 2000), particularly in participatory research. There are ongoing debates about the epistemic privilege relinquished in the interpretation of data through this practice (Motulsky, 2021) and its potential impact on an ongoing research project. However, as Indigenous scholars remind us (Bowman et al., 2015; Brayboy, 2005), the oppressive history between evaluators and Indigenous people requires researchers to relinquish their power and privilege, and to soften their goals of “impartial” analysis, in favor of interpretations that will be locally meaningful and impactful (Martinez et al., 2018).

These member checks thus served as an important moment for the CAITIE team to share the stories and knowledge collected as part of the project, to improve the analysis based on the circle's understanding of the analysis, so that the final report would be directly useful to Tribal partners (Caldwell et al., 2005). Furthermore, member checks allowed the CAITIE team to honor the principle of cultural reciprocity with participants in the talking circles; sharing preliminary findings and hearing feedback from the Tribal community is part of a researcher's duty as guests on Tribal Nation lands. While the CAITIE project was ultimately unable to resolve the tension stemming from data ownership by the CDPH/CTPP, member checks allowed for the incorporation of feedback from people who participated in the talking circles, ensuring that the results would be relevant for the participating communities. The evaluation team shared the final results of the research via a co-hosted webinar, public release of the final evaluation report, peer-reviewed publications, and presentations at workshops and conferences.

Discussion

The practical incorporation of Indigenous and mainstream evaluation practices underscores the complexities of navigating differing sets of values, methodologies, and objectives. A mixed AI/AN and non-AI/AN team was fundamental in this process, allowing for a more comprehensive and culturally sensitive approach. This blend of expertise enriched the evaluation process at all levels, including outreach, data collection, data analysis, and the dissemination of results. Reaching out to Tribal communities, scheduling interviews, and coordinating the many attendees for a talking circle are difficult tasks anywhere, but especially so when working with communities that are rightfully suspicious of government-affiliated researchers. By leading the outreach effort, AI/AN staff engaged more participants in the project and incorporated a broader range of perspectives into the evaluation.AI/AN staff also gave the evaluation credibility and demonstrated a commitment to respect and reciprocity. Advancing high-quality research and analysis with AI/AN communities therefore requires broader investment in the funding and training of AI/AN evaluation researchers.

Reconciling the ethical requirement of confidentiality with the cultural imperative of reciprocity was the evaluation's most persistent challenge. The DUA between the CDPH/CTPP and the participating Tribal governments and AISCOs placed data ownership and dissemination under CDPH/CTPP authority. While consistent with mainstream standards of rigor, those terms sit in tension with Indigenous evaluation, which centers sovereignty, local control of knowledge, and cultural reciprocity. The CAITIE team implemented member checks and community presentations to return provisional findings and refine the analysis. These practices advanced reciprocity but could not fully resolve a structural mismatch.

The impact of the DUA on the balance between Western scientific standards and the inclusion of Indigenous ways of knowing in the research emphasizes the importance of recognizing and explicitly addressing Western assumptions about data quality and ethics (e.g., anonymity) when negotiating similar evaluation agreements. The DUA established standards for data governance, interpretive authority, and accountability that the CDPH/CTPP, UCSD, Tribes, and AISCOs jointly developed and approved. Yet these standards limited the scientific rigor of the research by assigning participant recruitment to Tribes themselves, limited the ability of Tribes to use the data for their own needs because of requirements for aggregation and anonymity, and unfairly placed responsibility on the CDPH/CTPP for determining what data and analysis were considered ready for dissemination and publication. As a single evaluation, this work cannot resolve how evaluators should integrate Indigenous evaluation practices with mainstream approaches, but we hope it highlights areas for further thought and research.

The success of the CAITIE team in overcoming the many challenges and reaching the milestones and goals of the evaluation prompted CDPH/CTPP to request a renewal of the evaluation and make modifications based on the lessons learned from the first 5 years of the evaluation. The evaluation's long-term prospective design could have generated additional important insights into navigating complex Indigenous research and evaluation challenges, but CDPH/CTPP terminated funding for the renewed 5-year evaluation after 1 year because state budget cuts prioritized other funded projects over the evaluation.

Conclusion

Throughout the evaluation, the CAITIE team confronted complex challenges, made consequential methodological and ethical decisions, and worked to center core principles of Indigenous evaluation. The team approached this work with a clear sense of responsibility to contribute to the well-being of Tribal communities while holding the CDPH/CTPP's health initiative accountable. The experience underscores the importance of culturally responsive evaluation that honors the diversity of Tribal partners. We hope this manuscript, based on the lived experience of the CAITIE team, supports other researchers seeking to partner with Native Nations and Tribal communities and advances the broader conversation on integrating Indigenous ways of knowing into evaluation practice.