Appendix D – Indigenous Approaches to Evaluation by JoLee Sasakamoose

Introduction

The purpose of this paper is to identify key considerations in the design and implementation of Indigenous evaluation methods for victim services programs.

Terminology

The word “Indigenous” is used throughout this paper as an all-encompassing term to identify First peoples and practices that are separate from the mainstream Western colonial worldviews. Specifically, in Canada there are three Indigenous groups: First Nations, Métis, and Inuit, and within these groups is a multitude of distinct and diverse nations and communities. By utilizing the term Indigenous, this paper is laying out guidelines that can be adapted to working with Indigenous communities around the world; however, it is imperative that there is local cultural awareness to avoid assuming all Indigenous nations are the same. Additionally, we will use First Nations, Inuit, Métis and Aboriginal throughout the manuscript as we have adopted the terms used by the authors of literature to avoid masking any differences that may exist between references.

Setting the Context

There has been a call for culturally responsive interventions that take a target group’s values, norms, beliefs, and practices into account in the design, delivery, and evaluation of programs (Resnicow, Soler, Braithwaite, Ahluwalia, & Butler, 2000). Strategies in the design, delivery and evaluation of programs that support Indigenous victims’ services clients must also work to ameliorate the severe impact colonization has had on families and communities (Mussell, Cardiff, & White, 2004). By placing the high rates of violence, substance abuse, and poverty experienced by Indigenous families into the appropriate context of colonization and assimilation policies, evaluators can focus on the resiliencies these people have demonstrated. With this background, it can be seen that the deliberate suppression and elimination of culture and tradition has led to intergenerational trauma, the residues of which are visible today in the increased levels of social and mental health problems observed in many Indigenous communities (Elias, B., Mignone, Hall, Hong, Hart, & Sareen, 2012; Esquimaux-Wesley & Smolewski, 2004; Kirmayer, Simpson, & Cargo, 2003). This paper will outline the historic and current context for Indigenous evaluation research, provide a rationale for the use of Indigenous methods and ethics, identify a decolonized framework for research and evaluation, and provide actionable decolonizing options for evaluation development and implementation.

Indigenous peoples and scholars have been working diligently over the last several years to develop decolonizing theoretical models and frameworks that prioritize Indigenous methodologies and ways of knowing alongside evidence-based Western practices (Snowshoe & Starblanket, 2016; Kovach, Carriere, Montgomery, Barrett, Gilles, 2015; Sasakamoose & Pete, 2015; Wilson, 2009; Waziyatawin & Yellowbird, 2005; Kirmayer, Simpson & Cargo, 2003). Decolonization, once viewed as the formal process of handing over the instruments of government, is now recognized as a long-term process involving the bureaucratic, cultural, linguistic and psychological divesting of colonial power (Smith, 2012).

Decolonization involves dismantling structures that perpetuate the status quo, problematizing dominant discourses, and addressing unbalanced power dynamics. On the other hand, decolonization involves valuing and revitalizing Indigenous knowledge and approaches and weeding out settler biases or assumptions that have impacted Indigenous ways of being. Decolonization necessitates shifting our frames of reference with regard to the knowledge we hold; examining how we have arrived at such knowledge; and considering what we need to do to change misconceptions, prejudice, and assumptions about Indigenous Peoples (Antoine, Mason, Mason, Palahicky & Rodriguez de France, [nd], p. 4).

The Indigenous Cultural Responsiveness Theory (ICRT) (Sasakamoose, Bellegarde, Sutherland, Pete, & McKay-McNabb, 2017) is a decolonizing, locally adaptable framework utilized to support program development and its subsequent evaluation. Understanding the framework will allow the practitioner to engage in actionable steps to create an evaluative process that seeks to improve programming for participants based on their needs. The ICRT framework includes the important contextual factors (historical, social, cultural and environmental) for engaging Indigenous knowledge, methodologies and participatory frameworks. This framework was developed as there is a recognized need for an improved evidence base as a foundation for measuring systemic change. Culturally responsive changes are needed in the systems that affect social determinants of wellbeing (for example, income, education, health, research, governance, justice) and disparities will not be addressed unless those who practise within these systems embrace an Indigenous cultural responsiveness paradigm (FSIN, 2013). Cultural responsiveness refers to services that are respectful of and relevant to the beliefs, practices, culture and linguistic needs of diverse client populations and communities (Rural and Regional Health and Aged Care Services, 2009). Cultural responsiveness includes the capacity to respond to the issues of diverse communities, it requires the knowledge and capacity at varying levels of intervention: systemic, organizational, professional and individual.

Cultural responsiveness recognizes that colonial practices and policies will continue to be a detriment unless evaluators bring to the forefront the land, language and cultural practices specific to the people for whom they are meant to work (Sasakamoose, Bellegarde, Sutherland, Pete, & McKay-McNabb, 2017; Whitbeck, 2006). Any attempts to improve Indigenous mental, emotional, spiritual or physical well-being must involve co-participation from community members for whom the programs or interventions are designed (Snowshoe & Starblanket, 2016; National Aboriginal Health Organization [NAHO], 2007). When the Duty to Consult with Indigenous peoples is respected and an Indigenous model of cultural responsiveness is implemented, evaluators may influence change in areas that affect Indigenous peoples (Snowshoe and Starblanket, 2016; FSIN, 2013; Reading & Wein, 2009). We acknowledge Indigenous nations have distinct sacred knowledge, beliefs and traditions and that each individual community/population will locally adapt this framework to their own ways of knowing. Benefits of culturally-responsive program development and evaluation research include access and equity for all; improved communication and understanding between client and providers which fosters better compliance with the recommended program, reduced errors, improved satisfaction, client safety and quality assurance, and best use of resources (Stewart, 2006).

Ethical Implications with Indigenous Research and Evaluation

There has been considerable work undertaken in the area of articulating ethical and respectful practices in engaging in research and evaluation with First Nations, Inuit, and Métis. From a First Nations, Métis and Inuit perspective, concerns about research include the reluctance of old order research to address issues of historical social systems and structures that contribute to current day marginalization of Indigenous peoples. When evaluators neglect any of these areas in the design of the evaluation process, it is more likely that bias against Indigenous knowledge will affect the final product (Ermine, Sinclair, Jeffrey, 2004). Without proper training, evaluators design from deficit-based perspectives that focus on problems with outcomes rather than the social or structural issues underpinning the conditions for Indigenous peoples (Smith, 1999). Western based perspectives on research are problem focused, while Indigenous perspectives are asset-based and solution-focused (Sasakamoose, et. al., 2017; Crooks, Snowshoe, Chiodo, & Brunette-Debassige, 2013).

To support an Indigenous worldview in an evaluation framework, the following should occur: community cultural protocols are understood and observed; the evaluation process is built upon developing a relationship of trust and respect; important issues are identified and redressed and the communities’ or populations’ political, social and cultural values are incorporated into the methodology (Kovach et al., 2015; Steinhauer, 2002; Smith, 1999). Community members’ perspectives must be considered and included in the preparation of documents and reports, such as records of historical events and cultural ceremonies (Swisher & Tippeconnic, 1999). Program evaluation should be designed around improved programming for participants based on their needs. The future of culturally-responsive program evaluation will include the generation of Indigenous knowledge, methodologies, and participatory frameworks. Further it is important that Indigenous people be involved in producing research rather than merely participating as subjects (Swisher & Tippeconnic, 1999).

Performance measurement and evaluation, often share the same methodologies as social science research, however, stakeholders usually have a more immediate interest in the findings of an evaluation. Evaluation is characterized as focusing on producing practical knowledge for immediate use by clients for a specific goal or decision; whereas research focuses more on long-term understanding (Barnett & Camfield, 2016). Both fall short and often overlook ethical and policy guidelines for conducting research involving Indigenous populations. Because of these ethical failings First Nations, Métis or Inuit communities are on the forefront of creating and implementing their own research standards and ask that evaluators also uphold the protocols identified within the research community as identified below. Research and evaluation conducted involving Indigenous peoples should be able to identify, understand, and implement the appropriate ethical considerations and protocols specific to the community or population being evaluated. Navigating tensions between ethical rigour in conducting program evaluation is a challenging, yet a crucial undertaking. Evaluators should not assume that approval of an evaluation project by formal authority structures or administration guarantees the movement of the project forward within First Nations, Métis or Inuit communities. In some First Nations, Métis or Inuit communities, the authority to permit and monitor research lies with the community members designated by traditional protocols and in some First Nations settings, a coalition council spanning several communities may be recognized as having official jurisdiction for research and evaluation initiatives involving its members (Crooks, Snowshoe, Chiodo, & Brunette-Debassige, 2013). The identification of culturally appropriate outcomes and measures are challenging conceptual issues since most research has ignored the tribal cultures and traditions that may protect First Nations, Métis or Inuit communities from adverse outcomes (Friesen, Cross, Jivanjee, Gowen, Bandurraga, Bastomski, Maher, 2011).

In addition to community expectations for researchers, many organizations and First Nations, Métis or Inuit communities have developed their own formalized code of research ethics and those ethics will supersede any others in relation to the specific First Nations, Métis or Inuit community (Assembly of First Nations, nd). The Assembly of First Nations is a national advocacy organization representing First Nation citizens in Canada, which includes more than 900,000 people living in 634 First Nation communities and in cities and towns across the country. In consultation they developed a guide for First Nations ethics on Research and Aboriginal Traditional Knowledge. The Royal Commission on Aboriginal Peoples (1996) included an appendix in their seminal report outlining ethical guidelines for research that specified that the purpose of the code was to “ensure that appropriate respect is given to the cultures, languages, knowledge and values of Aboriginal peoples and to the standards used by Aboriginal peoples to legitimate knowledge (p. 325).”

The United Nations Declaration on the Rights of Indigenous Peoples, was adopted in Canada in 2016 (UN General Assembly, 2007). Article 19 of this document outlines the Duty to Consult, which advises that governments must “consult and cooperate…with Indigenous Peoples concerned…before adopting and implementing…measures that may affect them (pg. 6).” Following the United Nations recommendations, this means that before any programming design or evaluation is conducted, there must be proper consultation with Indigenous representatives. When the duty to consult with First Nations communities is respected and an Indigenous model of cultural responsiveness is implemented in the program evaluation design, research findings may influence change in the social determinants of health that affect Indigenous Peoples (Snowshoe & Starblanket, 2016; FSIN, 2013; Reading & Wein, 2009).

The Canadian Institutes of Health Research (CIHR), the Natural Sciences and Engineering Research Council of Canada (NSERC) and the Social Sciences and the Humanities Research Council of Canada (SSHRC) (2010) have prepared guidelines for researchers working with Indigenous peoples as part of the Tri-Council Policy Statement (TCPS): Ethical Conduct for Research Involving Humans. They published a specific chapter outlining core principles when research involves First Nations, Métis, or Inuit participants. The purpose of these guidelines is to “ensure, to the extent possible, that research involving Aboriginal people is premised in respectful relationships. It also encourages collaboration between researchers and participants (CIHR, NSERC, & SSHRC, 2010, p. 105).” It is an imperative starting place to have program designers and evaluators trained and familiar with Tri- Council Policy Statement Chapter 9 before implementation or data collection.

Ownership Control Access Possession (OCAP®): The First Nations principles of OCAP® are a set of standards that establish how First Nations data should be collected, protected, used, or shared. They are the de facto standard for how to conduct research with First Nations. OCAP® is inextricably linked to Indigenous Peoples’ agenda of self-determination because it serves to guide the re-appropriation of research activities and their outcomes. The protection of the cultural and intellectual property of Indigenous peoples is fundamentally connected with the realization of their territorial rights and right to self-determination (Simpson & Jackson, 1998). OCAP® is serving to enhance capacity building in Indigenous research by bringing the concepts of ownership and control to the attention of communities (Johnson & Ruttan, 1992). At the same time, the trend towards OCAP® is hastening the development of community-based research guidelines and agreements, is influencing how research ethics boards are conducting ethical reviews of Indigenous related research, and is also influencing how community-based research information is accessed and how research is conducted. There are several guidelines for conducting research with Indigenous peoples which help to refine our understanding of what it means to conduct culturally responsive evaluations in First Nations, Métis or Inuit communities and it helps to map future research directions.

The Decolonizing Evaluation Protocols and Framework

A significant challenge faced by researchers and evaluators stems from the lack of trust that many Indigenous peoples have for the research and evaluation processes driven from a Western scientific perspective. Evaluators tend to enter communities and conduct projects without the respect and reciprocity needed to make the evaluation relevant and beneficial to communities or participants (Kovach, 2012; Wilson, 2009). It is imperative that future evaluations be culturally responsive and have a keen understanding regarding the colonial history in an effort to do no further harm to Indigenous peoples. Evaluation measurement and research are both relevant to the development and delivery of effective programs and services. The shift to new paradigms of research and evaluation is the result of a decolonizing agenda. Indigenous peoples, communities and scholars are leading the way by melding science with tradition and laying new ground for developing culturally-responsive frameworks (Sasakamoose, et. al 2017; Fiedeldey-Van Dijk et al., 2016). Decolonized approaches make it possible to develop evaluation research processes that best meet the needs of clients while getting the most value from available resources. Decolonizing practices include privileging and engaging in Indigenous philosophies, beliefs, practices, and values that counter colonialism and restore well-being (Sasakamoose & Pete, 2015; Sasakamoose, et. al., 2017). Decolonizing models validate and support Indigenous histories and inherent rights and will generate research to reframe, rename, reclaim, and restore Indigenous methodological approaches (Kovach, 2012). Kirkness and Barnhardt (1991) have identified the “4 Rs” for developing research procedures in an Indigenous context and Kovach (2010) identified a fifth.

  1. Respect, or valuing the diverse Indigenous individual, cultural and community knowledge. Respect includes understanding and practicing community protocols, being reflective and non-judgmental, being able to hear what is being said, and building on cultural, social and spiritual values that arise from within the community.
  2. Relevance to community and cultural needs and experiences. Participants should be part of designing the research and evaluation methods and interpretation of findings.
  3. Reciprocity, where both the community and researcher/evaluator benefit from a two-way process of learning and research. A question to ask is, “What is the benefit to the community?”
  4. Responsibility, where there is active empowerment for community members through full engagement and participation. This means the evaluator develops and maintains credibility with the community by considering all perspectives and working collaboratively and sharing findings.
  5. Reflexivity (Kovach, 2010), changing as a result of the co-constructed relationship and knowledge sharing.

Fletcher (2003) has suggested that when working with Indigenous communities, the evaluator should: acknowledge power imbalances between community and evaluators; focus on relevant topics; foster autonomy and develop capacity in the community; engage community members; consider evaluation as an opportunity to provide public education about research; and respect the ethical guidelines of the communities and organizations that represent the interests of Indigenous people. Evaluators should ensure research objectives are transparent, local politics and protocols are respected, community authorities are recognized, confidentiality is ensured, culturally responsive tools are used and that a comprehensive dissemination strategy is employed. Finally, Sasakamoose & Brace (2018) outline evaluation planning strategies for ensuring participation in Indigenous communities. These include: allotting time for relationship building, community engagement, hosting and attending ceremony, and community events; allotting budget items for relationship building, food, cultural protocol items, Knowledge Keeper gifts, and travel; ensuring evaluation transparency; ensuring space for community input; identifying community assets and needs; and finally developing an evaluation and data sharing agreement prior to the research process. Engaging in a participatory evaluation framework along with professional and cultural codes of conduct will produce an evaluation with the greatest utility and impact.

The ICRT models a reconciliatory approach to health and wellness that is grounded in four protective factors: community-specific, strengths-based, trauma-informed and spiritually grounded (Sasakamoose et al, 2017; Snowshoe & Starblanket 2016; FSIN 2013). Founded in ceremony, steeped in Indigenous ways of knowing, harmonized with evidence based Western practices, and locally adaptable, the Cultural Responsiveness Framework (CRF) approach is uniquely situated to 1) support the restoration of Indigenous community-based health and wellness systems, 2) establish a middle ground for engagement between Indigenous and Western systems to decolonize health research by upholding the commitment to reconciliation and 3) to guide research that improves the Indigenous wellbeing (FSIN, 2013; Sasakamoose, 2017; TRC, 2015).

The key to developing a much-needed culturally adaptable framework for First Nations peoples lies not only in decolonizing the approach, but also in utilizing culture as a tool to engage wellness. Snowshoe and Starblanket (2016) identify four healing protective factors that are effective when applied as decolonized approaches to evaluation: Spiritually grounded, Community specific (engaged), Trauma-informed, and Strengths-based. These principles are appropriate because they fit with an Indigenous worldview and are already a component of establishing a middle ground.

Spiritually Grounded

One of the most overlooked aspects of developing culturally appropriate programming and evaluation for Indigenous people is a connection to the spiritual (Snowshoe & Starblanket, 2016). This is a broad term and has various interpretations, but it is intrinsic to being human. For Indigenous people being spiritually grounded includes being connected to language and culture, Indigenous worldviews, a holistic view of wellness, connecting to the land, sustaining relationships with family and community, the use of ceremony, and integrating cultural traditions.

Community-Specific

Healing for the individual is directly connected to community and cultural healing for Indigenous people. Kirmayer & Valaskakis (2009) indicate that the health of the community is linked to the sense of local control and cultural continuity. Therefore, any program development and its subsequent evaluation should support initiatives that follow the vision of the unique community based on the project, community needs, capacity, interest and engagement.

Trauma-Informed Perspective

It is essential that researchers, program directors, and evaluators understand that “history has had complex effects on the structure of communities, individual and collective identity, and mental health [of Indigenous peoples] (Kirmayer & Valaskakis, 2009, p. 27).” The relationship between Indigenous people and the Government of Canada is filled with abuses, mistrust, broken promises, racism, and control. From colonization, confinement on reserves, unfulfilled Treaty promises, the Residential School system, and bureaucratic control, there are a multitude of “inequities have arisen within the context of an extensive history of aversive treatment of First Nations peoples borne of political policies specifically aimed at the destruction of First Nations cultures (Snowshoe & Starblanket, 2016, p. 67).” Evaluators must consider the intergenerational impact of colonization and its associated negative impacts on the lives of Indigenous people. To take a trauma-informed approach, program designers need to familiarize themselves with the causes and effects of this history and create programing that does not perpetuate systems that have alienated Indigenous people or re-traumatize them.

Strengths-Based Perspective

Strengths-based approaches have the advantage of building competencies that lead to increased wellbeing and adjustment, and also help protect against a range of negative outcomes. A strengths-based approach is especially important for Indigenous peoples, because it takes the Canadian historical context into account (Crooks, et al., 2013). A strengths-based perspective attempts to identify what resources an individual has to positively address problems. It is a model that focuses on developing assets (Smith, 2006). As people develop greater awareness of their own strengths, they will be able to take control of their lives and make appropriate decisions to empower themselves (Smith, 2006). This translates to program evaluation directly in terms of asking “what is worth measuring?” A deficit model focuses on negative determinants, while a strengths-based model looks for opportunities for growth. The evaluation criteria should target what is going well, how are participants improving, and should work to reframe negative behaviour as developed coping mechanisms (Snowshoe & Starblanket, 2016). This doesn’t imply that evaluators should simply ignore the realities and the negative consequences to focus on positives; rather, strengths-based health promotion facilitates wellness by building on the pathways to resilience among Indigenous peoples (Snowshoe, Crooks, & Tremblay, 2017; Snowshoe, Crooks, Tremblay, Craig, & Hinson, 2015).

Ideally whatever will be evaluated regarding the framework and related initiatives will be done in close collaboration with the Indigenous populations being served. In some instances, evaluation tools and models from other jurisdictions may have some applications for what is being done locally and sometimes they may not. It is important to consider whether those tools being utilized have had significant Indigenous community input and are not merely an adopted model from outside (FSIN, 2013). Furthermore, Elders and others want clarification and assurances when tools are locally adapted to develop an evaluation framework that does not ‘give away’ the culture or medicines, as they are all too familiar with examples where knowledge and medicines were shared and later plagiarized, stolen or used without permission.” (FSIN, 2013, pg. 16).

Moving into Action

In order to adopt an Indigenous approach, there must be the realization that one homogeneous evaluation will not be all encompassing for Indigenous evaluation due to the diversity and complexity of community and cultural context (Gray, D., Saggers, S., Drandich, M., Walam, D., & Plowright, P., 1995). However, guiding principles are present in all Indigenous evaluations. The ICRT model provides an overarching framework to plan, develop, execute, and evaluate programs to support Indigenous peoples. Adhering to the spirit of this framework will begin the process of decolonizing programs in order to better address the needs of Indigenous people. In order to assist moving from the theoretical to a practical application, many concrete items should be considered to help shape the evaluation approach.

Summary

Utilizing Indigenous ways of knowing in program design, implementation and evaluation provides an opportunity to demonstrate program success and identify areas for program improvement. Findings interpreted from evaluations should be evidence-informed and should include participants in the interpretation of results. Indigenous scholars have identified that evaluations need to be culturally responsive and consider the population’s needs in the design, delivery and evaluation of programs. The principles of Indigenous evaluation are very dependent on the Indigenous group (First Nations, Métis, Inuit) and partners, which can vary considerably based on the situation, the communities’ needs, as well as the community’s capacity and interest in engagement (Ongomiizwin, nd).

Often there is hesitancy among those interested in getting involved in Indigenous research or evaluation because there is no cookie-cutter approach. By employing the Cultural Responsiveness Framework, we offer clear strategies for ethical approaches to engagement for any type of research including evaluation with Indigenous peoples. It is in the strength of these approaches that the success of the program and its subsequent evaluation should be built. The most important recommendation is that there is an ability to establish genuine partnerships and relationships between the evaluators and the community members in a manner that makes sense for the setting. Further, it is essential that evaluation results be used to produce direct community benefits. We suggest that partners work collaboratively to create participatory approaches to the design of both the program and the evaluation, underscoring the need to ground within the cultural and historic context of the specific community or population. We suggest that the participants or community be involved in the data interpretation and that creative dissemination strategies (i.e. native radio stations, written reports, videos, or group presentations at the grass roots level) are utilized. Finally, we recommend that the evaluation teams consider participating in Reflexive Reflexivity (LaVallie, 2019, Lavallie & Sasakamoose, 2016) and employing cultural humility. Engaging with these two approaches is a lifelong process of self-reflection and self-critique whereby the individual not only learns about another’s culture, but starts with an examination of his or her own beliefs and cultural identity as a foundation for building honest and trustworthy relationships. It is a process that requires the practitioner to face the power imbalances that exist in the relationship dynamics with their clients and the community partners.

References