Appendix C – Reflecting on Indigenous Evaluation Frameworks by Gladys Rowe

Introduction

The development of the field of Indigenous evaluation is both an act of resistance and resurgence in response to inequitable and colonial relationships between Indigenous peoples and settlers in Canada. Simpson (2011) asserts that resurgence is a pathway towards reconciliation. “Anything less than space, recognition, and respect for the necessity of cultural resurgence is not reconciliation in its fullest sense,” (Rowe & Kirkpatrick, 2018, p. 2). Making space and pushing back has occurred through the leadership of Indigenous people in many fields and professions. Recently, supported by the 94 Calls to Action (2015), organizations have also begun to recognize gaps in representation and the necessity to incorporate Indigenous ways of knowing, being, feeling, and doing, including in the fields of research and evaluation.

This paper has been drafted in response to a request from the Research and Statistics Division (RSD) and the Policy Centre for Victim Issues (PCVI) at the Department of Justice Canada to explore and identify Indigenous approaches and methods for evaluating services and supports for Indigenous victims and survivors of crime. The intent of this paper is to share actions and processes that support the development of evaluation frameworks in partnership with Indigenous organizations and communities. It is clear there is great interest in how such a framework can be used as a resource to programs who work directly with Indigenous peoples.

In order to strengthen the field of Indigenous evaluation, space must be made, and it follows that evaluators must reflect deeply on what this looks like. TRC Call to Action #40 enlists all levels of government to work collaboratively with Indigenous peoples to ensure that in the development of adequately funded and accessible Indigenous-specific victim programs and services, appropriate evaluation mechanisms are also designed and employed. Therefore, funding bodies and evaluators are responsible for identifying what they can do to support the tenets and processes set forth in the following document. Meaningful community involvement in the design of Indigenous evaluation must occur at the beginning, must be fully supported with resources, and must be given the time and space to build capacity (Grover, 2010, Grover, Cram & Bowman, 2007; LaFrance, 2004).

In my own work, I can only speak from my experiences and location, as an urban, mixed ancestry, Swampy Cree woman, who has been trained in the field of evaluation and has been working with communities in Indigenous research that is grounded, community directed, and participatory for over ten years. I bring with me experiences from my education within post-secondary institutions, on the land, and in ceremony learning from Elders and traditional knowledge keepers. Coming from this location, I am able to share what I have learned with the hope that this provides something meaningful as the field of Indigenous evaluation continues to expand.

In this paper, I hope to provide perspectives in this pursuit. It will begin by outlining a background as to why Indigenous evaluation is a necessity, describing the learning that has taken place to date, and listing the considerations for evaluators and organizations who are seeking to use this framework. Guiding principles and values will be outlined, sharing experiences and identifying areas one must ensure are addressed. This paper will share a challenge for knowledge users to consider using their role and this framework in relation to the Calls to Action (2015) and reconciliation. This can lead to personal and professional acts contributing to decolonization overall.

Background

Indigenous evaluation is intimately linked to assertions of self-determination and self-governance (Smith, 1999). Who will set a knowledge-seeking agenda, whose voice will lead the process, whose knowledge will be sought and valued, what methods will be used to gather the knowledge, and what will be the ultimate use and distribution of the results of the knowledge-gathering are all important elements that have been raised by Indigenous researchers for decades. The answers to the questions above are fundamentally about power over knowledge production and representation. It is important to consider these factors in the roles and responsibilities evaluators must adopt for decolonizing and reconciling.

Hart & Rowe (2014), in their examination of the field of social work, have asserted the necessity of working from an anti-colonial and decolonized space. They have provided guidance (see the list below) on individual and organizational responsibilities for helping professions. These recommendations also have important implications for research and evaluation by, with, and for Indigenous peoples. This is a starting point in the education and ongoing professional training that must be completed by evaluators working with Indigenous peoples and communities. The responsibility is held with each of us to examine the colonial lens upon which not only social work, but also research and evaluation, have been founded.

  1. Educating self about oppression in general and colonial oppression specifically.
  2. Learning about the untaught First Nations history that up to now has been absent from typical curricula.
  3. Developing critical reflexive skills, as well as critical analysis skills.
  4. Honestly looking at one’s unconscious participation and erroneously informed participation in the oppression.
  5. Educating others on oppression through social action, informal dialogues, and sharing of information.
  6. Developing an understanding of First Nations Peoples, cultures, perspectives, and experiences.
  7. Creating space for First Nations contributions and developments, which requires encouragement, acceptance of differences, and concrete support.
  8. Challenging the profession in relation to its privilege, whether those privileges stem from the types of practices that are utilized, the theoretical perspectives that are taught and learned, or the values and belief system that is followed.
  9. Supporting the continuing development of Indigenous social work practice, perspectives, and theories.
  10. Making space for Indigenous participation in all segments of the profession (p.36).

These are issues and the necessary responses are not new. However, they have begun to receive more attention from various professions, including evaluation.

Similar to Indigenous peoples’ history with research – the evaluation of programs, policies, and organizations serving Indigenous peoples has been fraught with challenges. Fundamental differences between Western and Indigenous ways of understanding the world, and the privileging of Eurocentric values and knowledge development have meant that Indigenous voices in evaluation have been lacking (Smith, 1999). Design of programs and the corresponding evaluations must reflect the values and principles of Indigenous peoples. Research continues to show that in order to provide meaning and assist in success, social services and health programs must include opportunities for connecting with traditional knowledge and ways of doing.

There are several challenges that have been experienced by Indigenous evaluators, programs, and organizations who have worked to incorporate Indigenous beliefs, values, and methods into their evaluations. One challenge has been in the philosophical foundations of evaluation and the methodologies used to design them. While conversations about cross cultural awareness and evaluation practice are starting to take place (i.e., by Linda Tuhiwai Smith in 1999), many Western trained evaluators remain unaware of these conversations and the need to address issues being raised. These issues include a long and challenging history of Indigenous peoples being objects of research whereby information was extracted for benefit of others. These discussions have moved from awareness to a call for culturally relevant evaluations, and then culturally competent evaluators (Barrados, 1999).

Even with the push for culturally competent evaluation, which was accomplished by training non-Indigenous evaluators to work with communities, challenges remain. One of the most glaring challenges is that evaluation is still being done on Indigenous peoples by non-Indigenous peoples. Indigenous evaluators continue to confront the status quo in evaluation. In Australia, an Indigenous Community Capacity Building Roundtable (2000) has developed the following eight principles to guide evaluations with Indigenous families and communities:

By the early 2000s, Indigenous-led evaluation practices were being discussed. During the 2003 Australasian Evaluation Society International Conference in Auckland, Russell Taylor gave a keynote address, “An Indigenous Perspective on the Inter-Cultural Context.” This was the first time that an Indigenous person spoke from centre stage about Indigenous evaluation (Hurwoth & Harvey, 2012). During the same event, discussions also included the development of a Kaupapa Māori evaluation framework, seminal work upon which many Indigenous communities have fashioned their own examples for evaluation and research.

Even with recognition and more space for Indigenous evaluation being made, the lack of training and expertise to address these issues within institutions remains.

Indigenous, culturally based, non-profit programs devote their limited resources to the delivery of services to underserved, low-income and special needs populations. Theirs is a constant battle to continue operating while persistently seeking financial support through either grant proposals, contributions, income development or other means. An equally critical need of these programs is dedicated infrastructure for program development and evaluation, without which best practices of these innovative community programs remain obscure and unsubstantiated (Morelli & Mataira, 2010, p. 1).

In addition to lack of institutional level training, granting organizations often provide another barrier in funding guidelines which hold specific measures of success, reporting, and evaluation expectations that do not align with the models upon which the programs are developed. The ways of working that could provide insight into successful outcomes are being lost due to this incongruence (Morelli & Mataira, 2010). In addition, evaluations within the typical scope of grant requirements provide a narrow view of program dynamics – with little understanding of the relational process and culture-based practices. This misses holistic connections and values-based practices that can impact long term outcomes and influence community well-being (Morelli & Mataira, 2010). Recognition of this misalignment in the field of evaluation dates back to the 1970s (Hurworth & Harvey, 2012). Requirements typically do not take into account the need for Indigenous evaluation frameworks, methods, evaluators, and the increased time required to build these models (Grover, 2010). This leads to evaluations that do not meet the underlying needs of the program or the mechanisms through which change typically occurs based on these models. This can mean that evaluations fail to adequately describe an Indigenous program’s strengths, which can fall out of the scope of traditionally quantitatively focused results.

The necessity of evaluators to educate themselves and have a clear understanding of the impact of research and evaluation on Indigenous peoples has also been asserted by Indigenous scholars (Smith, 1999; Wilson, 2004; Kovach, 2010; Hart, 2010; Maitara, 2000). Community context, current and historical, are foundational elements for an evaluator to understand at the beginning of building a relationship with a program or organization. Bowman-Farrell (2018) cautions against entering a community as an evaluator and imposing a model, design, instruments, or tools. Similarly, an evaluator must not impose assumptions about the value of Western knowledge and ways of knowing over traditional knowledge shared through generations. This can mean the evaluator must broaden their understanding of subject matter experts, sources of meaningful knowledge, and mechanisms for dissemination of results of an evaluation. Inclusion of Indigenous traditions is a place to begin, but there is much more that must occur. Changing mindsets, belief systems, behaviours, and resources is required in order to create institutional and systemic change. This requires evaluators to engage in critical reflection about their practice:

As a profession, we need to critically question the structures and systems that perpetuate or legitimize implicit or explicit racism. Using a strengths-based approach, the evaluation field can start by including Indigenous SMEs (Subject Matter Experts) on key initiatives and make access and resources more available, so that truly collaborative studies with Indigenous scholars can contribute to the “evidence- based” policies, programming, committees, and practice. (Bowman-Farrell, 2018, p.6)

Taken as a whole, this produces culturally responsive evaluation that:

…is a welcoming space where evaluators and evaluations honor the strengths, respect the diversity, and authentically include, engage, and empower evaluators and the communities they are working with (not “on”) in the evaluative process, so they can be their own social justice and transformative leaders for creating and sustaining local change. (Bowman-Farrell, 2018, p.10)

In order to change we must first be aware that there is a problem. For non-Indigenous evaluators, education is key, but action is also required through continual and consistent engagement in knowledge and capacity building within the profession. There are two paths that can be taken. One includes the education of current non-Indigenous evaluators on the importance of working within this framework. Outreach and inclusion are actions required in the second path. This can mean creating requirements for inclusion of Indigenous academics and organizations in prominent positions and significant contracts for evaluation, policy or research studies, and related training and technical assistance contracts (Bowman-Farrell, 2018). The strategic inclusion of Indigenous evaluators and Indigenous theories and methods in evaluation will allow for a building and strengthening of this field.

LaFrance, Nichols & Kirkhart (2012) describe fundamental elements of Indigenous Evaluation Framework epistemology and methods and give several examples of these elements from evaluations in American Indian communities. The Indigenous Evaluation Framework was developed based on requests from Tribal colleges in the US to be able to use an evaluation model that was respectful of their context. The framework was developed in collaboration with expert advisors and pilot tested within Tribal colleges and with Indigenous primary and secondary school educators.

In research and evaluation, validity is a key term used by scholars and evaluators to assess the completeness of a study. LaFrance, Nichols & Kirkhart (2012) argue that context is essential to the measurement of validity – programs can only be wholly understood through their relationship to place, setting, and community. “Methodological justifications of validity such as those argued by Rog must be placed in cultural context, supported by justifications grounded in theory, life experience, interpersonal connections, and concern for social consequences,” (LaFrance, Nichols & Kirkhart, 2012, p. 62).

Given the need to contextualize, an Indigenous evaluation framework will be specific to each community situation. Indigenous ways of knowing are founded upon traditions of specific cultural groups and can include their creation stories, clan origins, and experiences of their ancestors passed down through stories. This also includes empirical knowledge, gained through observation and from other perspectives acquired through dreams, visions, and ceremony (LaFrance, Nichols & Kirkhart, 2012).

In addition to the contextualized methods based on each unique community, evaluators must consider culturally specific methods of reporting of evaluation and research results. This includes being responsible for the relationships addressed in reporting, for example the community, description or not of any ceremonies, and acknowledgment of Elders and the knowledge shared (LaFrance, Nichols & Kirkhart, 2012).

…the framework suggests that those who want to apply an indigenous approach to research or evaluation consult tribal cultural experts to understand tribal ways of knowing for that community. This process is often implicit. It can be brought to life through language, protocols for behaving, deeply held relationships within the community and with the land, and the people’s lived experiences. (LaFrance, Nichols & Kirkhart, 2012, p. 65).

While there cannot be a standardization of knowledge applicable across Indigenous nations (Kovach, 2010), there are principles that can guide the development of methods based on Indigenous ways of knowing. Principles identified in the framework described by LaFrance, Nichols & Kirkhart (2012) include:

The Indigenous Learning Circle (a group in Winnipeg, Manitoba) led the development of an Indigenous Evaluation bundle. Na-gah mo Waabishkizi Ojijaak Bimise Keetwaatino: Singing White Crane Flying North, is a bundle that provides a community driven approach to evaluating based on Indigenous principles congruent with individuals, families, leaders, and organizations in the north end of Winnipeg. The use of the term bundle is important to highlight. This purposefully moves away from using the term framework or toolkit, terms often used to describe evaluations. “A bundle is a sacred gathering of objects, ideas, gifts, and teachings that take place of the lifetime of an individual,” (Rowe & Kirkpatrick, 2018, p. 5). Na-gah mo Waabishkizi Ojijaak Bimise Keetwaatino: Singing White Crane Flying North is more than a collection of values and principles comprising an intellectual model - it is also emotional, physical, and spiritual elements. Ceremony was an important process in the development of the bundle, with attention given to local protocols and processes for seeking and sharing knowledge.

The intent of the bundle is to provide new opportunities for organizations to evaluate based on meaningful measures of success, to provide evidence for ways of working based on Indigenous values and practices, and to share these stories with funders with the overall goal of systemic change (Rowe & Kirkpatrick, 2018). The Indigenous Learning Circle, in the gathering of the Bundle, felt that it was important to make note of the use of the term evaluation:

From what has been learned so far there is no word for evaluation in any Manitoba Indigenous language. What comes close to the term evaluation reflects a personal process of deep reflection and contemplation. This is more about a process: looking back and seeing what worked, what didn’t, and then determining the path ahead. These concepts of evaluation involve taking stock and reflecting upon previous experience in order to move forward. This is a guided self reflection of who you are, where you are at, and where you want to be. This does not employ an external set of indicators upon these questions of where you “should” be. Rather, this creates the space for people to learn from their experiences, reflect on what has worked for them, celebrate their journey, and take that learning into their future. This is a purposeful reflection on self, self in family, and self in community. Based upon this definition, it is also clear that ceremony will be an important element at different points of the evaluation process (Rowe & Kirkpatrick, 2018, p.3).

The bundle provides guiding principles and values that are important to consider in planning, implementing, and reporting evaluation findings and shares questions for evaluators to consider and assess during their use of the bundle. It also provides opportunities to assess merit and worth based on traditional values, sharing congruent methods for knowledge gathering and dissemination.

A review of the literature also identifies the key role that evaluation advisory groups can play to ensure the design of meaningful evaluations. Advisory group members can be community leaders, evaluation stakeholders, Elders and traditional knowledge keepers. Their roles in an evaluation advisory group would be to ensure that cultural protocols are adhered to within an evaluation and that the evaluation provides relevance to the groups being served (Johnston-Goodstar, 2012).

Indigenous program developers understand that establishing what works best for Indigenous families, communities, and organizations requires a commitment to program monitoring and meaningful data collection (Morelli & Mataira, 2010). However, the continued push to fit values-based programming into pre-determined measures of success that are incongruent with Indigenous values and principles remains a challenge that must be addressed. Research continues to confirm the necessity of culturally based programming as a mechanism to counter impacts of intergenerational trauma in families. Logically, the field of evaluation must work to provide strong frameworks based on these ways of working. Indigenous evaluators must be trained in methods congruent with these frameworks with funders providing adequate levels of resources for these evaluations to be completed.

Foundational assumptions in the development of an Indigenous Evaluation Framework

An Indigenous evaluation framework requires grounding in principles based on Indigenous ways of knowing, being, feeling, and doing – in other words, a foundation of Indigenous worldviews. As a Cree woman working in the field of evaluation, I acknowledge that this is a complex undertaking that requires many considerations and careful attention. One consideration in this work relates to the themes of representation and generalizability. While there is an increasing recognition and use of the term Indigenous as an umbrella term in Canada, it is important to understand the connotations and meanings from which this term was born. Indigenous has roots in a global movement in the solidarity of Indigenous peoples who have experienced colonization worldwide (Manual, 1974). The solidarity comes with the recognition that common experiences and movements can work together to create a shift in the power structures that have oppressed Indigenous peoples globally for hundreds of years.

In Canada, the adoption of the term Indigenous was first through the acceptance of Indigenous scholars, who, in many cases connected with other Indigenous scholars worldwide at conferences supporting the gathering and sharing of Indigenous thought and experiences. Previously, government reference to groups of people comprising Indigenous peoples in Canada was through the term Aboriginal, which remains in use in many public references today. Both Indigenous and Aboriginal terms, from a governmental definition refer to Status and non- Status First Nation peoples, Métis, and Inuit. This grouping served a purpose in the development of the Indian Act and regulations about membership. A complex and imposed definition of who belongs and who does not belong according to the Indian Act has been a contentious issue for generations.

Recognition must be made that there is no one Indigenous culture or worldview; that Canada has many Indigenous languages; and First Nation, Métis and Inuit communities have greatly varied Indigenous cultures. The foundation of each of these cultures is relational and is in direct connection to the lands and the waters originally home to the different groups. This relationality across varied landscapes means that culture, ceremonies, language – each components of worldview that must be taken into consideration in so many cases when Indigenous cultures are being used to inform the development of programs, policies, and evaluations. It must be taken into consideration before using this document, meant to provide a framework to assist in the design of an evaluation.

A second caution or point for reflection is that Indigenous evaluation is not a practice in professional objectivity. We can never truly be objective, in fact, from this stance subjectivity is the point. We bring with us all of our experiences and contexts. This is how we see the world around us. Given the Calls to Action (2015) this challenge relates directly to purposefully and meaningfully reconcile within not only organizations and governments but also at individual, familial, and community levels. These Calls to Action are not held outside of our personal lives and only within our professional realms, but as evaluators we are a part of this process as well. Engaging in Indigenous evaluation is inherently political. How will evaluation respond to these Calls to Action?

In the design of evaluations using an Indigenous framework it will be important to consider the context of the program under evaluation. Part of this assessment is the program design itself. Does the program include use of traditional Indigenous knowledge, culture, ceremony, language, or processes for engagement in the design? Does the program include an Elder or traditional knowledge keeper? This framework assumes that a combination of these elements is present, and that the population being served through the program or organization is for the most part Indigenous or specifically targets an Indigenous population.

Whether this population is children, youth, families, men, women, couples, grandparents, and/or two-spirit individuals also informs the development of the framework. Considerations on how to work best with each of these groups as active participants in the evaluation and which processes and methods will best engage. With reconciliation comes the call towards decolonization, where human dignity, value, and worth are a necessary consideration in an Indigenous evaluation framework. When organizations are supported in the use of the framework, this means a commitment to design and implement meaningful evaluations that honour the gifts from the Indigenous peoples who are sharing their experiences.

The following section provides questions for both organizations and evaluators to reflect upon in developing a framework.

Questions to consider

When beginning to develop a framework, there are critical questions for both organizations engaging evaluators in the design and implementation of an evaluation and the evaluator themselves. Some questions are intuitive and are found in the development of evaluations based in Western methodologies:

  1. What is the purpose?
  2. Who is the intended audience?
  3. What is the level of stakeholder participation in the design, implementation, and dissemination of the results?

Other questions require the evaluator to reflect at a deeper level on their relationship with colonization, decolonization, and resurgence of Indigenous peoples as identified in Hart & Rowe (2014) earlier in this paper. The following questions are based on the work of the Indigenous Learning Circle in the development of the bundle. The group felt that it was necessary for organizations to ask the following questions in the development and implementation of an evaluation:

Based on my own work in this field, the questions posed by the Indigenous Learning Circle are ones that require an ongoing commitment to personal and professional development and to building reciprocal relationships within communities of practice that allow for reflexivity. In my experience, this requires a great deal of humility and embodiment of the values and principles that will be described below. A key point of learning for me was to be okay with times of unknowing, discomfort, and recognizing important moments of learning. The questions posed by the group are not a cursory gate to Indigenous evaluation in perpetuity, but rather require consistent self-examination, which can also be supported by the guidance and coaching of an evaluation advisory group.

Guiding Principles

As indicated in the word of caution above, Indigenous worldviews, values, and beliefs are highly contextual and relational to the environment with which they engage. While this is the case, and it is important to err with caution in making assumptions about ways of knowing, being, and doing that will be suitable in the design of evaluation. In my experience, there are guiding principles that are likely to be congruent for many Indigenous nations in Canada. These principles are enacted in relation to the contexts that I spoke about earlier and therefore can look different in practice depending on the land, language, and culture within which this is being used. These are broad principles and are meant to be applied if they make sense in relation to local contexts, with flexibility to best meet the needs of the community, organization, or program being evaluated.

An important area to pay attention to is how this framework is used in an urban-Indigenous setting, which can incorporate even more diversity and have peoples represented from all over the lands of Canada. While this is not meant as a stumbling block or as a barrier upon which to remain stuck, it does require open conversations about how best to design using this framework in a way that honours this diversity.

The principles described below come from two key sources. The first, Dr. Michael Anthony Hart, a Cree scholar and member of Fisher River Cree Nation, has provided guidance in the development of an Indigenous research paradigm (2010). The second source is the bundle developed by the community knowledge gathering process undertaken for the Indigenous Learning Circle (2018).

Hart (2010) identifies eleven values, listed below, that are essential to an Indigenous research paradigm and reflect the ethical manner in which researchers must carry themselves and work with the community of focus.

  1. Indigenous control over research
  2. A respect for individuals and community
  3. Reciprocity and responsibility
  4. Respect and safety
  5. Non-intrusive observation
  6. Deep listening and hearing with more than the ears
  7. Reflective non-judgement
  8. To honour what is shared
  9. An awareness and connection between the logic of the mind and the feelings of the heart
  10. Self-awareness
  11. Subjectivity (p.9-10)

Core values identified by the Indigenous Learning Circle in the gathering of the bundle include:

  1. Relationships
  2. Seven Sacred teachings as values
  3. Trust
  4. Respect
  5. Strength-based
  6. Sharing food
  7. Reciprocity
  8. Responsibility
  9. Cultural Safety
  10. Attending to mind, body, spirit, and heart
  11. Acknowledging the continuum of existence (Rowe & Kirkpatrick, 2018, p. 13-15)

There is overlap between both sets of values, serving as guiding principles. The following paragraphs will outline my own experiences and observations as an evaluator using these principles as a guide to my work with families, organizations, and communities

I have come to learn the importance of community as the driver of the work being done. The first way that this occurs is by recognizing that the values listed above may be removed, adapted or, added to be based upon this understanding. The lists above are meant to be reflections for the evaluator and the organization to begin a conversation about appropriate and necessary elements for evaluating based on Indigenous principles and methods. Community as a driver in control of the research means that leadership, staff, and participants can each be consulted in the development of an evaluation that will best meet their needs. The metaphor of the community driving can be taken further – what vehicle, who will be the co-pilot, who will the other passengers be, what necessary supplies will be needed on the journey, and how will the group prepare to travel in a safe way to meet the final destination? Reframed into evaluative design questions to work through with community/organizations include: What purpose will the evaluation serve for the organization? What methods will best suit capturing data that will meet the purpose? Does the organization have a strong understanding of the processes that are promoting transformation and what data will be collected to capture these stories? For example, while surveys and key informant interviews may be methods that are more familiar, it could be helpful to consider methods that are relational or arts-based.

The evaluator who uses the framework must be aware of the roles that they are taking on in this process. This means that the evaluator has the ability to critically reflect on their role in the work as facilitator, capacity builder, mentor, advocate, and ally. This can occur when there is an opportunity for embedded evaluators who are working based on relational accountability. In other words, how is the evaluator responsible to the relationships that they hold in the community and how will they remain accountable to the participants, organizations, and knowledge being shared in their work together? Two skills that are critical for this to occur are the act of listening deeply and non-intrusive observation. Field notes, reflection in ceremony, and conversations with traditional knowledge keepers have been methods I have used for deep reflection. Another important aspect that ensures we as evaluators remain responsible to the relationships that we hold can mean supporting identified priorities in the framing of the evaluation and ensuring that meaningful data is gathered that will serve the organization in the long term. Considering the role of evaluator as advocate and ally reinforces that research and evaluation are inherently subjective and political in that when designed appropriately and from the needs of the organization/community they serve a larger purpose of equity and social justice.

The responsibility that comes with holding the stories of participants and organizations is not taken lightly within an Indigenous perspective. Stories are sacred and must be held with that in mind. How they are presented and for what purpose are important questions. In order to honour what has been shared I have often returned to participants to verify that I am representing their stories in a way that is accurate and that validates their experiences. Accurate representation of the story of a community and organization is also considered within this value. It can mean preliminary reactions and conversations about what is held within a draft report, making adjustments or clarifications as necessary. This speaks to respecting community context.

How knowledge is sought, cared for, and shared can include participation in ceremonies and facilitated through the use of medicines led by traditional knowledge keepers and Elders. In my work there has been considerable value in developing evaluation partnerships with knowledge keepers to ensure their insights also guide the work all the way through. In my experience, this means working in partnership with traditional knowledge keepers and organizations to identify what would be appropriate given the community context and identifying the resources required. Pipe ceremonies, sweat lodge ceremonies, smudging, and feasting have all been activities identified and led by knowledge keepers at appropriate times during evaluations.

The principle of reciprocity connects to relational accountability and responsibility to roles. What will be left behind at the end of the evaluation? Is the work extractive or are there valuable opportunities, resources, learning, and insights that lead to increased capacity? Entering into an Indigenous evaluation relationship is a commitment to ensure that these questions have meaningful answers for participants. In my experience, the role of evaluator as capacity builder is directly connected to relational accountability. What will I, as the evaluator or researcher, leave behind in order to ensure a deeper understanding of the knowledge, processes and skills necessary for this work? This could ultimately begin to meet a larger goal of Indigenous evaluation, to have more trained Indigenous evaluators completing this work.

Indigenous ways of knowing are fluid ways of knowing that have come from teachings shared from generation to generation through storytelling. The telling of stories is an important mechanism for sharing knowledge that comes from traditional languages, which place an emphasis on verbs. Knowledge is gained through dreams and visions via intuitive and introspective processes where there is opportunity for deep meaning making and inner journeys (Hart, 2010; Kovach, 2010). Ermine (1995) describes mamatowisin, a Cree term meaning, “the capacity to tap the creative life forces of the inner space by the use of all the faculties that constitute our being - it is to exercise inwardness” (p. 104). Indigenous ways of knowing, briefly described here, must be considered in the design of an Indigenous evaluation framework, knowledge gathering, analysis, and presentation of findings of the evaluation. Due to the centrality of spiritual and ceremonial practice in the gathering of knowledge, it is essential for this to be guided by Elders or traditional knowledge keepers. “Thus, an Indigenous research paradigm is structured within an epistemology that includes a subjectively based process for knowledge development and a reliance on Elders or individuals who have or are developing this insight,” (Hart, 2010, p. 8).

In practice, the enacting of these values includes facilitated self-reflection and learning, storytelling, land-based activities including participation in ceremony and the use of sacred fire, drumming and singing, and the use of traditional medicines (Rowe & Kirkpatrick, 2018).

One example of how I have partnered with an organization in the design and implementation of an evaluation based in Indigenous methodologies is in the first use of the bundle. The bundle, developed by the Indigenous Learning Circle, was first used in practice with the Community Education Development Association (CEDA), an organization in the north end of Winnipeg, Pathways to Education program to answer the question: how has CEDA impacted your family? Indigenous families with students in grades 9-12 who had participated in programming were invited to share their experiences in an evaluation designed based on the work of the bundle. The knowledge gathering process began with a teaching from Elder Don Robinson on the Circle of Courage, a pipe ceremony, and a circle where introductions were made, families spoke of their connection to the program, and were invited to participate in the evaluation with an offering of tobacco. Methods used in this evaluation were arts-based and used the circle as a process for sharing. Leaders at CEDA and the families provided feedback on the use of the bundle in the evaluation. Strengths included the use of ceremony, the inclusion of an Elder in the knowledge gathering, and the use of an arts-based method for exploring the questions. Limitations identified during the use of the bundle was the design of the arts-based method did not directly align with the Circle of Courage model of the organization, staff had hoped for families to identify more gaps, barriers, and challenges, and the lack of time to complete an analysis based on Indigenous ways of knowing, being, and doing (Rowe & Kirkpatrick, 2018).

It is important in the development of Indigenous evaluations that the evaluator be mindful to reflect on the congruence between methods and the philosophical values of an Indigenous paradigm (Kovach, 2010). One of the limitations described in the example above is the inability of the method to meet the full needs of the organization. I believe that a quick time period and my remote location both were factors in this gap. While I had a pre-existing relationship with the organization and the leadership, I have reflected that more time to understand the purpose and goals of the organization was necessary. One method that would have been useful in setting this foundation is described by Kovach (2010), who outlines the use of a conversational method for knowledge gathering. The conversational method is congruent with the orality of Indigenous ways of knowledge transmission. This method is relationally based, in that it requires a commitment from all participants to learn and share within a collective tradition. Kovach reflects:

The use of a conversational method within an Indigenous research framework has several implications for the researcher in relation. For the conversational method, the relational factor - that I knew the participants and they knew me - was significant. In each case I had known or met participants prior to the research. With this method the researcher must have a certain amount of credibility and trustworthiness for people to participate in the research,” (2010, p. 46).

This method, while used in knowledge gathering within Indigenous research, is also helpful in setting a strong foundation and partnership for the development of an Indigenous evaluation framework.

While knowledge gathering in the evaluation process is important, a framework based on Indigenous principles must also include reciprocity and responsibility to the knowledge that was shared. More attention must be paid to the dissemination and sharing of the results that were found within the evaluation process. This can include a series of community events or gatherings, videos, user-friendly materials that highlight the work completed, and sharing of the findings in traditional and non-traditional venues. Often during community events that share evaluations there can be space for reflection and reactions to the learning that took place. This can be a positive opportunity to not only connect with the broader community, but to also potentially understand more deeply community context that may not have been readily accessible prior to a reflection on the results.

Meaningful community involvement in the design of Indigenous evaluation must occur at the beginning, must be fully supported with resources, and must be given the time and space to build capacity. In my experience as an evaluator, community truly must be the driver. This means investing time and attention to the building of relationships to come to common understandings about priorities and needs. This also means designing a process and using methods that will ensure to meet these needs. While a framework can outline principles and values upon which to base evaluation activities, it is also just as much about the evaluator themselves and the way that they work with organizations/communities.

Summary

Given the diversity of cultures, languages, urban and reserve-based Indigenous populations across Canada, the task of implementing an Indigenous grounded evaluation framework can appear daunting. This paper is a call to the multiple levels of government to recognize that this is not going to be an easy action to support, but it is clearly time. Similarly, this is a call to evaluators to recognize and address the challenges identified in this document and to reflect deeply on the questions posed about whose voice and priorities must lead the way and how non-Indigenous evaluators can support the work of building capacity with Indigenous communities, organizations, and evaluators.

Opportunities stemming from the learning in this paper include:

A challenging question, beyond the scope of this document overall, is how can the government, in requiring evaluations of programs that are funded, acknowledge the diversity across Indigenous communities and cultures? Part of a solution lies within the development and application of Indigenous evaluations that are designed from the beginning and in collaboration with Indigenous communities. Right now there is an opportunity to acknowledge that there is not a one size fits all approach and that what we know as a field about how to evaluate is not congruent with Indigenous knowledge, practices, and ways of healing.

Indigenous evaluation is a critical field in development. It is connected to decolonization and reconciliation, both of which are actively being explored by individuals, organizations, and governments across Canada. If the purpose of evaluation and our roles as evaluators is to understand and measure success in order to fund programs and organizations that are making a difference in the lives of Indigenous individuals, families, and communities – it is time to pause and reflect. Indigenous evaluation has a central role in sharing outcomes of organizations working with Indigenous peoples. This requires not only evaluators, but also organizations and funders to become aware of how to design and implement this type of research. The previous sections outlined a history of evaluation and asserts the need for more Indigenous trained evaluators, resources to support this work, and granting organizations to make this a priority.

References