Evaluation of the Aboriginal Justice Strategy

3. Detailed Methodology

The following section outlines the methodology used for the evaluation. The evaluation was conducted with both internal and contract resources. The issues and questions addressed in the evaluation are included in Appendix A and the data collection instruments are included in Appendix B.

3.1. Document Review

The document review provided information for almost all evaluation questions, as well as detailed information on what has been achieved at the community level. In total, 27 documents were reviewed for this line of evidence, including examples of agreements, internal reports, program administration documents, program activity documents, meeting minutes, internal policy documents, and academic research papers.

3.2. File Review

A file review was conducted of funded projects under the Capacity-Building Fund between 2012 and 2015. Descriptions of these projects were used to identify the types of activities undertaken by the five types of eligible funding categories as per the Terms and Conditions of the Fund. Every funded project was counted and the total funding for each project in each of the five categories was summed. Based on a description of funded activities, their exact type (e.g., workshops, conferences or meetings) was also categorized. Table 2 presents the data obtained.

Table 2: Total Number of Projects and Total Funding in each Eligible Funding Category
Capacity Building Fund Objectives Count Approved Amount ($)
Training/development (with no community-based justice programs) 43 1,412,186
Training/development (for existing community-based justice programs) 69 2,033,318
Support reporting and data management (in current AJS communities) 45 296,790
New community-based justice programs (underrepresented and regional imbalance areas) 11 263,446
One-time/annual events (build bridges, trust, and partnerships) 21 886,465
Total 189 4,892,205

3.3. Surveys (Police/Crown and Community Justice Representatives)

Two national online surveys were administered for the evaluation. One survey for RCMP members, other police force members, and Crown counsel who work with the communities served by community-based justice programs. A second survey was administered for CJWs in the communities served by community-based justice programs. The surveys assessed level of awareness of the AJS, referrals to these programs, need for and effectiveness of AJS programs, and experience with the AJS.

The final response rate for the survey of CJWs was 68%, which is considered a very good response rate for an on-line surveyFootnote 5. The majority of respondents to the Crown and police representatives survey were RCMP officers in areas where there were AJS programs (73%), followed by Crown representatives (23%) and Aboriginal police representatives (4%).

Table 3: Types of Respondents to Crown and Police Representative Survey
  Number Percent
Crown prosecutor or other type of attorney/Crown 44 23.2
RCMP 138 72.6
Aboriginal police (e.g., band, council) 8 4.2
Total 190 100.0

Respondents to the Crown and police representative survey were most likely to currently be working in Saskatchewan (39%), Alberta (10%), British Columbia (10%), Nunavut (9%), Manitoba (8%), and Yukon (7%). The distribution of respondents does not represent the distribution of AJS programs across the country, which is important to keep in mind when interpreting survey results.

3.4. Economy and Efficiency Review

The evaluation assessed resource utilization of the AJS through three perspectives: economy, operational efficiency and allocative efficiency. Financial data and key informant (KI) interviews were examined to assess how resources were used and whether there are ways to increase the efficiency of how the AJS is being delivered. In addition, two analyses were undertaken to assess the impact the AJS-supported community-based justice programs have on the rate of recidivism of participants (see Appendix D) and costs related to processing a case through the MJS courts.

3.5. Key Informant Interviews

KI interviews sought to gather contextual information about the program to inform the analysis of the other lines of evidence. It covered most evaluation questions and focused on relevance and performance of the AJS.

Interviews with representatives from the AJD and other departmental staff were conducted at AJD National Headquarters (NHQ) in person and in the regions both in-person and by phone. Provincial and territorial representatives were also interviewed, usually by phone. The list of interviewees and their contact information was supplied by program representatives, after discussing the appropriateness of different types of respondents. The following groups of respondents and actual numbers of interviews were conducted:

The distribution of the selected respondents and completed interviews across the respondent groups was monitored to ensure that the final distribution achieved a good balance between internal and external stakeholders.

The interviews were semi-structured and did not last more than one hour in order to ensure a high response rate and to keep respondent burden to a minimum.

Once the interviews were completed, the interviewer input the data and information collected from the interview into a central database. The database allowed for the analysis of the qualitative information by interviewee group, as well as by evaluation question. This facilitated drawing out common themes by question across respondents either within specific stakeholder groups, or across groups.

3.6. Case Studies

Purpose and Approach

Six case studies were conducted across Canada (see Appendix C). Case studies focused on obtaining stakeholder assessments of progress with the local Indigenous community-based justice programs and their objectives, factors influencing progress, and identifying promising practices that may be applicable to other participating communities. The case studies obtained descriptive and evaluative information from a range of community members including Indigenous CJWs, justice committee members, community leadership, Elders, and representatives of health and social services, police and Crown serving the communities. In addition, the AJS administrative program files for the selected communities were reviewed.

Community Selection Strategy

Programs were chosen based on their suitability for case studies, in that they were recognized as representing well-developed AJS programs. Since there was no attempt to have the case studies be representative of the full population of AJS programs, an effort was made to enable the case studies to draw on the experiences of well-established programs to identify the strengths and weaknesses of different approaches to Indigenous community-based justice, the factors that can influence success, and examples of effective approaches that may benefit other similar programs across the country.

Programs were selected to ensure a fair regional representation, as well as ensuring a mix of urban, rural and remote communities that included Inuit, First Nation and Métis communities.

Programs Selected

  1. Tsilhqot'in Community Justice Program: (Western, remote, First Nation on-reserve);
  2. Saskatoon Tribal Council Community Justice Program: (Central, urban and rural, First Nation Tribal Council model);
  3. Métis Community Justice Program: (Central, urban, Métis);
  4. United Chiefs and Councils of Mnidoo Mnising Community Justice Program: (Ontario, rural, First Nation Tribal Council model);
  5. Elsipogtog Restorative Justice Program: (Atlantic, rural, First Nation on reserve); and
  6. Kwanlin Dun First Nation Social Justice Program: (North, urban, First Nation on reserve).

3.7. Methodological Limitations and Mitigation Strategy

Key Informant Interviews

KI interviews are used for many evaluations to obtain qualitative feedback on the program from key individuals but are vulnerable to sampling bias, response bias, sensitivity of respondents to the questionnaire and interviewer bias. To mitigate this, the evaluation ensured that the list of KIs to interview was a balanced list so that a knowledgeable pool of respondents and a variety of internal and external perspectives were gathered. Further, triangulation of KI evidence with other lines of evidence helped balance out any potential biases.

Case Studies

Case studies are useful to the evaluation as they illustrate impacts and insight into the program’s work at the community level, and examine impacts in individual communities. Because of the importance of ensuring cultural sensitivity during data collection, all case studies were conducted by senior evaluators and/or Indigenous evaluators who were able to combine observational data collection with meetings with managers, CJWs and beneficiaries of AJS community-based justice programs.

Survey

Surveys can be very useful in evaluations to generate quantitative data from a large group of program representatives or stakeholders in a cost-efficient manner. However, surveys can be susceptible to threats to reliability and validity of the data collected (e.g., sampling bias, non-response bias, sensitivity of respondents to the questionnaire, interpretation of the questions). Online surveys can also be prone to low response rates or partially completed surveys, particularly when the respondent group is not highly engaged in the topic or receive frequent requests for feedback. To maximize the reliability of the survey data, the surveys were written in accessible language and were relatively short. To further maximize the response rate, reminders were sent to contacts for whom direct contact information was available. Time was also provided for external stakeholder contacts (e.g., police, RCMP, Crown) to follow internal procedures for participation in external evaluations.

As the survey sample was developed through referral sampling, the number of people the survey actually reached is unknown. Although invitations were sent to an unknown number of Crown representatives, Indigenous police representatives and RCMP officers, from what is known, it is clear that a great percentage of the targeted population did not participate in the survey, and there are certainly sample biases (e.g., number of participants from some provinces is not proportional to the number of programs). In some smaller provinces, however, the sample is likely representative of the whole population. For this reason, survey findings from the Crown and Police representatives cannot be generalized to the knowledge or attitudes of the population and weight could not be employed to balance the sample.

Economy and Efficiency Review

The two analyses undertaken to assess the allocative efficiency of the AJS also had a number of limitations that were mitigated as best as possible during the evaluation. Additional information on the methodologies, limitations and mitigation strategy used for these two analyses can be found in Appendix D.

For the recidivism analysis, a main limitation was the lack of a true experimental design, as practical and ethical constraints precluded the random assignment of persons to participant and comparison groups. Pre-existing differences between the participant and comparison groups could lead to differential outcomes with respect to re-offending. However, to mitigate this limitation, a statistical approachFootnote 6 that could control underlying differences between the participant and comparison groups was utilized.

With regards to the costing analysis, although the full costs of the AJS were included in the analysis, only estimates of some MJS court-based (court services, prosecution and legal aid) costs were included through examining public accounts. Additional costs associated with the administration of justice (e.g., police, probation, custody) and those borne by the community for both the AJS and the MJS were excluded from the analysis. This reduces the ability to understand the full impact of the AJS, although it provides an indication of potential cost savings for one point of contact with the MJS.