3 Evaluation Methodology
The evaluation was guided by an evaluation matrix (evaluation questions, indicators, and data sources) which was developed through the evaluation scoping and design phase. The evaluation issues and questions are presented in Appendix A. The evaluation included multiple lines of evidence, and employed the data collection methods described below.
3.1 Document Review
The document review provided descriptive information on the Secretariat’s activities, as well as information responding to evaluation questions. The review was ongoing through the evaluation conduct phase and included the following types of documents:
- Administrative and internal documents; and
- Publicly available departmental and other government documents, such as federal Budgets, Budget Speeches and Mandate Letters.
3.2 Key Informant Interviews
A total of 32 interviews were conducted with stakeholders representing the following groups:
- Justice Canada representatives (n=16);
- Indigenous partner representatives (n=3);
- OGDs (n=6); and
- Reconciliation Secretariat representatives (n=7, including 2 interviews with former staff members).
3.3 Limitations and Mitigation Strategies
Given the nature of the Reconciliation Secretariat’s role, responsibilities and mandate, measuring its direct impact was challenging. Activities related to policy and provision of strategic advice are, in general, very hard to monitor and evaluate. It is very difficult to isolate the impact of a particular intervention from the influence of other factors and various actors. To mitigate this limitation, the evaluation focused on anecdotal evidence gathered through interviews.
In addition, there was limited information on the achievement of the outcome pertaining to progress on the IJS given that the Secretariat’s work on this file is relatively recent (the IEF has only been issuing funding since 2021-22). Additionally, evidence of capacity building within Justice Canada was limited. Therefore, the measurement of some outcomes relies heavily on interviews.
Securing participation from Indigenous partners and recipients also posed a challenge. In the end, only representatives from three Indigenous organizations were available to participate in the evaluation. These organizations receive a multitude of invitations and cannot accept every request to share their perspectives. Efforts were made to be flexible regarding the timing of interviews, which were booked over a two-month timeframe to maximize the available times to meet with these respondents. To make contact, three emails and two phone follow-ups were issued after the initial notification email from Justice Canada.
Respondent bias is also a limitation since the list of interviewees was provided by the Secretariat. Also, since participation in an interview was voluntary, there is a possibility that individuals with less to say, a lack of awareness of the Secretariat or having primarily negative feedback might have opted not to participate.
- Date modified: