Evaluation of Federal Support for Family Justice

3. Evaluation Scope and Methodology

3.1 Scope of the Evaluation

In preparation for the evaluation, an evaluation matrix was developed in consultation with an Advisory Committee consisting of representatives from the Department’s FCY Section, IAID and RSD. The matrix detailed several evaluation questions to align with the Treasury Board’s 2009 Evaluation Policy, which was originally intended to cover all areas of work undertaken by the FCY Section that extended beyond family justiceFootnote 5. However, at the time that the evaluation began in 2017-18, the scope was calibrated to focus more exclusively on work in the area of family justice by the Department. This approach aligns with the requirements under the new 2016 Policy on Results.

3.2 Lines of Evidence

The evaluation relied on multiple lines of evidence to arrive at findings and conclusions. Data collection occurred throughout the period covered by the evaluation, although most of the work was completed in 2017 and 2018. In addition to document, literature and file reviews, eight studies assessing costs were undertaken. Interviews were conducted with 26 key informants, including 16 representatives from the Department and 10 PT representatives, and a focus group was held with a further 20 PT representatives. Surveys were conducted with other federal and PT counterparts who participated in various training sessions or webinars. Surveys were also conducted with participants benefiting directly from programs and services administered by provinces and territories and funded, at least in part, by the Department. This included surveys of 16,000 PEP participants and over 650 mediation service participants, as well as a longer term follow-up survey with 283 people who participated in parent education, mediation, or family law and information centre services. The lines of evidence are summarized in the chart below (see Appendix C for more information).

Figure 1 : Overview of the Lines of Evidence

Figure 1 described below

Figure 1 : Overview of the Lines of Evidence

Figure 1 depicts the six different lines of evidence used for the evaluation data collection, with corresponding tools. The six lines of evidences are:

  1. Surveys, which include the FCY Partner Stakeholder Survey, the SFF Funded Exit and Follow-up Survey, the FCY Event Surveys, and the 2016 National Family Law Program Survey;
  2. Justice Canada Studies, which include the Environmental Scan, the Family, Children and Youth Legal and Legal Policy Study, the Overview and Assessment of Approaches to Access Enforcement;
  3. File and Document Review, which includes the file review of 36 annual progress reports from PTs, the completion reports on 28 SFF projects, performance data (FCY Metrics), financial and other data, and other documents;
  4. Key Informant Interviews, which include 16 departmental representatives, and 10 PT representatives;
  5. Focus Group, which includes 1 focus group with 20 PT representatives; and
  6. Process Mapping and Costing Studies, which include 5 studies focused on costs of ISO processes, 1 study focused on administrative recalculation, and 2 studies focused on court-based family services.

3.3 Evaluation Limitations and Mitigation Strategies

The timing of the evaluation did not allow for a detailed analysis of the impacts of the recently introduced CFJF, which replaced the SFF. To address this issue, the evaluation report describes the nature of the changes and considers the current context (related to the new permanent mandate, the recent introduction of the CFJF, and the bill introduced to amend the Divorce Act, the Family Orders and Agreements Enforcement Assistance Act and the Garnishment, Attachment and Pension Diversion Act) in the development of conclusions and recommendations.

Many interviewees were directly involved in federal family justice activities (i.e., providing support and services or benefiting from services), which can result in a positive response bias. In order to minimize bias, multiple lines of evidence (qualitative and quantitative) were used to triangulate findings and increase data reliability.

The wide range of activities, outputs and achievements reported by  SFF recipients, combined with the structure of the reporting template, made it difficult to meaningfully aggregate and summarize results. To mitigate the impact, the evaluation incorporated an extensive file review to summarize the results reported by provinces and territories. The reporting requirements and templates have been strengthened with the implementation of the CFJF in 2017-18.