Supporting Families Experiencing Separation and Divorce Initiative Evaluation

3. Methodology

3.1. Issues and Questions Addressed in the Evaluation

The five broad issues (see Table 2) addressed in the evaluation provide a framework for evaluating the direct, intermediate and long-term outcomes of the SFI.

Table 2. Evaluation Issues and Questions
Evaluation Issue Evaluation Question
Relevance
Continued Need for Program To what extent does the SFI address the needs of Canadian families experiencing separation and divorce?
Alignment with Government Priorities To what extent are SFI goals and objectives aligned with federal government and Justice Canada priorities?
Alignment with Federal Roles and Responsibilities To what extent is the SFI aligned with the federal government's roles and responsibilities in the area of family justice?
Performance
Achievement of Expected Outcomes To what extent has the SFI made progress toward strengthened federal capacity to respond/address needs of families experiencing separation and divorce?
To what extent has the SFI made progress toward improved capacity in provinces and territories to provide and deliver family justice services?
To what extent has the SFI made progress toward expanded accessibility of family justice programs and services?
To what extent has SFI made progress toward enhanced awareness and understanding of parental obligations, compliance, and the family justice system?
To what extent has the SFI made progress toward improved efficiency in enforcement tools and services?
To what extent has the SFI made progress toward enhanced capacity of parents to reach appropriate custody, access and support agreements?
To what extent has the SFI made progress toward enhanced ability of parents to comply with custody/access responsibilities?
To what extent has the SFI made progress toward increased parental compliance with financial support obligations?
To what extent has the SFI made progress toward increased effectiveness of the family justice system in addressing the needs of families experiencing separation and divorce?
Demonstration of Efficiency and Economy Has the SFI's resource utilization been appropriate in relation to the activities and outputs produced and its progress toward expected outcomes?

3.2. Approach

The evaluation methodology consisted of multiple lines of evidence including: a document and data review, key informant interviews, surveys, case studies and a file review. Triangulation of these multiple sources of data was used to verify and validate the findings and to arrive at the evaluation conclusions.

Document & data review

This review included both internal and publicly-available documents. More specifically, the review included: FCY administrative data; departmental communications statistics; enforcement reports and data; committee minutes and planning documents; FLAS statistics; best practice documents; memorandums of understanding; number of secondary research studies; Statistics Canada data; and exit survey data from parent education programs, mediation programs and train-the-trainer high conflict family justice professional training events. Footnote 12

Key informant interviews

The key informant interviews conducted for this evaluation addressed the majority of the evaluation questions and are a key line of evidence. Thirty-five key informant interviews were conducted by telephone. These included both individual and telephone group interviews using interview guides tailored to each respondent group. The interviews included 13 judges with extensive experience in the family justice system and 22 individual and group interviews with 39 federal, provincial and territorial family justice and maintenance enforcement program representatives. These included all the provinces and territories, all FCY units, the Communications Section of Justice Canada and the CCJS at Statistics Canada.

Family lawyer/mediator survey & interviews

A web-based survey of practising family and divorce lawyers and mediators was conducted in May and June of 2013. Survey involvement was voluntary and included 667 participants: 18% were family mediators; 57% were family lawyers; 20% were family lawyers who mediate; and 6% were others who answered the survey including program administrators, judges and social workers with family justice responsibilities. From the initial survey, 33 respondents self-selected to participate in a follow-up telephone interview to address some topics in more depth.

National parent surveys

Two national parent surveys were included in the evaluation. The national SFI Parent Survey was conducted between 2012 and 2013 involving telephone interviews with 1,200 parents in ten jurisdictions Footnote 13 about their custody access needs, agreements, trajectory of agreements, compliance levels and problems, and participation in family justice services such as mediation and parent education. Survey respondents were selected from research consent forms distributed by the provinces and territories in parent education and mediation programs.

A second national parent survey was commissioned by FCY in 2010 involving 1037 separated/divorced parents whose union had dissolved in the period 2003-2010. Survey data focussed on arrangements for children (living, decision-making and financial support), the inter-parental relationship and other variables focused on the post-separation/divorce family.

Case studies

Twenty-four case studies of SFF projects were undertaken as part of the evaluation. The case studies involved a review of SFF contribution agreements, as well as telephone and in-person interviews with the programs to assess how the projects or services assisted families and contributed to increasing the effectiveness/responsiveness of the family justice system.

Focus groups

In preparation for the evaluation, 12 focus groups were held with 70 parents in six jurisdictions Footnote 14. The purpose of the focus groups was to obtain feedback on the needs of parents and explore potential outcomes of the various family justice services in which they participated.

File review

SFF grants and contributions information on file was reviewed for the evaluation using templates to ensure systematic extraction, aggregation and analysis in an inter-relational database.

Program files from three components of the SFF were analyzed: the Family Justice Initiative (FJI) which contributes to a range of projects and activities in the provinces and territories; the Pilot Projects component, which supports one time provincial and territorial innovative pilot projects; and the PLEI and Professional Training component which provides funding for projects coordinated by NGOs. Results from the SFF analysis included all funding provided through the SFF between 2009 and 2012 to ensure that all reported data was final.

3.3. Limitations

The evaluation encountered a few challenges and limitations, including reliance on jurisdictional representation, differences in how data is collected and reported, difficulties in assessing both impacts of social programs and policy in general.

Jurisdictional representation

Given the absence of reliable data on the overall population of family lawyers or family mediators in most jurisdictions, it was not possible to ensure jurisdictional representation in the family lawyer/mediator web survey. Also for the exit surveys and the SFI Parent Survey, three jurisdictions Footnote 15 did not participate so these results are not fully national in representation.

Differences in how data is collected and reported

The review of SFF grants and contributions Footnote 16 files was challenging in many aspects as there was no consistent definition of a project to ensure the systematic categorization or aggregation of the data. Although a template for provincial and territorial reporting was developed through provincial and territorial collaboration, activities were still reported in different ways. Some applications or reporting documents identified a single larger activity title and dollar amount, while others broke down activity titles and dollar amounts into smaller units. There was a more consistent definition of project under the other components of the SFF (Pilot Projects and PLEI and Professional Training) that allowed for systematic categorization and aggregation of data. The categorization of data into smaller components was required to achieve consistency in how the projects were described. Although a template was used for the evaluation to systematically aggregate and analyze the data, it could not address all the variability found in the file documents and judgment was used to determine the most appropriate categorization of data to address the outcomes of the SFI.

Difficulties assessing impacts of social programs and policy

A significant part of the FCY activity is related to policy initiatives. In general, this type of activity is difficult to assess in quantitative terms. Rather, it is reflected in particular contexts such as CCSO-FJ activity or the discussion of legislation.

As with many social programs, attributing results to an intervention can be difficult. Families experiencing separation and divorce use a variety of resources to assist them and, in some cases, involvement in SFI funded activities such as parent education, although valuable, are of a short duration and other factors may ultimately outweigh the benefits. It should also be noted that results from parent surveys and research literature have concluded that factors such as the levels of parent conflict and cooperation are the most significant contributors to parental compliance with custody, access and support and that these are primarily influenced by personal/interpersonal characteristics and communication styles. High conflict parents have complex and pre-existing needs that may be beyond the scope of many existing services to address.

In addition, the time frame for the follow-up with parents in the SFI Parent Survey was limited. All the respondents had been involved in an SFI supported service within the previous three years.

Mitigation Strategy

Several mitigation strategies were used to address these issues. The use of data extraction templates provided a consistent and systematic framework for the extraction and analysis of SFF file data despite the challenges previously described.

The ability to attribute federal, provincial and territorial outcomes directly to the SFI was partially addressed through surveys and interviews with representatives and family justice professionals to qualitatively assess the impact of the SFI directly on the behaviour and attitudes of parents. Parent opinion was collected through several lines of evidence and particularly in relation to a national parent survey conducted for the evaluation which consisted of 1,200 parents in ten provinces and territories. Exit surveys were also used to collect parent opinion on the impact of parent education and mediation programs.

The expansion of the family lawyer/mediator web survey (to 667 participants, from an original target of 250) allowed for a greater sampling of this group and more jurisdictional representation.

The use of multiple data sources allowed for the triangulation of findings to more clearly define areas of consensus. This strengthened the conclusions arising from the findings.