Unified Family Court, Summative Evaluation

3. Evaluation Methodology

This section of the report summarizes the evaluation objectives, methodological approach and research activities undertaken to complete the Summative Evaluation of the Unified Family Court.

3.1. Evaluation Objectives

The summative evaluation was designed to measure the impact of the UFC model by determining the outcomes and effects associated with UFCs. Specifically, the evaluation addressed the following issues and related outcomes:

  1. Relevance: The continued relevance of the UFC concept and objectives.
  2. Success: The extent to which the UFC model has achieved its planned results (i.e., intermediate and longer-term outcomes).

In addition to the two issues cited above, issues related to cost-effectiveness and alternative models were also examined, although not in depth.

3.2. Research Design

A quasi-experimental design was used to determine the extent to which the UFC model achieves its intended objectives. To this end, a comparison group of courts that operated according to the more traditional model was identified and the results compared to those of selected UFCs. The evaluation was based on the feasibility and research design completed in 2007.16

In this evaluation, the main issue to be addressed is the extent to which the UFC model is effective in achieving its intended outcomes. The hypothesis is that cases processed through a UFC would have better outcomes with respect to a less adversarial and more coordinated approach to case resolution than would cases processed through a traditional court model that did not have the same supportive elements in place as in the UFC model.

3.2.1. Selection of UFC and Comparison Locations

The quasi-experimental design relied on the development of a comparison sample. In particular, court locations in each group (i.e., UFC and traditional) had to be clearly distinguishable with respect to the characteristics that define the UFC models and the more traditional court model. At the same time, other possible influencing factors had to be controlled such as the demographic characteristics of the population served in the selected locations, case volumes and/or any idiosyncratic business procedures/practices in place at the court registry or in the court.

All of the comparison sites in the evaluation provide access to some FJS. The difference in the range of FJS provided for both UFCs and traditional courts presented significant challenges with respect to measuring the performance of the UFCs overall. These challenges are described in more detail at the end of this section.

An inventory of court locations by province/territory was established, documenting key attributes (e.g., population size and characteristics, FJS available, dispute resolution mechanisms available, relevant family law rules/procedures). From this inventory, four UFC locations and four traditional court locations were selected for the evaluation. The site selection was based on a number of criteria, including population density and characteristics of the population served by the court to ensure that the sample of cases reviewed for the evaluation would be more likely to resemble the sample of cases selected from the UFC sites.

Implementation of the research design in the summative evaluation was dependent on the provinces/territories agreeing to participate. The proposed court locations were subject to the approval of their respective jurisdictions. After consulting with the provinces and territories, some of the original locations were approved, while alternative locations were proposed in both the UFC and comparison groups.

It should be noted that some of the alternative locations did not meet the selection criteria identified in the research design. For example, some of the UFC sites included in the evaluation were struggling with static judicial resources in a period of rapid population growth and increasing case loads, making it difficult to rely solely on the specialized bench (i.e., generalist judges were cycled into help with family cases). At other UFC sites, the full conceptual UFC model was not implemented (e.g., there were limited or no intake services). The degree to which the population demographics of the UFC locations matched the traditional court locations was also affected. Ultimately, these differences had a significant effect on the ability of the evaluation to detect impacts.17 A more detailed discussion of the challenges encountered as a result of the site selection can be found at the end of this section (Section 3.4) and in Section 6.

The following locations were approved by the participating provinces for the evaluation:

3.2.2. Quasi-Unified Family Court Locations

As noted in Section 2, a continuum of court models was identified and was used to help identify court locations that would be appropriate for the UFC and comparison groups. From the continuum, "quasi-UFC" locations were also identified as possible alternatives to the UFC model. Like UFCs, these sites offer non-court dispute resolution alternatives and a range of FJS, both of which can be coordinated with the court and/or the court registry, although clients do not have to be involved in the court process to access the services. The key difference between these locations and the UFC model is that the quasi-UFC model maintains the federal and provincial jurisdictional division of family law.

One component of the evaluation was to establish the extent to which the UFC model achieved its objectives relative to outcomes found for alternative models identified in Canada. Therefore, quasi-UFC locations were examined as an alternative model to the UFC.

The potential court locations were identified by the researchers and the respective provincial representatives agreed to participate in the study by providing access to information on court usage. The following locations were identified in consultation with the provinces for the purpose of comparison:

3.2.3. Case File Sample Frame

Parameters used to define the case file sample included the type of case (i.e., divorce or non-divorce) and the reference period (i.e., new files opened within a certain period of time). Matters of family law are often dealing with multiple issues, and parties can return to the family justice system multiple times to address new issues that emerge or old issues that re-emerge. Theoretically, family law cases remain open indefinitely. Therefore, the length of time since the case file was opened was an important consideration, to allow sufficient time to have passed to resolve the matters at issue in the original application. In addition, some of the impact measures included in the evaluation framework are time sensitive (e.g., repeat applications, compliance). The reference period, therefore, had to be of sufficient duration to allow enough time for applications to have a conclusion and permit the estimation of the time-sensitive indicators.

Given the various time considerations, the sample consisted of new cases opened during the 12-month period from January 1, 2004 to December 31, 2004. Data collection took place from October 2007 to January 2008. Therefore, the amount of time that could have passed from the time a case file was opened to the time of review was from 36 to 45 months.

Cases included in the sample were those covered under the Divorce Act and the Maintenance and Custody Act in Nova Scotia, the Family Law Act and Children's Law Reform Act in Ontario, the Children's Law Act in Saskatchewan, as well as the Family Relations Act and the Family Maintenance Enforcement Act in British Columbia. For the protection of privacy, child protection cases were excluded from the sample.

Cases were selected based on stratified random sampling to ensure statistical representativeness of the sample and account for differences in size between the selected sites. Table 3-1 presents a summary of the number of court files and the individual filing activities reviewed at each site. A filing activity is defined as the initial filing activity (i.e., the initiating document) or an application, an application to vary or a notice of motion that initiates a "new" series of activities to address new or recurrent issues after previous issues had been dealt with. It should be noted that the final sample sizes were above those required as indicated in a power analysis completed for the feasibility study.18 Therefore, significant effects or differences between the UFCs and traditional courts could have been detected given the sample sizes.

Table 3-1: Case File Review Summary

UFC Locations
Court Location Files Reviewed Filing Activities Tracked
Hamilton, ON 109 156
Oshawa, ON 134 177
Halifax, NS 109 135
Saskatoon, SK 100 156
Total UFC Locations 452 624

Traditional Non-UFC Court Locations
Court Location Files Reviewed Filing Activities Tracked
Sarnia, ON (provincial) 36 59
Sarnia, ON (superior) 44 51
Sudbury, ON (provincial) 103 169
Sudbury, ON (superior) 92 99
Pictou, NS (provincial) 25 38
Pictou, NS (superior) 10 12
Truro, NS (provincial) 17 35
Truro, NS (superior) 9 15
Total Traditional Non-UFC Court Locations 336 479

Quasi-UFC Locations
Court Location Files Reviewed Filing Activities Tracked
Vancouver (provincial) 100 143
Vancouver (superior) 122 141
Toronto -University 193 197
Toronto -Sheppard 134 246
Toronto - Jarvis 98 143
Total Quasi-UFC Locations 647 871
Total Sample 1,435 1,973

3.2.4. Case Characteristics of the Unified Family Court and Comparison Samples

The information captured in case files of parties who had accessed the family justice system at each of the locations selected for the evaluation represented a key source of data. The following provides an overview of the characteristics of those cases included in the analysis for the current evaluation to provide information about the general case complexity. Data are presented in the tables below for the UFC, traditional non-UFC and quasi-UFC samples.

As summarized in Table 3-2, cases reviewed for the evaluation most frequently dealt with issues related to child support and custody, with slight differences occurring between the UFC and traditional non-UFC samples in terms of proportion of cases with these issues. The quasi-UFC sample had a larger proportion of applications dealing with child support issues and a similar proportion of cases dealing with custody. The proportion of cases that involved access issues in the UFC and traditional samples was similar, while there was a lower proportion in the quasi-UFC locations. The UFC sample (39.9%) had more divorce cases than the traditional sample (26.9%) and the quasi-UFC sample (34.7%). Similarly, the UFC locations were more likely than either traditional non-UFC or quasi-UFC locations to have cases with issues dealing with other issues that are only under the jurisdiction of the superior court (i.e., property). Some family justice cases still required that provincial child protection services representatives be involved although the case was no longer considered a child protection case, per se. Only a small percentage of cases at the three types of courts involved children protection services representatives19 while quasi-UFC locations were more likely to deal with matters that warranted an application for a restraining order or non-removal of the child/children. These latter two types of issues are typically dealt with in court, by a judge.

Table 3-2: Issues by Type of Court (% filings)
Issues UFCs (n=624) Traditional Non- UFC* Provincial (n=301) Traditional Non- UFC Superior (n=178) Traditional Non- UFC Average (n=479) Quasi- UFC Provincial (n=533) Quasi- UFC Superior (n=337) Quasi- UFC Average (n=870)
Child Support 51.0% 59.5% 30.3% 48.6% 59.1% 17.5% 57.0%
Custody 47.4% 67.8% 28.1% 53.0% 63.8% 21.1% 47.2%
Divorce 39.9% N/a 72.5% 26.9% Na 89.6% 34.7%
Access 34.8% 44.9% 22.5% 36.5% 33.2% 15.1% 26.2%
Property 24.8% 2.3% 37.6% 15.4% 0.4% 9.8% 4.0%
Spousal support 20.4% 7.3% 27.5% 14.8% 15.6% 8.6% 12.9%
Restraining Order/Non-removal protection 10.1% 11.3% 6.2% 9.4% 27.2% 5.6% 18.9%
Involvement of Children's Aid Services 2.6% 4.0% 0.6% 2.7% 3.6% 0.0% 2.2%

Source: Case file review

With respect to overall case characteristics, cases in UFC locations had a higher average number of issues per application (2.35) than the average number of issues per application in traditional non-UFC locations (2.14) and quasi-UFC locations (2.00).20

Not all parties in family law matters have a lawyer representing them. The analysis of the legal representation in the cases reviewed reveals that in close to half (44.1%) of the cases, neither party had legal representation and in 30.6%, only one party had representation (see Chart 3-1). Parties in UFCs were more likely to have legal representation than in comparison locations.21

Chart 3-1: Legal Representation of Parties

Chart 3-1: Legal Representation of Parties

[ Description ]

Source: Case file review

The differences in the characteristics of the sample required that these differences be controlled for in the analysis of outcomes, as type and number of issue per application could influence outcomes. Furthermore, it was found that legal representation also has an effect on resolution. Variables that were found to have an influence on case outcomes in terms of time to resolution and less adversarial resolution were controlled for through regression techniques.

3.3. Scope of Work

The Summative Evaluation of the UFC relies on multiple lines of evidence, including qualitative and quantitative data. The scope of work completed for the evaluation is briefly described in the remainder of this section.22

3.3.1. Document and Literature Review

Documentation was reviewed to obtain contextual and background information on the UFC and its impact. In addition to documents detailing the UFC itself, such as the UFC RMAF, documentation pertaining to each of the locations selected for the evaluation was reviewed. Research reports and literature were reviewed to identify issues facing separating/divorcing families, recent trends in Canadian families, and approaches to family justice in other jurisdictions.

3.3.2. Inventory Development

An inventory of available services and site characteristics was developed for all family court locations in Canada using available secondary sources (e.g., the Department of Justice inventory Website, provincial/territorial Websites) as well as telephone interviews with provincial/territorial representatives and individuals from each site. This inventory was then utilized in the selection of the court sites that were originally proposed for the evaluation.

3.3.3. Court Case File Review

The court file review was completed using a tracking sheet designed to collect information on various outcomes, to identify issue resolution patterns and to estimate net impacts. A total of 1435 file reviews was conducted manually, on-site at participating court location. Each filing activity that initiated a new set of events was tracked separately (n=1,973) including motions, applications to vary and new applications that initiated a new set of issues that were not part of the original application and response.

3.3.4. Key Informant Interviews

Key informant interviews were used to identify perspectives and opinions regarding the unified family court concept, operations/procedures in place, and challenges experienced at specific locations that could influence the results of the case file review and evaluation.

The breakdown of the key informant interviews completed for the evaluation is summarized in the table below.

Table 3-3: Key Informant Interviews
Key Informant Group # Interviews
Federal Officials 2
Provincial/Territorial Officials 9
Members of the Judiciary 30*
Family Justice Representatives:
  • Court Registry
  • FJS
  • Other
 12
Total 53

* Includes two instances where more than one judge participated in the interview at the same time.

In addition to the interviews noted above, a number of unstructured/informal discussions were completed with provincial/territorial officials throughout the evaluation, as the study was completely reliant on the participation and cooperation of the provinces/territories.

3.4. Challenges and Limitations to the Evaluation

Several challenges were encountered in this evaluation and discussed below.

3.4.1. Site selection and variability in Unified Family Court implementation

The final sites selected for study in the evaluation differed from those that had been proposed in the original evaluation design; not all of them met the site selection criteria that had been set out in the design report. This has had an impact on the fidelity of the evaluation design, on the ability to test the research hypothesis, and has introduced unanticipated intervening variables into the design that could affect outcome measures for the sites. Therefore, when analyzing and presenting outcomes at an aggregate level for the UFC sample to assess the success of the UFC model overall, site specific issues may have attenuated any potential net impacts and the results for the UFC model (on an aggregate level) may not look substantially different from the results for the traditional model. To better understand the potential effects of the intervening variables on the overall results, site descriptions of all UFC sites included in the analysis were prepared and the results for each site were compared to the results; comparisons between the specific UFC sites were not made, as was agreed with the participating provinces.

Overall, the variability with respect to the implementation of the UFC model at individual sites has resulted in substantial challenges with respect to estimating the net impact and success of the UFC model in meeting its original objectives.

3.4.2. Difficulty isolating impacts

As outlined in Section 2, the UFC now operates in a context of various existing family justice initiatives supported through federal and provincial/territorial initiatives in all court locations. In the context of these other strategies, the development and support of FJS, therefore, are only conceptually linked to the UFC but are not unique to UFC locations. While UFCs are expected to provide a minimum level of service that might not all be available or as comprehensive at other courts, access to some services is provided across all locations regardless of the court model in place. Therefore, it was not possible to attribute outcomes that are associated with access to FJS and dispute resolution mechanisms as identified in the logic model to the court model.

3.4.3. Inability to cross-reference court file information with FJS files

Ideally, the full range of dispute resolution mechanisms and supports accessed by the parties could be tracked through the court case file. Court-related alternative dispute mechanisms (i.e., case conferencing) were tracked in the court files at all locations. However, as determined in the feasibility study and confirmed in the evaluation, records of FJS activity are maintained by family justice service providers and are completely separate from the court registry (i.e., there is no cross-reference between file numbers), making it very difficult to reliably cross-reference between court-based and non-court activities on a case-by-case basis. In only one location included in this evaluation were intake services and referrals to other FJS made in the case files housed in the court registry. At this court, intake and referrals were tracked in a case file regardless of whether the parties ever went to court or saw a judge. The lack of connection between the court and non-court FJS records is itself a finding of the current evaluation that will be discussed in more detail in Section 4 of the report.

Apart from one location, the extent to which the utilization of these services was tracked in the reviewed registry files was very limited and reflected registry practices rather than actual utilization. As a result, evaluation questions pertaining to the utilization of FJS and out-of-court dispute resolution mechanisms could not be addressed through the court file review.

3.4.4. Time measures and time tracking

Other measurement challenges encountered in the evaluation affected the ability to address evaluation questions on time to resolution. In most locations, case tracking or management systems are generally not designed to track time in court, meaning that the length of time parties spend in court at hearings, trials, or other court activities is not recorded. Therefore, it was not possible to determine whether the UFC model has had an impact on the amount of time Canadian families spend in court when facing separation and divorce. As such, net impacts of the UFC model on the amount of time spent in court could not be determined.