State of the Criminal Justice System Methodology Report

Development of a Performance Monitoring Framework

In the early stages of the development of the Framework, a number of research and consultation activities were undertaken to identify outcomes and indicators to include in a Canadian CJS Framework. These are as follows:

  1. As a first step, JUS reviewed the literature on the development, characteristics and elements of a successful performance monitoring framework, including the selection of outcomes and indicatorsFootnote 3, Footnote 4.
  2. Then, JUS conducted an environmental scan and review of national and international sources to identify relevant examples performance monitoring  frameworks and compile an inventory of potential expected CJS outcomes and indicators.
  3. JUS also conducted public opinion research and public consultations on the value of performance monitoring in the CJS through focus groups and surveys as part of its annual National Justice Survey (NJS) in 2016 and 2017.
  4. Finally, JUS engaged and consulted with key CJS partners, stakeholders and experts.

These various research and consultation activities, as well as key findings and results, are further described below.

Literature Review

The process of creating a performance monitoring framework began with a review of relevant literature in order to help identify and clearly define the concepts, elements, and methods required for performance measurement. The aim of the review was to develop a clear understanding of the goals of performance measurement in general, in order to guide the creation of a CJS-specific framework. Though not as abundant as the literature on performance measurement in general, literature on CJS-specific performance measurement was also reviewed to further guide framework development. The following section outlines the main findings of this review.

Performance measurement includes expected results or “outcomes”Footnote 5 which are measured by indicatorsFootnote 6 where data are collected regularly over time in order to indicate areas of success or improvement, as well as emerging and persistent challenges. Performance measurement can inform decision-making. It can also be used to encourage accountability and transparency. Ensuring that performance information and data are accessible to the public can also demystify the CJS. Research findings highlighted the following as necessary in the development of a successful performance measurement framework for the CJS:

The outcomes included in a performance measurement framework must reflect the values, goals, standards, structures and activities of the program, policy, strategy or in this case, the system for which performance is measured. As noted by Dandurand et al. (2015), the values and goals of a system such as the CJS are multidimensional and may reflect competing views; however, a good performance measurement framework should be able to integrate multiple perspectives.

Indicators can be quantitative or qualitative information collected over time and may be compared to benchmarks or targets to identify areas of success or challenge. A functional performance measurement framework is sustainable and manageable, and therefore includes a limited number of indicators. Indicators were therefore carefully selected. To guide the selection of indicators, JUS considered Dandurand et al. (2015) who proposed a number of factors, including ensuring that indicators were 1) actionable through policies, reforms and strategies; 2) dynamic and capable of capturing change over time; and 3) flexible and adaptable to changing circumstances within the system. Other considerations included the availability and quality of the data, ensuring that measures were understandable, politically neutral and considered the needs of all those involved in the system, including clients, partners and stakeholders.

The change in an indicator over time can be defined in a number of ways, based on the type of information collected and the performance outcome expected. Some indicators can be measured against predetermined performance targets. Targets are often used in situations in which issues or challenges have been identified, and present a goal for performance improvement. Other indicators may lend themselves to the use of benchmarking, where an existing standard is used to measure performance. This standard may be based on past performance of an indicator within a system, performance of a similar indicator within another system (e.g., another jurisdiction), or an existing minimum requirement (Dandurand et al., 2015). 

Although setting specific performance objectives may be ideal, many criminal justice indicators do not currently have agreed upon targets or benchmarks; the CJS always has room for improvement. For example, it would be unachievable and also difficult to interpret a rate of 0 for self-reported victimization or police-reported crime. In addition, due to the inherent differences between provincial/territorial justice systems (e.g., provincial legislations, social programs, socioeconomic amd geographical characteristics), it was inappropriate to set national targets/benchmarks for the indicators included in the performance measurement framework. Therefore, performance objective setting is limited to directional targets (e.g., increase, decrease, neutral) rather than precise numeric targets (e.g., % increase).

It is important to note that while there have been attempts to “measure” or “grade” the performance of the CJS in the past, the JUS framework is more specifically a monitoring tool rather than a measurement tool. Without the use of benchmarks or targets, the framework provides an overview of what the system is doing and how it is performing at various points in time. Therefore, the term “performance monitoring framework” is used from this point on rather than “performance measurement framework”, for accuracy.

Environmental Scan and National and International Review

There are many approaches to performance monitoring and reporting. An environmental scan of practices and initiatives developed by other jurisdictions/organizations within and outside of Canada identified a number of relevant examples that have been applied to the CJS and other social systems (e.g., health system). In particular, the development of the Framework was informed by the MacDonald-Laurier Institute’s “Report Card on the Criminal Justice System” (Perrin & Audas, 2016), the University of Waterloo’s Canadian Index of Wellbeing (Michalos et al, 2011), and the Scottish Government’s Online Justice Dashboard (Scottish Government, 2012).

JUS also compiled an inventory of expected CJS outcomes and related indicators from national and international sources, including in Canada, Australia, the European Union, Ireland, New Zealand, Northern Ireland, Scotland, Sweden, USA and the United Kingdom. In addition to frameworks specific to CJS outcomes, the review was expanded to CJS-related indicators (i.e., mental health), paying particular attention to indicators that were identified in the international CJS review. The resulting inventory provided insight into outcomes and indicators that were internationally and domestically comparable, and informed the selection of outcomes and indicators in the Framework.

Public Opinion Research and Public Consultation

In the development of CJS performance outcomes and indicators, JUS also sought input and feedback from Canadians through focus groups and surveys as part of its annual National Justice Survey (NJS) in 2016 and 2017 (Department of Justice Canada, 2017; Department of Justice Canada, 2018b).

The 2016 NJS explored Canadians’ views of the goals and objectives of the CJS.Footnote 7 Results showed that Canadians value a CJS that is timely, transparent and fair; that considers and addresses underlying social factors; that considers the circumstances of vulnerable and marginalized people; that focuses on crime prevention; and, that promotes confidence and trust in the system. In addition, 2016 NJS asked Canadians about their sources of information on the CJS and their ideas on how best to share information on the CJS. Results showed that participants noted that they prefer to obtain information about the CJS through the Justice Canada website compared to other sources such as community organizations or the media. Results suggested a strong need for reliable and easily accessible information on the CJS in Canada.

The 2017 NJS asked for Canadians’ views on specific aspects of the CJS including performance measurement.Footnote 8 The results showed that Canadians were most interested in seeing performance reporting on the following: maintaining public safety, restoring relationships, holding offenders accountable, helping victims meet their needs, and offender rehabilitation. Many Canadians noted interest in information about recidivism rates, the use and effectiveness/success of alternative approaches to the traditional use of courts and jail (e.g., with use of community-based programs, supports and rehabilitative programs), the use and effectiveness/success of rehabilitative or support programs in jail, and victim input and satisfaction with the process and engagement of the CJS. 

JUS also took into consideration the results of its extensive public consultation on transforming the CJS into a modern, efficient and responsive CJS, undertaken between November 2017 and January 2018 as part of the Government’s commitment to review the CJS. Canadians identified the following nine topics as important for transforming the CJS:

Consulting and Engaging CJS Partners, Stakeholders and Experts

JUS conducted two rounds of consultations with key CJS partners, stakeholders, and experts. JUS also contracted a subject matter expert (SME) with expertise in Canadian CJS performance measurement to provide expert guidance and review of the proposed approach, method, and Framework. These are further described in the following sections.

Consultation 1

JUS undertook a first round of consultation with key CJS partners, stakeholders, and experts, between November 2017 and January 2018. The following CJS partners, stakeholders and experts were consulted:

The purpose of the consultation was to obtain input toward defining the expected outcomes of the CJS. Based on the review of official documents from Canadian and international jurisdictions described in the previous section, a preliminary list of system level outcomes was identified and presented to the stakeholders and experts who provided written feedback. A series of questions were posed:

The list of outcomes for the CJS was revised based on the input from the consultation.

Consultation 2

The second consultation with CJS partners was launched in March 2018 and involved Federal-Provincial-Territorial (FPT) government representatives from the Liaison Officer Committee (LOC)Footnote 10 and other non-FPT participants. The consultation included a revised list of proposed outcomes and focused on the identification of indicators and data sources to measure the outcomes. Participants were presented an extensive but non-exaustive list of indicators. They were asked to identify which indicators the Framework should include and if their jurisdiction/organization collects related data. For the indicators they supported, they were required to identify the desired direction of change (directional target).Footnote 11 Finally, participants also had the opportunity to suggest other indicators for consideration.

The feedback from the second consultation was used to identify and assess the indicators based on logistical/practical considerations, quality of the indicator/data, the coverage of the indicator (e.g., national, selected provinces, etc.), the frequency of collection, and the overall use/relevance of the indicator in informing the outcome. Results from the feedback led to another iteration of the outcomes and indicators of the framework.

Subject Matter Experts

JUS contracted a subject matter expert (SME) with expertise in Canadian CJS performance measurement. This was to provide an expert assessment and review of the proposed approach, method, Framework, and Report.

JUS also contracted with six SMEs in the area of Indigenous justice, Indigenous legal traditions and/or the experience of Indigenous individuals interaction with the CJS. The purpose was to identify approaches to reporting performance data in a way that incorporates Indigenous perspectives and lived experiences or storytelling. The engagement with the SMEs was done in three phases. The first phase involved having the SMEs complete an engagement questionnaire similar to the second consultation activity noted in the previous section. The purpose was to obtain Indigenous perspectives on the proposed CJS outcomes, indicators and data sources for the development of the performance monitoring framework. The second phase involved a one-day in-person research workshop to discuss the results of the engagement questionnaire and a proposed approach to reporting data on Indigenous individuals’ interaction with the CJS. The third phase required the SMEs to submit their proposed approaches, while taking into consideration the workshop discussions. Input from SMEs was incorporated into the approach and resulting Framework and Dashboard, particularly the Indigenous Peoples theme (see page 18).