Appendix C: Evaluation methodology

• Previous Page
• Table of Contents

The evaluation of the DTCFP relied on multiple lines of evidence that were triangulated and form the evidence base for the study’s conclusions and recommendations. The data collection occurred between December 2020 and May 2021. DTCIS data was extracted up until June 17, 2021, due to delays in data entry. The lines of evidence included a review of literature, documents and data, a review of DTCIS data, an online survey of stakeholders and key informant interviews.

Document, Literature and Data Review

Document review: A document review was conducted to provide background information for the DTCFP, including context, design and implementation, and was also used as evidence to address the evaluation issues and questions. The types of documents that were reviewed included:

• corporate and program foundational documents (e.g., legislation, mandate letters);
• program implementation documents (e.g., policies and guidelines, PT funding agreements, FPT working group materials);
• performance information (e.g., previous program evaluation and associated action plans and follow-up, DTC evaluations);
• other special studies and reports; and,
• financial information on planned and actual funding by fiscal year.

Literature review: A focused review of literature was undertaken, including: (i) peer-reviewed articles; and (ii) published and unpublished reports and other documents produced mainly by and for government, agencies and non-governmental organizations (grey literature). The review particularly focused on systematic reviews and empirical studies of the effectiveness and cost effectiveness of DTCs.

Data review: Criminal justice statistics from the Canadian Centre for Justice Statistics (CCJS)59 were used to explore trends in drug offences.

DTCIS

Descriptive analyses of DTCIS data were conducted. The DTCIS is used by most DTCs to record information related to DTC implementation. Data was extracted from the DTCIS into Excel format and included all DTC participants who started the program at some point between April 1, 2015 and June 17, 2021. Once duplicate records and invalid entries were removed, in total, 1,053 records were included in the analysis. The records were imported into R for data processing and analysis. The DTCIS includes data on the socio-demographic characteristics of participants, program status indicators (e.g., completed, non-completed) and results of urine testing.

Survey of DTC Stakeholders

Each DTC was asked to provide a listing of stakeholders for their DTC, including: DTC Directors, judges, Crown and defence counsel, other DTC governance or advisory committee members and external service providers. All DTCs, except those from Alberta and Ontario, provided a listing of names. Alberta and Ontario elected to distribute the link to the survey independently. In total, 162 names were received.

The survey was conducted online, programmed using Réseau Circum’s CallWeb software. The instrument included a mix of closed and open-ended questions and was about 20-30 minutes in duration. To encourage a higher response rate, email reminders were sent to non-responders, including two follow-up emails. The survey data collection was launched February 22, 2021 and closed March 23, 2021. In total, 139 stakeholders completed the survey. The response rate for the DTCs that provided lists was 49%, with a range in response rates across DTCs. There were somewhat lower response rates from case study DTCs (Regina and Whitehorse). As it is not known how many respondents were provided the open link in Alberta and Ontario, a response rate cannot be calculated.

Note the number of responses for the open links were as follows: Ottawa (n-5), Toronto (n-40), Calgary (n-7), Edmonton (n-5).

Most respondents were members of the dedicated DTC team (56%), while one in five was an external service provider.

n = 139; Source: Survey of DTC Stakeholders. Multiple responses possible

Key Informant Interviews

Key informant interviews were conducted to gather in-depth information from DTC representatives and federal officials. In total, 63 interviews were conducted, with the majority of these (n = 59) being conducted in the context of four DTC case studies. The four DTCs that volunteered to participate in the evaluation as case studies were Toronto, Regina, Edmonton and Whitehorse. Each of the DTCs were asked to recommend interviewees for their case study, and the final distribution of interviews by type is presented below.

Tailored key informant interview guides were developed for DTC representatives and for participants. Interviews were conducted with key informants via teleconference in the official language of choice and were approximately 45 to 60 minutes in duration. A copy of the key informant guide was provided in advance of the interview for DTC representatives to allow respondents an opportunity to review/reflect on the questions beforehand.

Interview notes were taken electronically during the interview and combined with documentary sources to provide a detailed description and illustration of the operation and outcomes of each of the case study DTCs. A template for the case study narratives was developed to ensure consistency in reporting across all the case studies. The narrative reports were used to inform evaluation questions related to performance.

In addition to the case study key informant interviews, four interviews were conducted with federal officials from Justice and PPSC.

---

• Previous Page
• Table of Contents