National Anti-Drug Strategy Evaluation
3. Evaluation Methodology
3.1 Method of Study
In accordance with the TB Evaluation Policy, the evaluation addresses issues of continued relevance, effectiveness, and demonstrated efficiency and economy of the Strategy. The evaluation questions are presented in Appendix B. The evaluation was designed in three phases. The first phase, which involved development of an evaluation plan and methodological tools, was finalized in April 2011. The second phase consisted of the collection of qualitative and quantitative data from multiple lines of evidence and was completed in December 2011. The third phase consisted of data analysis and the development of this final evaluation report.
The evaluation design takes into consideration the Strategy’s characteristics, specifically, that it is a complex, horizontal initiative involving multiple departments, a large budget, broad objectives, with three action plans comprising many components at varying levels of implementation maturity. Some of these components are very large and offer programming at a national or provincial scale; others are more modest where the reach and impact would more likely be measured at a local or community level. Given all of these considerations, the evaluation analysis is conducted largely at the action plan level so that the collective impact of the components can be assessed.
Since the development of the Strategy, the partners have emphasized the need for collecting and analyzing ongoing performance information. The evaluation design incorporated a significant document and file review that made extensive use of performance data, evaluations, documents, files and other information compiled on the various components and action plans.
Multiple lines of evidence were used to complement the document and file review, including a literature review; interviews with key informants; a series of learning circles, case studies and an online survey focused on specific components within each action plan; a cost-efficiency analysis; and a series of focus groups. Taken together, the learning circles, case studies and online survey focused on eight of the ten components under the prevention and treatment action plans. In addition, a case study was conducted involving the inter-relationships among components of the Enforcement Action Plan.
3.1.1 Document and File Review
An extensive review of documents was conducted to collect information on all core evaluation issues pertaining to relevance and performance of the Strategy. This methodology proved to be an efficient way to access comprehensive information on the activities, outputs, challenges and achievements of all 20 components of the Strategy that have been carried out so far. Some of the key materials reviewed included the Strategy’s foundational policy and program documents (e.g. Reports on Plans and Priorities, Departmental Performance Reports for the participating departments, and references to the Strategy in Speeches from the Throne), terms of reference and meeting minutes of committees and working groups, the 2010 National Anti-Drug Strategy Implementation Evaluation, evaluation reports of some components, and surveys and statistics produced by the various components. In addition, an extensive review was conducted of the detailed annual performance information (from 2007/08 to 2010/11) prepared by each component on their activities and outcomes.
3.1.2 Literature Review
The literature review focused on illicit drug use among youth and other segments of the population as well as illicit drug production, trafficking, importation and exportation, and illicit drug treatment services in Canada. In addition to peer-reviewed publications, the literature included reports prepared by Justice Canada (e.g. Costs of Crime in Canada) and other federal government departments (e.g. the Criminal Intelligence Service of Canada [CISC] Report on Organized Crime, and the RCMP Report on the Illicit Drug Situation in Canada); non-government organization publications (e.g. the Canadian Association of Chiefs of Police Drug Abuse Committee and the CCSA reports; workshop proceedings); and national surveys such as the Uniform Crime Reporting Survey (Canadian Centre for Justice Statistics), the Canadian Alcohol and Drug Use Monitoring Survey (HC), the Youth Smoking Survey (HC), and the Canadian Addiction Survey (HC). International reports were also reviewed, including the Hemispheric Drug Strategy by OAS/CICAD, the International Narcotics Control Strategy Report (U.S. Department of State), and the World Drug Report 2010 (United Nations).
Literature was also reviewed on other horizontal initiatives in Canada (e.g. the Federal Tobacco Control Strategy, the Federal Initiative to Address HIV/AIDS in Canada, and the Youth Employment Strategy) as well as drug policies and strategies in other countries. The information collected through these documents allowed us to compare the Strategy with other similar strategies at national and international levels. Appendix C provides a list of types of literature reviewed.
3.1.3 Interviews
All 12 departments involved in the Strategy were asked to identify candidates for interviews. The candidates proposed by the SER were invited to participate in an interview. In addition, snowball sampling Footnote 18 was used to recruit additional Strategy stakeholders for interviews. In total, 82 interviews were completed with three distinct groups of Strategy partners and stakeholders.
Nine different interview guides were developed for various groups of interviewees who had involvement in one, or more than one, of the three action plans as well as for those who were involved in the Strategy as a whole. Footnote 19 A description of each group of interviewees is as follows:
- 50 departmental representatives participated in the evaluation interviews. Eighteen interviewees were involved in multiple action plans, five were involved in the Prevention Action Plan only, nine were involved in the Treatment Action Plan only, and 18 were partners of the Enforcement Action Plan. Representatives were from all partner departments.
- 23 direct stakeholders including funding recipients (14), partners involved in program delivery (7), a project evaluator, and program consultants of the Prevention, Treatment, or Enforcement Action Plans. Strategy stakeholders had been involved with the Strategy for an average of four years.
- 9 external stakeholders including key individuals involved in issues related to the Strategy at the provincial, territorial and municipal levels as well as key academics and experts. External stakeholders have been involved in their respective area of activities for an average of 22 years, with a minimum of seven years, and maximum of 36 years.
3.1.4 Case Studies
Five case studies were conducted to examine the extent to which the Strategy has achieved its intended outcomes and to deepen our understanding of the themes, performance and impacts of the action plans. Case studies focused on describing Strategy-funded projects or operational scenarios.
Four case studies were completed of selected projects funded under various components of the prevention and treatment action plans. The methodology for these case studies included a review of documents and outputs, interviews with project representatives, sponsors, partners, and other stakeholders, as well as site visits, when possible. In total, 13 representatives were interviewed including three project leads, four project staff and six project partners. The projects were selected based on the results of the document and literature reviews, in consultation with the managers of the selected components and the SER. The selection of cases considered various criteria including location, target group, level of completion/progress, level of reporting, and availability of outcome information.
Component | Project (Program) | Location | Project Start Date | Target Population |
---|---|---|---|---|
Prevention | School-Aged Children and Youth Substance Use Prevention Initiative (SACY)-expansion (DSCIF) | Vancouver, B.C. | 2008/09 | Youth in schools |
Prevention | Velocity (NCPC) | St John's, N.L. | 2009/10 | At-risk youth in community |
Treatment | In Roads (DTFP) | Edmonton, Calgary and Red Deer, AB | 2009/10 | At-risk youth and non-profit organizations that are in contact with at-risk youth |
Treatment | Native Horizons Treatment Centre (NNADAP Modernization) | Hagersville, ON | 2010/11 | Treatment services for Aboriginals First Nations and Inuit |
In addition, a case study of the Enforcement Action Plan was undertaken to demonstrate the horizontal collaborative approach in enforcement activities. The study described how information flows among partners and how the work of a group of Strategy partners contributes to the work of other partners. The selection of the RCMP MGO and Clandestine Lab teams for case study was made considering the results of the document review, issues frequently raised during interviews, the complexity of the component, and the high level of partnership and coordinated responses that are required to disrupt MGOs in a safe manner. The mapping resulting from this scenario was comprehensive and covered the activities of all enforcement partners as they progressed from intelligence-gathering to charges and disruption of illicit drug operations.
3.1.5 Learning Circles
Five learning circles
Footnote 20 were conducted with stakeholders to provide deeper insight into the Strategy’s themes, performance and impacts “on the ground.”
The sessions brought together groups of stakeholders associated with the Prevention or Treatment Action Plans to discuss what has been achieved, their challenges, and what has been learned in relation to reaching outcomes. With the exception of one session, the learning circles were incorporated into planned meetings, workshops or training sessions that occurred during the evaluation period and were organized by the Strategy partners. The SER assisted in identifying such opportunities and organizing the activities. Forty-four stakeholders participated in the circles.
Component | Program | Location | Participants | Number of Participants |
---|---|---|---|---|
Prevention | DSCIF | Vancouver, B.C. | Representatives from projects funded by DSCIF | 17 participants (in two concurrent sessions) |
Prevention | DOCAS-Aboriginal Shield Program | Saskatoon, SK | Representatives of various organizations who were trained to deliver the ASP (e.g. RCMP, police, First Nations representatives, and youth workers) | 9 participants |
Treatment | YJADS | Ottawa, ON | Researchers in the area of youth substance abuse, managers and staff of Youth Justice-funded treatment projects, and project evaluators | 7 participants |
Treatment | NYIDP | Grand Prairie, AB | Representatives from RCMP and organizations to which young offenders were referred for assessment and treatment | 11 participants |
3.1.6 Online Survey
An online survey was conducted to obtain information on a component of the Treatment Action Plan which was not otherwise the focus of a case study or a learning circle. The survey was sent to a list of 23 proponents of projects funded under the CIHR Research on Drug Treatment Models, of whom eight proponents completed the survey. The information collected complemented the data obtained through interviews of stakeholders.
3.1.7 Cost-efficiency Analysis
Reflecting the recommendations of the cost-effectiveness feasibility study prepared for the Strategy in 2010, a template was developed and distributed to representatives associated with funding components of the prevention and treatment action plans. This template was designed to collect data on component costs, inputs and outputs. Based on the data, comparative tables were developed for three components of the Prevention Action Plan and four components of the Treatment Action Plan. As the Strategy’s investments in enforcement activities represent incremental additions to the overall investment made by the partners in related activities, the Enforcement Action Plan components representatives did not complete the template, but the components were included in the overall cost-efficiency analysis.
3.1.8 Focus Groups
Following a presentation of the preliminary findings to the SER, three focus groups were conducted to explore issues raised in the evaluation findings and to validate certain findings. Each group focused on a different topic including collaboration and cooperation across strategy partners, communication and information sharing, and effectiveness and efficiency. In total, 23 representatives from the Strategy departments participated in the focus groups.
3.2 Limitations, Challenges and Mitigation Strategies
The evaluation encountered some challenges and limitations that are outlined below:
- Ability to aggregate impacts: Given the wide range of activities, intended outcomes and performance indicators associated with the Strategy, it is difficult to quantify and aggregate program outcomes in a meaningful yet concise way. A variety of qualitative research methods have been used to address this issue.
- Reliance on input from departmental representatives and direct stakeholders. The ability to obtain direct input from the key target of the Strategy (e.g. at-risk youth, offenders, and Aboriginal people) is limited by the size of the evaluation as well as by privacy rights. This issue has been addressed, in large part, through the extensive docu ment review, particularly evaluation and survey reports, as well as case studies of individual programs that directly worked with the Strategy’s target groups. Furthermore, given the complexity and breadth of the Strategy, awareness of the full range of Strategy activities is low among the external stakeholders who were interviewed (i.e., key individuals involved in illicit drug prevention, treatment, and enforcement at the provincial, territorial and municipal levels, as well as key academics and experts). Most of the external stakeholders that were interviewed, therefore, were not in a position to provide specific input on the overall Strategy or even individual action plans. This emphasis on departmental representatives and direct stakeholders could have led to potential bias in responses; however, this was mitigated, as much as possible, through the use of multiple lines of evidence as well as validating findings through other primary and secondary research.
- Attribution: It is difficult to attribute particular outcomes to the Strategy relative to other resources, programs and trends affecting the target groups. In some cases, the funding under the Strategy was used to expand existing capacity rather than to establish new initiatives or services; as such, it is difficult to attribute particular impacts to the new resources provided under the Strategy versus previously existing resources. This challenge has been addressed by encouraging key informants to provide specific examples of activities or projects when discussing impacts. In addition, the document review and case studies of specific activities or projects further captured impacts attributable to the Strategy.
- Limited data available on program delivery costs: Individual components provided information on program budgets and expenditures. However, given the range of representatives who may be involved in particular activities and the absence of activity-based costing data, it is often not possible to determine the specific resources dedicated to the Strategy. In turn, this has made it more difficult to assess program economy and efficiency. To respond to this limitation, qualitative questions on program efficiency were included in the key informant interviews.
- Inconsistent performance data: In some instances, the annual performance reports contained inconsistent performance indicators across the Strategy components. This made it difficult to aggregate qualitative data and to present the impacts that have occurred over the four-year evaluation period. To mitigate this challenge, references to single-year accomplishments were included, with a focus on more complete activities as well as balancing those figures with a qualitative assessment of trends and major accomplishments over the four years.
Overall, the study limitations were mitigated, as much as possible, through the use of multiple lines of evidence and triangulation of data to demonstrate reliability and validity of the findings.
- Date modified: