Federal Victims Strategy Evaluation, Final Report
3. EVALUATION METHODOLOGY
The methodology used to evaluate the FVS included a document review, review of six sub-studies, interviews with key stakeholders, a file review, and case studies of projects funded under the Victims Fund. This section provides further details on each research method.
3.1 Data Collection Methods
3.1.1 Document Review
A wide variety of relevant documents were identified and reviewed. These documents can be grouped under the following categories:
- Performance information documentation such as Departmental Performance Reports, Departmental Reports on Plans and Priorities, the Results-Based Management and Accountability Framework (RMAF) for the Federal Victims Strategy, and previous mid-term and summative evaluation reports;
- Strategic documentation such as Speeches from the Throne and Canada's Economic Action Plan, briefing notes and memoranda pertaining to the PCVI's key initiatives and activities under the Strategy;
- Financial information/records such as Federal Budgets, Grants and Contributions Information Management System (GCIMS) reports and IFMS reports;
- FPT Working Group on Victims of Crime meeting minutes; and
- Research literature on topics such as available services as well as gaps in services provided to victims of crime, different models in working with victims of crime, and research and statistics related to victims of crime. This included a review of PCVI research publications and PLEI website materials. A broad-based review of similar programs for victims offered by the United States government (i.e., the Office for Victims of Crime) was also undertaken.
3.1.2 Sub-studies
A number of evaluation activities and sub-studies were undertaken by the Department of Justice Evaluation Division to help inform the evaluation. The results of these sub-studies contributed significantly to this evaluation. The following describes these studies:
- Case Studies Report: Use of the Victims Fund to Support Legislative Amendments to Facilitate the Testimony of Vulnerable Witnesses (2009): This study included a descriptive and exploratory analysis of a range of different projects that had been undertaken in seven jurisdictions to implement the legislative amendments aimed at assisting vulnerable witnesses. The study focused on equipment purchases to enhance the ability of the provinces and territories to meet the testimonial support provisions of the amendments. The methodology of the study was primarily qualitative, with most of the data collected during site visits.
- Survey of Victims that received funding to attend Parole Board of Canada (PBC) hearings (2006-2009): A survey was mailed to registered victims of federally supervised offenders who received financial assistance for travel and accommodation to attend a PBC hearing between 2006-07 and 2008-09. The purpose of the survey was to determine the extent to which the financial assistance component of the Victims Fund results in reduced financial hardship and improved the experience of victims of crime in the criminal justice system. Participation in the survey was voluntary and 463 victims completed the survey.
- Survey of Support Persons for Victims that received funding to attend PBC hearings (2007-2010): A feedback survey was sent to support persons who accompanied a victim at a PBC hearing between 2007 and 2010. The survey was completed by 58 individuals.
- National Victims of Crime Awareness Week Surveys (2007-2009): Voluntary surveys were distributed to NVCAW event participants and organizers in 2007, 2008, and 2009. The surveys assessed the effectiveness of the week in raising awareness of victim issues. In total, 87 event organizers (58%) and 1,900 participants responded to the surveys.
- Survey of Northern Conference (2009): Conference delegates were asked to complete a short survey to gauge the overall usefulness of the conference and workshops, opportunities for networking, increased knowledge and awareness of victims' issues in the North. In total, 162 participants (60%) responded to the survey.
- Crown Witness Coordinator Program Sub-study (2010): The CWC Program Sub-study was conducted to assess the Program's relevance, capacity, responsiveness, challenges, impacts and outcomes. The methodology included site visits and interviews with northern criminal justice professionals. A total of 45 interviews and focus groups were held with CWCs (10), Chiefs and Associate Chiefs (5), Federal Prosecutors (3), members of the judiciary (3), defence counsel (15), Crown prosecutors (6), a representative of territorial government victim services programs, a representative of the PPSC, the Northern Region, and a representative of the PCVI. The evaluation covered the time period from January 2003 (the date of the first CWC program assessment) to May 2009.
3.1.3 Interviews with 46 Key Informants
Interviews were conducted with PCVI staff, federal, provincial and territorial members of the Working Group on Victims of Crime, members of the Advisory Committee and federal stakeholders who are not members of the FPT Working Group on Victims of Crime. The distribution of the 46 key informants by group is shown in Table 5. The interviews were designed to address evaluation questions related to the relevance and performance (i.e. effectiveness, efficiency and economy) of the Strategy. Two interviews were conducted in person and the remaining by telephone.
| Respondent Type | Number Interviewed |
|---|---|
| PCVI Directors, Managers, Staff | 11 |
| Federal Members of FPT Working Group on Victims of Crime | 7 |
| Provincial Members of FPT Working Group on Victims of Crime | 13 |
| Territorial Members of FPT Working Group on Victims of Crime | 3 |
| Advisory Committee Members | 10 |
| Federal Stakeholders (not members of FPT Working Group on Victims) | 2 |
| Total | 46 |
3.1.4 File Reviews of 57 Victims Fund Projects
The file review of 57 Victims Fund funded projects encompassed 44 projects from the Projects and Activities Component and 13 projects from the Provincial/Territorial Component. The sample represented 62% of the projects funded under the Provincial/Territorial Component and 40% of the projects funded under the Projects and Activities Component.[9]
Funded projects from across the country were selected for this review. Projects were selected based on the availability of information on file regarding project outcomes and impacts. The sample included a cross section of projects by type and size (i.e. the amount of funding and/or duration). All four types of projects funded under the Provincial/Territorial Component (i.e. Implementation, Sentencing, Underserved, and Emergency Fund) were included in the review. For the Projects and Activities Component, the sample was drawn from PLEI, enhancements to victim services, training, and research and evaluation projects. Bill C-2, An Act to protect children and other vulnerable persons (2006) projects,[10] hereafter referred to as testimonial aid projects, and NVCAW projects were excluded from the review as these types of projects have been previously examined in the sub-studies conducted by the Department of Justice. Conference attendance funding was also excluded.
3.1.5 Case Studies of 11 Victims Fund Projects
Eleven projects were selected for the case studies based on the results of the file review and in consultation with the Department of Justice Evaluation Advisory Committee. The sample was selected from the reviewed projects which had adequate outcome data, project results, and products available for review. The location, budget, objectives and target population of the projects were also considered.
Ten case studies were conducted in English and one in French. The case studies included interviews with project representatives, sponsors and partners/stakeholders, when possible. A total of 18 case study interviews were conducted, including 11 with project sponsors, 5 with project staff and 2 with project partners.
In addition to project summary reports, which funded recipients are asked to complete and provide to the PCVI at the conclusion of their project, other project products such as publications, presentations and websites were reviewed as part of the case studies.
3.2 Evaluation Strengths and Limitations
Several methods and mechanisms were used to enrich the data collection process and increase confidence in the overall results. The strengths of the evaluation methodology included:
- the use of multiple lines of evidence through different methods of data collection (i.e. interviews, case studies, file review) in order to triangulate findings and increase the reliability of data;
- multiple stakeholders from six distinct groups (i.e. funding recipients in NGOs, P/T funding recipients, PCVI staff, members of the FPT Working Group on Victims of Crime, members of the Victims of Crime Advisory Committee, and external federal stakeholders who are not members of the Working Group) and a broad regional representation; and
- use of both quantitative and qualitative data in order to accurately address multidimensional concepts.
There were also some limitations associated with the implementation of the methodology. The limitations were mitigated as much as possible through the use of multiple lines of evidence and validation of findings through other primary and secondary sources of data. The limitations include:
- Due to their interrelated and complementary nature, it was not possible to attribute an accurate dollar figure to the PCVI's Criminal Law Reform and Policy Development, FPT Secretariat and Working Group, PLEI, Capacity Building in the North activities, making it difficult to assess the economy of each of these activities;
- Lack of information on certain project files for the file review (i.e. final project summary reports were not available for some projects). This limited the number of projects included in the file review;
- Potential respondent bias in the Victims Fund case study findings, which are primarily from the perspective of project funding recipients (based on their reports), and therefore could be positively biased; and
- The perspectives of victims are limited to a survey of victims who received financial assistance to attend a PBC hearing, which limits the scope and depth of the evaluation report. It would have been difficult to survey/interview victims for this evaluation while still respecting their privacy rights.
- Date modified: