Family Violence Initiative Evaluation, Final Report

1. INTRODUCTION

1.1 The Family Violence Initiative

The Family Violence Initiative (FVI) is a long-term federal commitment to a complex, deeply rooted, and costly long-term societal problem with myriad social, justice, and health dimensions. It was first announced on June 5, 1988, with $40 million in time-limited funding, to provide 200 new short-term shelter units for crisis assistance for abused women and their children, and to establish a process for developing a long-term federal approach to family violence prevention.

Currently, there are 15 federal departments and agencies that participate in the federal FVI, of which eight, including the Department of Justice Canada (Justice), receive an initiative-specific funding allocation. The federal FVI is associated with a $7 million annual allocation. In recent years, the budget for the Justice FVI has been $1.13 million annually with a staff of 3.64 full-time equivalent employees (FTEs).

The mandate of the federal FVI is to enhance awareness about family violence, build the knowledge base, and strengthen the ability of the communities and the justice, housing and health systems to prevent and respond to family violence. The objectives are to:

The Justice FVI mandate is grounded in the departmental mandate to ensure that Canada is a “just and law-abiding society with an accessible, efficient and fair system of justice” and to “promote respect for rights and freedoms, the law and the Constitution”. Towards this end, Justice reviews, researches, develops and reforms criminal and civil justice laws and policy; provides project funding for community-based family violence projects; and provides public and professional legal education and information support on family violence issues. The Justice FVI is linked to the following departmental initiatives and activities:

1.2 Purpose of the Evaluation

Although the Department conducted periodic internal reviews of the FVI, the Justice FVI has never been formally evaluated. In 2010, in accordance with the TBS Policy on Evaluation, the Department initiated an evaluation of the Justice FVI. The results will be used by Justice for planning and reporting purposes. The evaluation was completed over a nine-month period. The scope of the evaluation encompasses projects funded from 2005-2010 and policy activities undertaken from 2000-2010.

The purpose of the evaluation is to examine the continued relevance of the Justice FVI, the program’s design and implementation, the accomplishments and achievements of activities implemented within the scope of the Initiative, and the demonstrated efficiency and economy.

1.3 Evaluation Questions

The evaluation of the Justice FVI addresses issues of design, implementation and performance monitoring, relevance, effectiveness, efficiency and economy. The evaluation questions are as follows:

Relevance

Justice FVI Design and Implementation

Effectiveness

Efficiency and Economy

1.4 Evaluation Methodology

Under the FVI, Justice undertakes a range of activities (legal policy development and law reform, strategic coordination and partnership development, investments, and information and educational supports) involving various stakeholder groups including other federal government departments, provincial and territorial departments, Public Legal Education and Information (PLEI) organizations, and other non-government organizations (NGOs). The level of risk associated with the program is considered low given the small budget involved, the long operating period, and the nature of its activities. Reflecting these characteristics, the evaluation methods incorporated multiple lines of evidence including literature review, key informant interviews, six case studies, and a document review of funded projects. The evaluation team adopted a common methodological approach to ensure the reliability validity and reliability of data, observations and conclusions. To that end, an evaluation matrix was developed to guide data collection and analysis.

Literature Review:
A literature review was undertaken of family violence in Canada and legislation related to family violence. The review focuses on topics such as FVI alignment with the government priorities, roles and responsibilities, and international obligations; national and international family violence trends; policies and legislation implemented in Canada to combat family violence; and changes in level and nature of public awareness of family violence in Canada.
Interviews:
Of the 33 key informant interviews, 12 were undertaken with Justice employees, 8 with representatives from other federal government partners, 6 with provincial/territorial justice officials, and 7 with other stakeholders (health and social services agencies, NGOs, and academic institutions). The questionnaires used for these interviews are provided in Appendix A.
Case Studies:
Six case studies covered the activities and projects supported through Justice FVI. The case studies were selected in consultation with Justice employees, considering the impacts reported during key informant interviews. The cases were selected in a manner to ensure balanced representation vis-à-vis types of outputs, location, size and duration, and target groups/beneficiaries. Each case study involved a review of relevant documentation, interviews with project leads and with partners, users, beneficiaries and participants. Seventeen case study interviews were completed. The guides used for these interviews are provided in Appendix A.
Document Review of Funded Projects:
The document review included an in-depth review of documentation related to 30 of the 55 projects funded through the Justice FVI and implemented between 2005 and 2010. A summary of the document review is provided in Appendix B.

1.5 Limitations, Challenges and Mitigation Strategies

The evaluation encountered challenges in aggregating and attributing impacts, reviewing longer-term impacts, assessing delivery costs, and obtaining input from stakeholders and other respondents. These study limitations were mitigated, as much as possible, through use of multiple lines of evidence and triangulation of data to demonstrate reliability and validity of the findings.

Aggregated impacts:
It is very difficult to quantify and aggregate program outcomes in a meaningful way, particularly those associated with legal policy development and law reform as well as strategic coordination and partnership development. This challenge has been addressed through the use of case studies and by encouraging key informants to provide specific examples of Justice FVI activities/projects when discussing outcomes.
Attribution:
Similarly, it is difficult to attribute and separate out the outcomes of the Initiative from other programs and initiatives undertaken by various levels of government. For example, the Justice FVI provides legal and policy advice, information and research which may influence decisions that have an impact on the capacity and responsiveness of the justice system; however, the Justice FVI is not the decision-maker and the outputs of the Justice FVI may be only one of a range of factors that influence the ultimate decisions. This challenge has been addressed by encouraging key informants to provide specific examples of Justice FVI activities and projects when discussing impacts. In addition, a document review and case studies of specific Justice FVI activities and projects further captured impacts attributable to the Initiative.
Long-term nature of expected impacts:
Justice FVI activities are designed to develop or promote tools, methodologies, strategies and resources to be used by policy-makers, organizations, and other target groups to build capacity in addressing family violence. Therefore, it is difficult to measure the long-term impacts of these activities on family violence. In addition, many of the Justice FVI activities included in the evaluation are still underway or only recently completed and the long-term impacts are not yet known. This challenge has been addressed by identifying key informants and case study interviewees who were in a position to comment on information, tools, resources or assistance that was generated; what was done with that information; what resulted from these actions; and to what extent those results are attributable to Justice FVI activities or projects.
Limited data available on program delivery costs:
The program provided information on program budgets and FTE staffing levels. However, given the range of representatives who may be involved in particular activities and the absence of activity-based costing data, it is not possible to determine all of the specific resources dedicated to particular FVI activities. In turn, this constraint has made it more difficult to assess program economy and efficiency. To respond to this limitation, qualitative questions on program efficiency were included in the key informant interviews.
Low awareness of the full range of program activities amongst stakeholders:
A notable challenge associated with the evaluation is that many activities associated with the Justice FVI (e.g., legal and policy advice, international contributions, and research) are not visible to most external stakeholders. As a result, when providing feedback on effectiveness, federal partners, provincial and territorial justice officials, and other stakeholders may tend to understate the impacts of the Justice FVI (e.g., by providing lower impact ratings). These external stakeholders have tended to provide higher ratings regarding the impacts of PLEI activities, which are the most visible of the activities and outputs generated by the Justice FVI. This challenge has been addressed by conducting extensive interviews with those most directly involved in the Justice FVI (the Justice employees) as well as conducting document review and case studies.
Potential for respondent bias:
Given the limited availability of secondary data, interviews with Justice employees were a key line of evidence. The evaluation findings, therefore, are based in part on the views of those who may have a vested interest in the program and be potentially biased in their responses regarding program outcomes. Several measures were taken to reduce the effect of respondent biases and validate interview results: (i) interviewers communicated the purpose of this evaluation, its design and methodology, and strict confidentiality of responses clearly to participants; (ii) interviews were conducted by telephone by skilled interviewers; (iii) the respondents were asked to provide a rationale for their assessments, including a description of specific activities which contributed to the reported outcomes; (iv) where selected for review, the impacts generated by particular activities were confirmed through case studies; (v) the answers were cross-checked for consistency and validation with those of other groups as well as with program and project documentation; and (vi) a variety of descriptive and inferential statistical analyses were conducted to further support comparative findings.

1.6 Structure of the Report

This report is divided into four chapters. Chapter 2 describes the design and implementation of the Justice FVI. Chapter 3 presents the findings of key informant interviews, case studies, literature review, and document review, while Chapter 4 discusses the conclusions arising from the evaluation. The appendices contain key informant and case study interview guides, as well as a summary of the document review.