Triennial Evaluation Plan 2007-2010
Appendix A: Department of Justice Program Evaluation Policy
Introduction
The Department of Justice Program Evaluation Policy is built upon the principles of the www.tbs-sct.gc.ca/pubs_pol/dcgpubs/TBM_161/ep-pe_e.html Government Evaluation Policy (Treasury Board Secretariat, April 1, 2001). This policy, in keeping with the new management framework for the Government of Canada, http://www.tbs-sct.gc.ca/res_can/siglist_e.html Results for Canadians, reflects the view that public service managers are expected to define anticipated results, continually focus attention towards results achievement, measure performance regularly and objectively, and learn and adjust to improve efficiency and effectiveness.
What is Program Evaluation and how is it used?
Program evaluation[1] employs a set of applied research instruments that provides a systematic, objective assessment of elements of a policy's or program's[2] performance. Program evaluation contributes to strategic/corporate decision-making, innovation and accountability practices at all levels. Its purpose is to provide managers and other stakeholders with timely, relevant, credible and objective information on the continued relevance of government and departmental policies and programs, the impacts they are producing and opportunities for using alternative and more cost-effective policy and programming instruments.
Program evaluation acts as a feedback loop within the policy development process. It serves as a test of the ultimate success of policies by determining whether they accomplished what they set out to and, if not, why not? Program evaluation provides support to policy makers and line managers on matters such as the identification of expected policy and program outcomes, the development of performance frameworks, the monitoring of program and policy implementation, accountability reporting and the establishment of client-oriented service standards.
Program evaluation also provides information mid-way through a program (while the program activities are forming or happening) by examining various processes including: the delivery of the program, the quality of its implementation and the assessment of the organizational context, and program inputs.
Program evaluation assists in promoting organizational learning within government, for example by communicating benchmarks for the use and management of policy instruments and program delivery mechanisms.
Finally, program evaluation as one element of the Departmental comptrollership function, is conducted in co-operation and co-ordination with other review processes, specifically audit and management-led reviews.
The Glossary of Terms at the conclusion of this document provides more detailed information on the components and concepts involved in the evaluation process.
Policy Objective
The objective of the Department of Justice Program Evaluation Policy is to ensure that the Department has credible, timely, strategically focussed, objective and evidence-based information on the performance of its policies and programs.
Policy Statement
It is Department of Justice policy that key departmental policies and programs are:
- designed such that they clearly define expected results and embody sound performance measurement, reporting and accountability provisions at their outset; and,
- evaluated strategically and cost-effectively in a rigorous and objective manner and that departmental managers use evaluation findings to improve and report on policies and programs.
Key departmental policies and programs are those that involve large expenditures or a high level of risk, those for which the government or the Department requires strategic information, or those in which the central agencies, Parliament or the public has expressed a particular interest.
Roles and Responsibilities
The Deputy Minister
The Deputy Minister[3] is responsible for:
- ensuring that departmental policies and programs are achieving their intended results;
- appointing and providing support to a senior manager responsible for conducting strategic and cost-effective program evaluations in accordance with government standards;
- approving the Triennial Program Evaluation Plan on an annual basis;
- approving the Departmental Program Evaluation Policy and any changes to it;
- ensuring that evaluation findings, conclusions and recommendations are used in strategic decision-making on policies and programs and in accountability reporting;
- ensuring that evaluation reports are made accessible to the public with minimal formality; and,
- participating in centrally-led evaluations as directed by Treasury Board and/or Cabinet.
Audit and Evaluation Committee
The Audit and Evaluation Committee meets periodically to assist the Deputy Minister in discharging his/her responsibilities with respect to audit and program evaluation. It should be noted that, periodically, the Chairperson, as a member of Executive Council, will inform the Executive Council of the activities of the Audit and Evaluation Committee.
In its role with respect to evaluation activities, the (AEC) is responsible for:
- providing advice and counsel to assist the Deputy Minister in discharging his or her responsibilities for program evaluation;
- assisting in ensuring that the roles and functions of program evaluation and the responsibilities of all personnel involved in the evaluation are communicated and understood in the Department;
- advising the Deputy Minister of the implications of issues raised by central agencies and other governmental organizations in relation to program evaluation;
- examining the Department's Program Evaluation Policy periodically and, recommending proposals for change;
- reviewing and recommending approval of the Triennial Evaluation Plan on an annual basis;
- reviewing and approving evaluation reports including recommendations and management responses and where appropriate bringing issues to the Deputy Minister's attention;
- receiving reports periodically prepared by the Evaluation Division concerning the status of actions taken by managers in response to evaluation reports and problems encountered by managers in implementing recommendations;
- following up with Direct Reports for action plans in cases where there are serious issues requiring follow-up; and,
- informing Executive Council periodically of its activities and submitting the Triennial Program Evaluation Plan to the Council for review and ratification.
Direct Reports and Policy and Program Managers
Direct Reports and Policy and Program Managers are responsible for:
- ensuring that the expected outcomes of new policies and programs are defined in terms amenable to subsequent evaluation (in consultation with Evaluation Division)[4];
- monitoring the performance of their programs and operations;
- demonstrating program performance and acting on performance information;
- ensuring evaluators (including contract staff) have access to all departmental information that evaluators consider essential for the proper conduct of an evaluation and to interview departmental employees to obtain the required information, to the extent permitted by legislation and government policy.
- preparing Results-based Management and Accountability Frameworks (RMAFs) that provide for appropriate measuring and reporting of results, as related to the purpose of providing resources through transfer payments and where requested to meet Treasury Board needs (Policy and Program Managers in consultation with Evaluation Division);
- approving Results-based Management and Accountability Frameworks that provide for appropriate measuring and reporting of results, as related to the purpose of providing resources through transfer payments and where requested to meet Treasury Board needs (Direct Reports);
- submitting a management response to the Evaluation Division within 45 working days of receiving the final evaluation report and recommendations. The management response must address each of the recommendations contained in the evaluation report;
- appearing before the AEC to present and discuss management responses to recommendations;
- preparing a communication's plan where necessary to address any concerns that may arise from the evaluation report; and,
- ensuring that recommendations as approved by the Audit and Evaluation Committee are implemented.
Evaluation Division Director and Staff
The Evaluation Division is responsible for:
- conducting objective evaluations and providing advice and recommendations to the Deputy Minister and senior management on the continued relevance, success and cost-effectiveness of key Department of Justice policies and programs (summative evaluations) and on the effectiveness of the management systems, processes and practices (implementation evaluations);
- recommending measures for improving the policies and programs of the Department and enhancing the accountability of managers for program performance;
- communicating relevant and useful evaluation findings, conclusions and recommendations to program managers, senior management, the Audit and Evaluation Committee and the Deputy Minister in a clear, balanced and timely manner;
- apprising the Audit & Evaluation Committee on a regular basis of the development and approval of Results-based Management and Accountability Frameworks;
- working with managers to help them enhance the design, delivery and performance measurement of departmental policies and programs by providing advice, support (and management of the process where requested) on the development of Results-based Management and Accountability Frameworks;
- ensuring consistency in the development of Results-based Management and Accountability Frameworks across programs;
- providing methodological support and training regarding the development of performance measurement instruments and practices;
- submitting draft annual and triennial evaluation plans to the Audit and Evaluation Committee on an annual basis;
- forwarding copies of all evaluation reports to Treasury Board Secretariat; and
- submitting reports periodically to the Audit and Evaluation Committee concerning the status of actions taken by managers in response to evaluation reports and problems encountered by managers in implementing recommendations;
- informing policy and program managers when approved reports will be posted on the Internet and allowing sufficient time for the preparation of a communications plan; and,
- posting of approved reports on the departmental Intranet and Internet sites.
Public Access
Once completed and approved, all reports are posted on the Department's Internet and Intranet sites in both official languages within 60 working days after AEC approval . The reports are also accessible by the public in accordance with the Treasury Board Review Policy and the Access to Information and Privacy Acts.
Other Considerations
When designing the evaluation approach and especially in the preparation of evaluation questions for the evaluation of any departmental program or policy, special consideration will be given as to the relevance and inclusion of questions that examine the differential impacts of programs and policies on employment equity groups, linguistic groups, gender and other relevant diversity groups.
References
Enquiries
Enquiries about this policy should be directed to:
Director, Evaluation Division
Office of Strategic Planning and Performance Management
Glossary of Terms
- Activities:
- An operation or work process internal to an organisation, intended to produce specific outputs (e.g. products or services). Activities are the primary link in the chain through which outcomes are achieved.
- Goal:
- A broad, high-level statement of a desired outcome, in general terms, to be achieved over an unspecified period of time. A goal should reflect an organization's “Mission”.
- Logic Model:
- A graphic representation of the program “theory” or “action”. It consists of a logical chain of if-then relationships; if x occurs, then y will occur that shows the linkage from the activities through the sequence of outcomes
- Mission:
- A statement identifying an organization's business, purpose and reason for existence – critical areas within which goals, objectives and standards should be set.
- Objective:
- A statement of specific results to be achieved over a specified period of time. Objectives are generally lower-level and shorter term than a goal.
- Outcome/Result:
- The effect of the outputs of a program on client or target groups. In other words, outcomes/results are the changes a program or policy hopes to achieve. Outcomes/Results focus on what the program or policy makes happen rather than what it does (i.e. the intended results of the project, not the process of achieving them). They may be described as: immediate, intermediate or final, direct or indirect, intended or unintended.
- Output:
- A unit of service provided, product provided, or people served by a program or policy; or a count of goods and services produced.
- Performance Measurement:
- Consists of tracking program performance against goals over time to provide an assessment of a program's performance, including measures of productivity, effectiveness, quality, and timeliness. Performance Measurement can help provide objective perspectives for defending or expanding a program, rather than allowing it to suffer from relatively arbitrary or habitual decisions. Ongoing monitoring systems, which emphasize indicators and analysis linked to improvement, can help track and improve results over time and can also prove to be a valuable source of information in the formal evaluation process.
- Program evaluation:
- Employs a set of applied research instruments to provide a systematic, objective assessment of elements of a program's performance. This information provides managers and other stakeholders with timely, relevant, credible and objective information on the continued relevance of government and departmental policies and programs, the impacts they are producing and opportunities for using alternative and more cost-effective policy and programming instruments. Depending on the timing of the evaluation, it can consist of:
- a formative, implementation or mid-term evaluation which provides information mid-way through a program by examining the delivery of the program, the quality of its implementation and the assessment of the organizational context, personnel procedures and inputs; or
- a summative or impact evaluation which determines the overall impact a program has had by examining the effects or outcomes of programs.
Summative Evaluations focus on three primary concerns:
- issues of relevance, or more aptly, whether or not program or policy instruments, continue to address strategic priorities and/or actual needs; i.e. the extent to which the objectives and mandate of the program or policy are still relevant and the extent to which the activities and outputs of a program or policy are consistent with the mandate and plausibly linked to the attainment of stated objectives and intended impacts;
- issues of success, including the degree to which program or policy instruments are meeting stated objectives (i.e. impacts), and without unwarranted, undesirable impacts, and
- issues of cost-effectiveness such as whether the most efficient means are used to achieve objectives relative to alternative approaches including whether another level of government could assume responsibility for the policy or program instrument.
- Program Evaluation Process:
- Consists of four stages: planning and design; data gathering and analysis; reporting; and follow-up.
[Description]
- Results-based Management and Accountability Framework (RMAF):
- A blueprint for managers to help them focus on measuring and reporting on outcomes throughout the lifecycle of a policy or program. RMAFs are a requirement of the http://www.tbs-sct.gc.ca/Pubs_pol/dcgpubs/TBM_142/ptp_e.html Treasury Board Policy on Transfer Payments and are commonly required by Treasury Board (TB) in the approval of new or renewed programs. RMAFs are also called for under the TB Evaluation Policy whenever they make sense for the purpose of measuring and reporting on results. RMAFs generally include:
- a clear statement of the roles and responsibilities of the main partners involved in delivering the policy or program;
- a clear articulation of the resources to be applied and the objectives, activities, outputs and key results/outcomes to be achieved, along with their linkages (see Glossary of Terms for a description of each of these terms);
- an outline of the performance measurement strategy, including costs and performance information (key indicators) that will be tracked;
- the schedule of major evaluation work expected to be done; and
- an outline of the reporting provisions as appropriate for funding recipients and those for the department, including
RMAFs are a useful management tool for significant policies or programs, regardless of whether they are produced in compliance with an “official” government requirement. However, when an RMAF is not specifically required by TB and where a manager nonetheless wishes to have a framework to assist in the evaluation of a program or policy, it is sometimes called an evaluation framework, assessment framework or evaluation workplan. Essentially, these terms are equivalent to an RMAF but have more flexibility in their components (because they are not required by TB).
- Date modified: