Evaluation Policies

Treasury Board of Canada Secretariat, Government Evaluation Policy - April 1, 2001

Department of Justice Program Evaluation Policy May 1^st, 2003

1. INTRODUCTION
2. POLICY OBJECTIVE
3. POLICY STATEMENT
4. ROLES AND RESPONSIBILITIES
5. PUBLIC ACCESS
6. OTHER CONSIDERATIONS
7. REFERENCES
8. ENQUIRIES
9. GLOSSARY OF TERMS

I. INTRODUCTION

The Department of Justice Program Evaluation Policy is built upon the principles of the Government Evaluation Policy (Treasury Board Secretariat, April 1, 2001).  This policy, in keeping with the new management framework for the Government of Canada, Results for Canadians, reflects the view that public service managers are expected to define anticipated results, continually focus attention towards results achievement, measure performance regularly and objectively, and learn and adjust to improve efficiency and effectiveness.

What is Program Evaluation and how is it used?

Program evaluation^[1] employs a set of applied research instruments that provides a systematic, objective assessment of elements of a policy’s or program’s^[2] performance.  Program evaluation contributes to strategic/corporate decision-making, innovation and accountability practices at all levels. Its purpose is to provide managers and other stakeholders with timely, relevant, credible and objective information on the continued relevance of government and departmental policies and programs, the impacts they are producing and opportunities for using alternative and more cost-effective policy and programming instruments.

Program evaluation acts as a feedback loop within the policy development process. It serves as a test of the ultimate success of policies by determining whether they accomplished what they set out to and, if not, why not?  Program evaluation provides support to policy makers and line managers on matters such as the identification of expected policy and program outcomes, the development of performance frameworks, the monitoring of program and policy implementation, accountability reporting and the establishment of client-oriented service standards.

Program evaluation also provides information mid-way through a program (while the program activities are forming or happening) by examining various processes including: the delivery of the program, the quality of its implementation and the assessment of the organizational context, and program inputs.

Program evaluation assists in promoting organizational learning within government, for example by communicating benchmarks for the use and management of policy instruments and program delivery mechanisms.

Finally, program evaluation as one element of the Departmental comptrollership function, is conducted in co-operation and co-ordination with other review processes, specifically audit and management-led reviews.

The Glossary of Terms at the conclusion of this document provides more detailed information on the components and concepts involved in the evaluation process.

II. POLICY OBJECTIVE

The objective of the Department of Justice Program Evaluation Policy is to ensure that the Department has credible, timely, strategically focussed, objective and evidence-based information on the performance of its policies and programs. 

III. POLICY STATEMENT

It is Department of Justice policy that key departmental policies and programs  are:

• designed such that they clearly define expected results and embody sound performance measurement, reporting and accountability provisions at their outset; and,
• evaluated strategically and cost-effectively in a rigorous and objective manner and that departmental managers use evaluation findings to improve and report on policies and programs.

Key departmental policies and programs are those that involve large expenditures or a high level of risk, those for which the government or the Department requires strategic information, or those in which the central agencies, Parliament or the public has expressed a particular interest.

IV. ROLES AND RESPONSIBILITIES

i) The Deputy Minister

The Deputy Minister^[3] is responsible for:

• ensuring that departmental policies and programs are achieving their intended results;
• appointing and providing support to a senior manager responsible for conducting strategic and cost-effective program evaluations in accordance with government standards;
• approving the Triennial Program Evaluation Plan on an annual basis;
• approving the Departmental Program Evaluation Policy and any changes to it;
• ensuring that evaluation findings, conclusions and recommendations are used in strategic decision-making on policies and programs and in accountability reporting;
• ensuring that evaluation reports are made accessible to the public with minimal formality; and,
• participating in centrally-led evaluations as directed by Treasury Board and/or Cabinet.

ii) Audit and Evaluation Committee

The Audit and Evaluation Committee meets periodically to assist the Deputy Minister in discharging his/her responsibilities with respect to audit and program evaluation. It should be noted that, periodically, the Chairperson, as a member of Executive Council, will inform the Executive Council of the activities of the Audit and Evaluation Committee.

In its role with respect to evaluation activities, the Audit and Evaluation Committee (AEC) is responsible for:

• providing advice and counsel to assist the Deputy Minister in discharging his or her responsibilities for program evaluation;
• assisting in ensuring that the roles and functions of program evaluation and the responsibilities of all personnel involved in the evaluation are communicated and understood in the Department;
• advising the Deputy Minister of the implications of issues raised by central agencies and other governmental organizations in relation to program evaluation;
• examining the Department's Program Evaluation Policy periodically and, recommending proposals for change;
• reviewing and recommending approval of the Triennial Evaluation Plan on an annual basis;
• reviewing and approving evaluation reports including recommendations and management responses and where appropriate bringing issues to the Deputy Minister’s attention;
• receiving reports periodically prepared by the Evaluation Division concerning the status of actions taken by managers in response to evaluation reports and problems encountered by managers in implementing recommendations;
• following up with Direct Reports for action plans in cases where there are serious issues requiring follow-up; and,
• informing Executive Council periodically of its activities and submitting the Triennial Program Evaluation Plan to the Council for review and ratification.

iii) Direct Reports and Policy and Program Managers

Direct Reports and Policy and Program Managers are responsible for:

• ensuring that the expected outcomes of new policies and programs are defined in terms amenable to subsequent evaluation (in consultation with Evaluation Division)^[4];
• monitoring the performance of their programs and operations;
• demonstrating program performance and acting on performance information;
• ensuring evaluators (including contract staff) have access to all departmental information that evaluators consider essential for the proper conduct of an evaluation and to interview departmental employees to obtain the required information, to the extent permitted by legislation and government policy.
• preparing Results-based Management and Accountability Frameworks (RMAFs) that provide for appropriate measuring and reporting of results, as related to the purpose of providing resources through transfer payments and where requested to meet Treasury Board needs (Policy and Program Managers in consultation with Evaluation Division);
• approving Results-based Management and Accountability Frameworks that provide for appropriate measuring and reporting of results, as related to the purpose of providing resources through transfer payments and where requested to meet Treasury Board needs (Direct Reports);
• submitting a management response to the Evaluation Division within 45 working days of receiving the final evaluation report and recommendations. The management response must address each of the recommendations contained in the evaluation report;
• appearing before the AEC to present and discuss management responses to recommendations;
• preparing a communication’s plan where necessary to address any concerns that may arise from the evaluation report; and, 
• ensuring that recommendations as approved by the Audit and Evaluation  Committee are implemented.

iv) Evaluation Division Director^[5]and Staff

The Evaluation Division is responsible for:

• conducting objective evaluations  and providing advice and recommendations to the Deputy Minister and senior management on the continued relevance, success and cost-effectiveness of key Department of Justice policies and programs (summative evaluations) and on the effectiveness of the management systems, processes and practices (implementation evaluations); 
• recommending measures for improving the policies and programs of the Department and enhancing the accountability of managers for program performance; 
• communicating relevant and useful evaluation findings, conclusions and recommendations to program managers, senior management, the Audit and Evaluation Committee and the Deputy Minister in a clear, balanced and timely manner; 
• apprising the Audit & Evaluation Committee on a regular basis of the development and approval of Results-based Management and Accountability Frameworks; 
• working with managers to help them enhance the design, delivery and performance measurement of departmental policies and programs by providing advice, support (and management of the process where requested) on the development of Results-based Management and Accountability Frameworks; 
• ensuring consistency in the development of  Results-based Management and Accountability Frameworks across programs; 
• providing methodological support and training regarding the development of performance measurement instruments and practices; 
• submitting draft annual and triennial evaluation plans to the Audit and Evaluation Committee on an annual basis; 
• forwarding copies of all evaluation reports to Treasury Board Secretariat; and 
• submitting reports periodically to the Audit and Evaluation Committee concerning the status of actions taken by managers in response to evaluation reports and problems encountered by managers in implementing recommendations; 
• informing policy and program managers when approved reports will be posted on the Internet and allowing sufficient time for the preparation of a communications plan; and, 
• posting of approved reports on the departmental Intranet and Internet sites.

V. PUBLIC ACCESS

Once completed and approved, all reports are posted on the Department’s Internet and Intranet sites in both official languages within 60 working days after AEC approval . The reports are also accessible by the public in accordance with the Treasury Board Review Policy and the Access to Information and Privacy Acts.

VI. OTHER CONSIDERATIONS

When designing the evaluation approach and especially in the preparation of evaluation questions for the evaluation of any departmental program or policy, special consideration will be given as to the relevance and inclusion of questions that examine the differential impacts of programs and policies on employment equity groups, linguistic groups, gender and other relevant diversity groups.

VII. REFERENCES

• Treasury Board Policy on Internal Audit (April 2001)
• Treasury Board Policy on Evaluation (April 2001)
• Treasury Board Policy on Transfer Payments (June 2000)
• Access to Information Act
• Privacy Act
• Results for Canadians: A Management Framework for the Government of Canada (March 2000)
• Official Languages Act
• Employment Equity Act
• Policies are found on the Treasury Board internet site

VIII. ENQUIRIES

Enquiries about this policy should be directed to:

Director, Evaluation Division
Policy Integration and Coordination Section
Policy Sector

GLOSSARY OF TERMS

Activities

An operation or work process internal to an organisation, intended to produce specific outputs (e.g. products or services). Activities are the primary link in the chain through which outcomes are achieved.

Goal

A broad, high-level statement of a desired outcome, in general terms, to be achieved over an unspecified period of time. A goal should reflect an organization’s “Mission”.

Logic Model

A graphic representation of the program “theory” or “action”.  It consists of a logical chain of if-then relationships; if x occurs, then y will occur that shows the linkage from the activities through the sequence of outcomes 

Mission

A statement identifying an organization’s business, purpose and reason for existence – critical areas within which goals, objectives and standards should be set.

Objective

A statement of specific results to be achieved over a specified period of time. Objectives are generally lower-level and shorter term than a goal.

Outcome/Result

The effect of the outputs of a program on client or target groups.  In other words, outcomes/results are the changes a program or policy hopes to achieve.  Outcomes/Results focus on what the program or policy makes happen rather than what it does (i.e. the intended results of the project, not the process of achieving them). They may be described as: immediate, intermediate or final, direct or indirect, intended or unintended.

Output

A unit of service provided, product provided, or people served by a program or policy; or a count of goods and services produced.

Performance Measurement:

Consists of tracking program performance against goals over time to provide an assessment of a program’s performance, including measures of productivity, effectiveness, quality, and timeliness.  Performance Measurement can help provide objective perspectives for defending or expanding a program, rather than allowing it to suffer from relatively arbitrary or habitual decisions. Ongoing monitoring systems, which emphasize indicators and analysis linked to improvement, can help track and improve results over time and can also prove to be a valuable source of information in the formal evaluation process.

Program evaluation

Employs a set of applied research instruments to provide a systematic, objective assessment of elements of a program’s performance.  This information provides managers and other stakeholders with timely, relevant, credible and objective information on the continued relevance of government and departmental policies and programs, the impacts they are producing and opportunities for using alternative and more cost-effective policy and programming instruments.  Depending on the timing of the evaluation, it can consist of:

• a formative, implementation or mid-term evaluation which provides information mid-way through a program by examining the delivery of the program, the quality of its implementation and the assessment of the organizational context, personnel procedures and inputs; or  
• a summative or impact evaluation which determines the overall impact a program has had by examining the effects or outcomes of programs.

Summative Evaluations focus on three primary concerns: 

• issues of relevance, or more aptly, whether or not program or policy instruments, , continue to address strategic priorities and/or actual needs; i.e. the extent to which the objectives and mandate of the program or policy are still relevant and the extent to which the activities and outputs of a program or policy are consistent with the mandate and plausibly linked to the attainment of stated objectives and intended impacts;
• issues of success, including the degree to which program or policy instruments are meeting stated objectives (i.e. impacts), and without unwarranted, undesirable impacts, and
• issues of cost-effectiveness such as whether the most efficient means are used to achieve objectives relative to alternative approaches including whether another level of government could assume responsibility for the policy or program instrument.

Program Evaluation Process:

Program Evaluation Process Consists of four stages:

• planning and design
• data gathering and analysis
• reporting
• follow-up

Stage 1: Planning Stage

The planning stage consists of developing plans for the approach to the evaluation of existing, new or substantially altered programs or policies. The planning stage involves intensive consultations with program managers, clients and other interested stakeholders. It is important that this be done at the beginning of a new program or policy or as early on as possible in the development of a program or policy  to ensure that the objectives are stated in a manner that allows for the ready identification of performance indicators and the systematic collection of performance information required for organizational learning and management decision-making.  As part of the planning stage, evaluation undertakes an analysis of available data to determine the degree to which a range of issues can be addressed using existing data as well as the need for the collection of new data elements.  The planning stage culminates in the production of a Results-based Management and Accountability Framework (RMAF) document (or an evaluation framework, assessment framework or evaluation workplan). 

Stage 2:  Data Gathering and Analysis

The data gathering and analysis stage involves the actual fieldwork for the completion of the evaluation project as well as the analysis of the findings from the various sources, including the monitoring of ongoing performance measures.  For more complex projects , the data gathering and analysis stage may extend over more than one fiscal year.

Stage 3:  Reporting Stage

The reporting stage consists of reporting evaluation findings to the Deputy Minister, departmental managers, central agencies, Parliament and ultimately the public.

Stage 4: Follow-up

Follow-up activities involve the formulation of recommendations for changes where warranted in terms of any of the four areas listed above.  The Program area being evaluated is required to prepare a management response.  The Evaluation Division is available to assist program managers to formulate action plans as part of their management response to ameliorate any outstanding issues based on evaluation findings. This follow-up evaluation service can also include assistance in monitoring the implementation of the action plan.

Results-based Management and Accountability Framework (RMAF):

A blueprint for managers to help them focus on measuring and reporting on outcomes throughout the lifecycle of a policy or program.  RMAFs are a requirement of the Treasury Board Policy on Transfer Payments (http://www.tbs-sct.gc.ca/Pubs_pol/dcgpubs/TBM_142/ptp_e.html) and are commonly required by Treasury Board (TB) in the approval of new or renewed programs. RMAFs are also called for under the TB Evaluation Policy whenever they make sense for the purpose of measuring and reporting on results.  RMAFs generally include: 

• a clear statement of the roles and responsibilities of the main partners involved in delivering the policy or program;
• a clear articulation of the resources to be applied and the objectives, activities, outputs and key results/outcomes to be achieved, along with their linkages (see Glossary of Terms for a description of each of these terms);
• an outline of the performance measurement strategy, including costs and performance information (key indicators) that will be tracked;
• the schedule of major evaluation work expected to be done; and
• an outline of the reporting provisions as appropriate for funding recipients and those for the department, including

RMAFs are a useful management tool for significant policies or programs, regardless of whether they are produced in compliance with an “official” government requirement.  However, when an RMAF is not specifically required by TB and where a manager nonetheless wishes to have a framework to assist in the evaluation of a program or policy, it is sometimes called an evaluation framework, assessment framework or evaluation workplan.  Essentially, these terms are equivalent to an RMAF but have more flexibility in their components (because they are not required by TB).

---

• ^[1] The terms “program evaluation” and “evaluation” are used interchangeably in this document.
• ^[2] The term “program” in this document, also refers to/includes “initiatives”.
• ^[3] In the Treasury Board evaluation policy these responsibilities rest with the Deputy Minister.  However, the Deputy Minister may delegate these responsibilities to the Audit & Evaluation Committee.
• ^[4] Evaluation staff routinely participate early in the policy and program development process in order to ensure that expected outcomes of policies are defined in terms which are amenable to subsequent evaluation.
• ^[5]The Director, Program Evaluation has a line reporting relationship to the Director General, Policy Integration and Coordination, Policy Sector as well as a functional reporting relationship to the Deputy Minister.