Multi-Year Evaluation Plan
2008-09 to 2010-11

3. Evaluation Process

The evaluation process for any given program component consists of four stages: planning; data gathering and analysis; reporting; and follow-up. The planning stage consists of developing plans for the approach to the evaluation of existing, new or substantially altered programs, policies or special initiatives. The planning stage involves intensive consultations with program managers, clients and other interested stakeholders. It is important that this be done at the beginning of a new initiative or as early in the development of an initiative as possible to ensure that the objectives are stated in a manner that allows for the ready identification of performance indicators and the systematic collection of performance information required for organizational learning and management decision-making.

As part of the planning stage, evaluation undertakes an analysis of available data to determine the degree to which a range of issues can be addressed using existing data as well as the need for the collection of new data elements. The planning stage culminates in the production of a RMAF document. The document describes the program (e.g., component profile), outlines the linkages between the elements (logic model), identifies the range of issues to be addressed, including the specific questions and indicators which will be used, and specifies the approach to be taken as well as the timeframe for the completion of the evaluation. In addition, where warranted, recommendations are made as to what data elements should be collected by program or policy managers in order to obtain ongoing measures of performance.

The data gathering and analysis stage involves the actual fieldwork for the completion of the evaluation project as well as the analysis of the findings from the various sources, including the monitoring of ongoing performance measures. For more complex projects, the data gathering and analysis stage may extend over more than one fiscal year.

The reporting stage consists of reporting evaluation findings to the Deputy Minister, departmental managers, central agencies, Parliament and ultimately the public.

Evaluations focus on three primary concerns:

  1. issues of relevance, or more aptly, whether or not program or policy instruments, including special initiatives, continue to address strategic priorities and/or actual needs, i.e. the extent to which the objectives and mandate of the program or policy are still relevant and the extent to which the activities and outputs of a program or policy are consistent with the mandate and plausibly linked to the attainment of stated objectives and intended impacts;

  2. issues of success, including the degree to which program or policy instruments are meeting stated objectives (i.e. impacts), and without unwarranted, undesirable impacts; and

  3. issues of cost-effectiveness, such as whether the most efficient means are used to achieve objectives relative to alternative approaches including whether another level of government could assume responsibility for the policy or program instrument.

Follow-up activities involve the formulation of recommendations for changes where warranted in terms of the areas listed above. The Program area being evaluated is required to prepare a management response. The Evaluation Division is available to assist program managers to formulate action plans as part of their management response to ameliorate any outstanding issues based on evaluation findings.

An area of growing importance is the monitoring of the implementation of the recommendations and action plans. Treasury Board has indicated that this is an area that Justice needs to strengthen. As a result, the Evaluation Division has begun including Management Action Plans with the evaluations that are submitted to the Audit and Evaluation Committee. Periodic and systematic monitoring is carried out on the implementation of these action plans.