Public Safety and Anti-terrorism (PSAT) Initiative,
Summative Evaluation

4. Methodology

4. Methodology

The evaluation included two methods of data collection: a document and data review and interviews with key contacts. This section describes these methods and discusses limitations to the methodology.

4.1. Document and data review

The document and data review provided contextual information on the Initiative: its rationale and expected results, the operation of the Initiative, its outputs, and outcomes achieved. Each Departmental section that participated in the evaluation was asked to provide relevant documents to the evaluators. Examples of documents reviewed for the evaluation include:

Information from iCase, the database that the Department uses to track the time that counsel spend on cases and files, is the main source of data for the case/file review. However, the value of iCase for this evaluation is limited (as discussed in section 4.3). In the Department's audit of iCase, national portfolio managers questioned the accuracy of iCase data and the study found that ownership of iCase data is unclear, which means that use of the data may require the consent of every section.[7] For these reasons, the evaluation worked with individual sections, focusing on data that were ostensibly reliable and useful.

4.2. Interviews

For the evaluation, 58 key contacts participated in interviews (44 Department of Justice contacts and 14 external clients). Most interviewees participated through telephone or in-person interviews, and four chose to submit written responses. Because of the decentralized nature of the Initiative, interviewees were asked to provide names of other key individuals who have either worked on PSAT activities within the Department or who have been external clients of the Department. Accordingly, the key contacts comprise all individuals who were identified and who agreed to be interviewed within the study time frame. The organizational affiliations of the key contacts are in Table 6.

Table 6: Key contacts participating in the evaluation

Department of Justice
External clients

The key contacts and clients were interviewed using a guide developed to respond to the RMAF's summative evaluation framework. The evaluation framework is in Appendix A.

4.3. Methodological limitations

The evaluation faced several methodological limitations and other challenges.

Few lines of evidence were available.

The evaluation relies on available documents, interviews conducted with key contacts, and past evaluation findings (the formative evaluation and the evaluation of the Dorval Pilot Project). Other potential lines of evidence, such as iCase and performance measurement data, were limited, as described on the next page.

iCase limitations.

Overall PSAT Initiative statistics are not available through iCase. This is due to several factors:

Performance measurement data are limited.

The availability of performance measurement data is limited and this was noted in the formative evaluation. The Department is in the process of reviewing its data collection and performance measurement strategy. This process involves each section identifying key performance indicators that would be useful for ongoing performance measurement and reporting.

Identifying performance measures.

The evaluation of the Dorval Pilot Project drew attention to the challenges involved in developing performance measures and collecting performance data on activities such as providing legal advice. The summative evaluation also experienced these challenges; for provision of legal advice and policy development, the distinction between activities, outputs, and outcomes is often unclear.

Attribution challenges.

It is difficult to attribute the results of the department's PSAT-related work to the funding provided under the Initiative. For the most part, the funds are intended to provide additional support for activities that the department already conducts, such as providing legal advice and assistance. This aspect of the initiative, as well as its cross-cutting nature, means it is difficult to define precisely which activities fall under the Initiative. As a result, one cannot make a direct link between the funds provided, the distinct activities and outputs these funds support, and the resulting impact or outcomes. Furthermore, the Department's financial systems are not designed in a manner to provide a complete picture of the activities funded under the Initiative.

Identifying key contacts outside of HQ.

Some LSUs had difficulty knowing which activities were PSAT funded and, therefore, several contacts in LSUs turned down requests for an interview, citing insufficient knowledge of the Initiative.

[7] Department of Justice (2003). Internal audit of the iCase system. Retrieved on September 20, 2006 at /en/dept/pub/audit_reports/2003/iCase2003-e.pdf.