Public Safety and Anti-terrorism (PSAT) Initiative,
Summative Evaluation

Appendix A: Evaluation Framework

Summative Evaluation Framework Table

Relevance
  Issues Indicators Data Sources Data Collection / Analysis Methods
1. What is the ongoing level of need for a program that supports elevated efforts to combat public security issues and the threat of terrorism in Canada and abroad? Identified needs of users of DOJ PSAT programming (departments, agencies, P/T jurisdictions, foreign jurisdictions)

Extent to which current programming and PSAT resources can address those needs		 Users and departmental representatives

Documentation		 Interviews

Documentation review
2. Are the Department's PSAT objectives in line with the government-wide initiative? Consistency with the government-wide PSAT Initiative Departmental reports

Key contacts		 Document review

Interviews

Success
  Issues Indicators Data Sources Data Collection / Analysis Methods
3. To what extent have the DOJ PSAT activities engaged departmental representatives, client organizations and other jurisdictions in relevant public safety and anti-terrorism activities? List of other departmental representatives, client organizations and other jurisdictions involved in public safety and anti-terrorism activities and types of relationship

Number, nature and complexity of requests for advice / assistance on PSAT-related issues		 Departmental files

Caseview		 File review

Database analysis
4. In what ways is the DOJ, through its PSAT activities within the Department, in client organizations and with other jurisdictions, contributing to the development of more effective laws, regulations and policies? Early involvement of all relevant stakeholders in the development of laws and regulations

Involvement and contribution to the development of interdepartmental initiatives

Number, nature and complexity of agreements / MOUs worked on

Assistance provided to other jurisdictions		 Key contacts

Departmental files		 Interviews

Documentation review
5. To what extent have DOJ PSAT activities contributed to increased knowledge and understanding among key participants in the justice system and others of laws and regulations related to public safety and anti-terrorism? # of training sessions

# of participants

Change in levels of awareness by participants within the justice system of the ATA and other PSAT-related legislation for which the Minister of Justice is directly responsible

Change in levels of awareness within ethno-cultural communities, DOJ and justice system practitioners about the impact of the legislation on those communities		 Departmental files

Key contacts and partners

Research studies (e.g., ethno-cultural questionnaire, focus groups)

Website statistics		 File review

Key contact and partner interviews

Document review

Statistical analysis
6. To what extent have DOJ PSAT activities enhanced the ability of departmental representatives, client organizations and other jurisdictions to investigate, litigate and / or prosecute cases where terrorism or other threats to public safety and anti-terrorism are involved or where disclosure of sensitive information is an issue? Types of criminal charges laid under the ATA

Use of recognizance with conditions and investigative hearing provisions of the ATA.

#, nature and complexity of requests for legal advice and support by investigative agencies

# and scope of cases where advice / assistance was provided with respect to applications of s.38 (CEA) of cases was used

# and scope of cases where s.38 was used or considered for use

#, nature and complexity of litigation and prosecutions carried out by DOJ counsel

Number of cases where entry was denied or removal from Canada occurred on national security grounds		 Departmental files

Caseview

RCMP, provincial AG & CCJS reports

Annual Report to Parliament and provincial AG reports

Key contacts and partners		 File review

Database analysis

Document review

Key contact and partner interviews
7. To what extent has legal aid been provided to the economically disadvantaged accused affected by public safety and anti-terrorism initiatives? Number of legal aid cases by type of proceeding

Amount of funding spent in compliance with federal / provincial / territorial agreements		 Departmental files File review
8. Are there more effective programs, use of intelligence, investigation and detection by federal / provincial / territorial governments related to public safety and anti-terrorism? Increase in awareness of security threats / terrorist activity among intelligence and law enforcement agencies

Contribution of DOJ support to enhancing the effectiveness of programs, use of intelligence, investigation and detection		 Key contacts and partners Key contact and partner interviews
9. In what way have DOJ PSAT activities contributed to more effective litigation and prosecution? Timeliness in bringing cases to courts / tribunals and in processing them

Trends in case outcomes / dispositions

Assessments of the effectiveness of DOJ litigation and prosecutions		 Departmental files

Caseview

Key contacts		 File review

Database analysis

Key contact interviews
10. What role has DOJ had in improving international efforts in the fight against terrorism? DOJ's contribution to Canada's efforts in the global fight against terrorism

Outcomes / dispositions of mutual legal assistance requests and extradition proceedings

Membership in, and levels of support provided to foreign governments and international bodies		 Key contacts

Departmental files		 Key contact and partner interviews

File review
11. What has been the contribution of DOJ PSAT activities to greater access to justice? Number and types of challenges and their outcomes / use of tools available

Number and circumstances of unrepresented defendants in terrorism-related cases

Contribution of DOJ in greater access to justice		 Departmental files

Key contacts and experts		 Key contact and expert interviews
12. What have been the unintended impacts, positive or negative, of DOJ PSAT activities? Incidence of unintended impacts Key contacts and departmental representatives Interviews

Cost-Effectiveness and Alternatives
  Issues Indicators Data Sources Data Collection / Analysis Methods
13. Is the level of resources allocated to the Department sufficient to address the current need in relation to the PSAT Initiative? Planned vs. actual resource use

PSAT-related workload

International comparisons

Number and description of activities that could not be undertaken or could not be undertaken in a timely manner		 Departmental financial data

Departmental operational reports

Document review

Key contacts		 Data analysis

File review

Key contact interviews
14. Are PSAT-related cases and resources within the Department being managed in an efficient and effective manner Trends in plea bargaining and use of agents

Extent to which resources are allocated to areas of highest priority and workloads

Comparisons with similar measures in other jurisdictions		 Departmental files

Key contacts

Documents		 File review

Key contact interviews

Document review
15. Are there more effective ways of achieving the objectives of the initiative? Comparison with other jurisdictions

Comparison of the effectiveness of alternative approaches		 Departmental files

Documents

Key contacts		 File review

Document review

Key contact interviews

Performance Measurement System
  Issues Indicators Data Sources Data Collection / Analysis Methods
16. How effective is the ongoing performance measurement system of the PSAT Initiative? Extent to which the new RMAF performance measurement strategy has been implemented

Extent to which it is useful for decision-making purposes

Extent to which it is complete (i.e., that the information required is available)		 Departmental representatives

Departmental files		 Interviews

File review