Department of Justice
www.justice.gc.ca
Common menu bar links
Evaluation of Public Legal Education And Information: An Annotated Bibliography
3. Government Documents (continued)
3. Government Documents (continued)
3.1 PLEI-Related Documents (continued)
General Overview
As noted in the title, this paper confirms the lack of PLEI materials available at that time for immigrant women who were victims/survivors of wife assault. Through interviews and a literature review the author identifies the need for PLEI materials in this area and some of the major challenges that must be recognized if and when the materials are produced. When read as a form of needs assessment, this document reiterates that comprehensive and effective evaluation requires informative and thorough needs assessment on the front end. Only by understanding the obstacles faced by various populations to accessing legal information can the provision of that information be assessed in a meaningful way.
The Challenges of PLEI Evaluation
- Although the basic need for PLEI materials was undisputed by various organizations and agencies that dealt with immigrant women and/or victims of wife assault, the stronger message from these groups was the need for this material to be sensitive to corresponding issues faced by the target groups.
- Due to complexity, it is very difficult to evaluate cultural sensitivity.
- The author recounts that
"PLEI is not useful if it is not retained"
(p.13). This is an important point for evaluators to remember, since the common post-program survey does not account for retention levels. Therefore, while some key information such as suitability of location and time may be lost if evaluation is postponed, other data cannot be assessed immediately. Depending on the purpose of the evaluation, it may be pertinent to conduct follow up interviews after one, six, or twelve months have passed.
Challenges of Funders' Evaluation Requirements
- The objectivity gained by funders' evaluations must be offset against the unparalleled feedback that community groups can offer regarding the effectiveness of PLEI materials. Just as genuine understanding of particular circumstances is critical at the needs assessment phase, evaluation must also be tailored to reflect the cultural, social and political characteristics of the targeted group.
Evaluation Methods
- PLEI materials in this area were being field tested in consultations with relevant communities throughout the province.
Moliner, M. (1997). Public legal education and information review: Tools for moving forward. Programs Branch, Department of Justice Canada.
General Overview
This review offers the most recent assessment of the Department of Justice's involvement in PLEI programs. The report articulates multiple reasons for the Department's continued connection and suggests that Justice Canada clarify its role in PLEI delivery, which may be accomplished by ceasing to produce PLEI materials and distinguishing Departmental "Communications" from PLEI. After looking at PLEI delivery across Canada and consulting with many providers, the Advisory Committee proposes that the Department set criteria for funding PLEI, as well as introduce a multi-year funding strategy and a funding infrastructure for a national NGO such as PLEAC. The review further advises that funding and policy initiatives be targeted in order to fill existing gaps in PLEI (both population-related and issue-related gaps) and that the Department should take a leadership role in ensuring that this "targeted approach" is adopted by core PLEI providers. With regard to PLEI evaluation specifically, the review reaffirms that "evaluation of PLEI program delivery is necessary to ensure that it is achieving results"
and that the Department of Justice should "support the creation of an evaluation tool to assess whether or not PLEI is having the desired impact."
It further suggests that any evaluative framework developed be consistent with the targeted approach taken to funding and policy issues: the "needs, gaps, and priorities"
analysis. Evaluation is seen here as a shared responsibility between funders and core PLEI providers. Accordingly, the role of the Department of Justice and other funders is to identify evaluation objectives and the provider's task is to make the evaluation relevant. The single evaluation-related recommendation is found in "Proposed Direction 7": It is proposed that Justice Canada require evaluations of audiences reached by the PLEI initiatives it funds and the impact they achieved.
The Challenges of PLEI Evaluation
- Problems arise when evaluation purposes are not clearly defined, and more specifically what motives/agendas/biases are informing the evaluation.
- Many providers think it is too hard and too expensive to measure the impact of their services.
- The value of PLEI work remains largely unmeasured and therefore the willingness to continue funding programs is faltering.
- Due to the complexity of the PLEI context many critical questions go unanswered:
- Against what standard (internal or universal) is PLEI being judged;
- In any given evaluation is it PLEI that is "on trial," a particular approach, or a specific activity;
- How are relevant comparisons made when activities and audiences vary greatly;
- How do you place a value on the impact realized from each separate effort or activity;
- How do you evaluate the evaluation i.e., the results may be predetermined depending on whether the evaluator places greater "value" on cost efficiency as opposed to reaching a small group of multiply-disadvantaged persons.
Connections between Goals of PLEI and its Evaluation
- PLEI organizations understand and routinely insist that the evaluation be tied to goals and objectives.
- The necessary objectives cited for determining whether a PLEI program is successful were taken from a 1979 Report of the "Law and the Layman Committee." Though unquestionably important, these criteria were fairly simplistic and uninformative in terms of providing guidance for how evaluations may be addressed.
Challenges of Funder's Evaluation Requirements
- Concern that funder-driven evaluations will be misinterpreted and lead to skewed funding decisions
- An ongoing concern for the Department is that
"the reputation of [its] PLEI program remains misunderstood if it is unable to present a business case which indicates that the services it funds are effective as well as cost-efficient"
(p.39). - A program review initiated by the Law Foundation of Alberta resulted in cutting 40% of funding to programs such as PLEI intermediary organizations. This illustrates that the fear that evaluations equal funding cuts is valid.
- Even when methods for evaluating PLEI are devised, it is difficult to attach a dollar value to the various successes or shortfalls that emerge.
Evaluation Methodologies
- Types of "soft" evaluation methods discussed include: gathering anecdotal comments, counting heads, using evaluation forms, soliciting expert critiques, field testing and comparing efforts of one PLEI program to another to learn from the experiences of colleagues
- One PLEI provider suggested the use of a
"pre-test experimental design [that would] assess changes in knowledge, attitudes, etc"
(p.42). - Another method suggested was to engage in participant observation (i.e. in small claims court) to observe changes in self-help skills.
- Other methods discussed in the electronic conference were:
- Qualitative – anecdotal comments, focus groups, field testing;
- Formative – early client input, focus groups of target populations, field testing;
- Peer review – specific activities or informal/spontaneous feedback
- Methods can be facilitated by provincial and national PLEI associations and it is generally agreed that experiences should be shared to a greater extent.
- Due to a lack of resources longitudinal studies are not realistic and other types of evaluative interactions/efforts have not been well documented. However, feedback seems to suggest that these less formalized methods are beneficial, but that they must be synthesized in some manner in order to track changes and incorporate transformations more effectively.
- The report does not indicate that PLEI providers would like to see formal evaluations substituted for other methods of assessments and impact analysis.
Useful Materials for PLEI Evaluation
One possible evaluation methodology was suggested where the purposes of evaluating PLEI would be to:
- Evaluate what PLEI needs are met by core PLEI providers. Specifically:
- Who is reached?
- How did they gain access?
- Did they understand the information they received?
- Were they able to act effectively on the basis of the legal information provided?
- Evaluate what gaps exist in PLEI delivery:
- Who is not reached and why?
- It is anticipated that adhering to this type of evaluation would facilitate priority setting by identifying which populations are most in need and are not being adequately served by existing programs.
- Who is not reached and why?
Public legal education and information: An evaluation assessment. (1986). Programme Evaluation Section, Department of Justice Canada.
General Overview
This document looks at the current state of PLEI delivered and funded by the Department and contemplates whether the operation as a whole should undergo a full-scale evaluation. The evaluation is focused on the level of support by the Department to other PLEI providers, and therefore, it is somewhat limited in its applicability to independent PLEI organizations who are engaged in self-evaluation. In light of the interrelationship between definitional clarity and useful evaluation, a portion of the text is dedicated to defining the terms "public" "legal" "education" and "information." The following section provides a profile of the Department's PLEI program both pre and post 1984 by outlining various initiatives. It is noted that there is a definite lack of policy development in this area. The paper divides major issues of evaluation and methodology into two parts: (i) program management issues (highlighting flaws in design and implementation) and (ii) impact and effectiveness issues. A series of questions accompany each "issue."
Under the program management rubric, questions are styled to help refine and focus the program and the manner in which funding decisions are made. The questions under the second rubric are more directly relevant to PLEI organizations or their funders who are seeking accountability. Despite the general nature of the questions, they may be useful for guiding development of evaluation criteria and methodology. However, this assessment is not universally applicable to PLEI provider evaluation. For example, a PLEI organization would want to elicit responses that enable them to measure a program's effectiveness for a particular target group. Yet, in this study the question of "how much understanding of the information provided do the recipients display"
was deemed to be "non-evaluable."
In summary, the division between management and impact/effectiveness used in this study may still be useful for Departmental assessments when considering its role in PLEI. Beyond that there is little practical utility to this paper for PLEI providers as the basis for this particular assessment was to determine the degree of risk for the Department if it made a decision about its PLEI programs with or without the results of a full-scale evaluation. Thus, where the goals and objectives of an evaluation are related to overall efficiency of resource use for the Department the issues raised are still applicable, whereas for purposes of ongoing monitoring and specific program evaluation, they are not critical. Finally, the costs involved in a full evaluation such as the one described are much higher than could be realistically spent by a PLEI organization, even on the provincial level. Therefore, the only PLEI partner who could benefit from using the information provided in this document would be the Department of Justice as they endeavour to assess their role in the Canadian PLEI landscape.
Evaluation Methodologies
- Interviews with Department of Justice officials
- Expert assessment of PLEI materials
- Documentation review
- Project review
- Surveys (e.g. PLEI sponsors, other PLEI programs, etc.)
- Incorporating information from other evaluations
Useful Materials for PLEI Evaluation
Some of the questions included in the impact and effectiveness section include:
- Whether the PLEI projects funded by the Department have produced the intended effect of providing the general public and/or specified target groups with information related to their needs
- Whether the availability of PLEI has increased the general public's and/or selected groups' knowledge and understanding of the law
- Whether the PLEI programme has led to a change in public and/or target group attitudes to the law and the legal system
- Whether the Department has promoted the development of new methods and techniques for PLEI delivery
3.2 Other Evaluation Documents
The Department of Justice has undertaken many evaluations of the PLEI projects it has funded. Although this practice has been less frequent in recent years, the report noted below is a good example of a governmental impact assessment.
Ellis, P. (1995). Educational programs that alter knowledge, attitudes and behaviours of youth. Research and Statistics Division and Evaluation Directorate, Department of Justice Canada.
This is a good review of literature and discussion on changing attitudes through Youth-in-school programs, Youth-out-of-school programs and also with regard to adult education and attitude changes in general. It looks at 8 possible behaviour modification components and surveys 46 documents that study and address attitudinal change though education.
