Evaluation of Public Legal Education And Information: An Annotated Bibliography

• 3. Government Documents
  • 3.1 PLEI-Related Documents

• Previous Page |
• Table of Contents |
• Next Page

---

3. Government Documents

3.1 PLEI-Related Documents

A number of the documents reviewed in this section precede the five-year period mandated in the bibliography's objectives. Some older materials were included if they were believed to be relevant to current evaluation contexts. Others, such as the 1986 evaluation assessment,were included to provide an historical perspective on the types of research conducted by and for the Department of Justice. This report can be contrasted with the most recent, comprehensive PLEI evaluation from 1997: Tools for moving forward.

Abdelahad, L., Sansfacon, D. & Beaulne, A. (1989). Access to justice: research reports on PLEI, research notes. Research and Development Directorate, Department of Justice Canada.

General Overview

This compilation of "notes" summarizes twelve PLEI-related research reports written for the Department of Justice's "Access to Justice" series between 1986 and 1987. Although the summaries do not provide guidance for evaluating PLEI in themselves, four of the entries may prove useful in developing an evaluation framework. In 1984 the Department of Justice created the "Access to Legal Information Fund" along with a strategy to fund new PLEI organizations in six provinces and territories. The funding was granted for a three-year period and toward the end of that time a call for tenders to evaluate these programs was issued. Two external consulting companies were hired to evaluate three out of the six programs (Manitoba, Yukon and Newfoundland), and their reports are annotated in this document.

• R. Hikel, C. Meredith, C. Wihak, A. Woods (ABT Associates of Canada) (1986): Evaluation of the Community Legal Education Association [Manitoba], Department of Justice, Ottawa.
• R. Hikel, C. Meredith, C. Wihak, A. Woods (ABT Associates of Canada) (1987): Evaluation of the Yukon Public Legal Education Association, Department of Justice, Ottawa.
• D.H. Access Research Associates Inc. (1987): Evaluation of the Public Legal Information Association of Newfoundland, Department of Justice, Ottawa.

The other summary in this collection that is especially relevant to PLEI evaluation was on the Resource Book created by Focus Consultants as a result of the proposal annotated below.

Currie, Janet & Roberts, Tim (1986): An Evaluation Resource Book for Public Legal Education and Information Organizations, Department of Justice, Ottawa. [See section 4.2]

Alderson-Gill and Associates. (1995). Public legal education and information program: A new vision for PLEI. Department of Justice Canada.

General Overview

This document outlines a new role for the Department of Justice within the existing PLEI network and proposes three areas for emphasis and development. The three specified areas are: (i) cultivation of partnerships; (ii) ACJ Net; and (iii) focus on education. The new "vision" was created with the intention of addressing limited availability of resources while maximizing national benefits of PLEI activities. Each area of emphasis identified in the report situates the Department in a position to undertake the necessary evaluation components. Moreover, there is explicit recognition that assuming a leadership role in specialized research and evaluation of PLEI is a cost effective measure for all parties participating in PLEI. Under the mandate of cultivating partnerships, the Department can facilitate networking between PLEI providers. Through centralized coordination, providers can be certain that activities and efforts are not unnecessarily duplicating those of others. This encourages information sharing and should ultimately lead to incremental improvements for PLEI. Examples given for the Department's role include fostering a national network of organizations, creating a national PLEI advisory process, maintaining a current telecommunications network, and coordinating national strategies. Much of this work can be facilitated by ACJNet, which is the second major area of emphasis recommended. The link between evaluation and an emphasis on education is clearly anticipated in the report. By focusing on this area, Departmental contributions will increase knowledge about the relative effectiveness of different approaches to legal education through research and evaluation that is too costly for individual jurisdictions to fund in a systematic way. As the need for research and evaluation is widely acknowledged, the Department has a role to play in demonstrating leadership and coordinating partnerships in this area.

Challenges of Funder's Evaluation Requirements

• Paper states that due to scarce resources funders are most interested in knowing whether PLEI services are having an impact and that "evidence of impact is easier to come by in the case of targeted PLEI programs than it is in the general distribution of PLEI pamphlets and booklets to the public" (p.13).
• Funders that focus on quantitative data i.e., how many pamphlets were distributed, overlook the more active meaning of PLEI dissemination that "incorporates a responsibility to ensure that the recipients receive the information, understand it and act on it."

Evaluation Methodologies

• PLEI involving plain language in legislation, etc. requires testing on consumer/client groups to determine if the language is clear and if the impact of the change can be perceived
• Use of ACJNet to facilitate ongoing consultations, both formal and informal about programs and materials

Currie, J. & Roberts, T. (1984). An evaluation resource book for PLEI organizations: A proposal. Department of Justice Canada.

General Overview

This is a proposal for the creation of an evaluation resource book for PLEI organizations. The Canadian Law Information Council ("CLIC"), which was the clearinghouse for PLEI in Canada at the time of the research, contracted out this project to Focus Consultants. The paper identifies and discusses a number of basic challenges and specific problems that inhibit PLEI evaluation. Although most groups contacted did engage in some form of evaluation, the general consensus was that current evaluations were ineffective, incomplete and poorly utilized for the desired purposes. In advocating for the creation of a Canadian PLEI evaluation resource book, the authors note that "while there is a great deal of evaluation material being produced for human services and educational organizations, [they] found little that relates specifically to PLEI. PLEI incorporates aspects of both human service and educational programming but it has a distinctive overlay because of its legal focus and involvement of legal professionals" (p.23). Moreover, the PLEI providers interviewed identified a number of evaluation skills that were lacking within their organizations. They all thought that the proposed resource book would be extremely useful, especially for ensuring a common understanding between the Government, other funders and among PLEI organizations. Most of the challenges and specific problems identified in the report can be addressed though centralization of evaluation materials and criteria by an agency such as PLEAC or a branch of the Department of Justice. Many of the obstacles to meaningful evaluation come from a lack of clarity as to definitions and objectives of the projects and the evaluation itself. By providing guidelines and the skills/methods required to undertake evaluation, some of the burden may be alleviated. The challenges and obstacles enumerated in this report are still relevant to PLEI evaluation today. Together with the resource book that was written by the same authors in 1986, this report has enduring significance and should be used in developing evaluation materials and information.

The Challenges of PLEI Evaluation

Contextual Factors

• The findings of this extensive research revealed a number of contextual factors that present significant barriers to PLEI evaluation. These include:
  1. Lack of experience and expertise
  2. Diversity in the structure and delivery of programs. Differences in focus and format often relate to varying definitions and/or intentions of the program. For example, a program targeted at knowledge acquisition ought to be assessed differently from one that seeks to effect attitudinal changes. Similarly, some services tend to be case-flow oriented and therefore outcome-driven, while others adopt broader objectives of empowerment, equality enhancement and increased democratic participation. Again, it is necessary to understand these objectives before approaching a program for evaluation.
  3. Different emphasis and goals held by funders and providers leads to distinctive modes of evaluation (qualitative versus quantitative)
  4. Resistance and inability of lawyers to elicit feedback for their services
  5. Ideological opposition to reinforcing barriers between participants and the authoritative "teachers"
  6. Practical considerations include: confidentiality of client information and lack of funds
     1. Lack of funding in the system generally (evaluation = cutting programs)
     2. Lack of funds in organizations: low priority to direct resources toward evaluation
     3. Lack of significant funds from any one funder to support comprehensive evaluation
• The cumulative effect of these contextual barriers was that evaluation was seen as "too difficult, too time consuming and too painful."

Problem Areas Discovered

• Contextual factors are often difficult to change
• Internal evaluations:
  1. Lack of evaluation skills in the staff and lawyers working in the organizations
  2. Lack of objectivity
  3. Lack of information sharing between PLEI organizations
  4. Resort to purely quantitative evaluation to satisfy funders
  5. Questionnaires and surveys are not sufficiently critical or in depth to assist with increasing effectivity
  6. Poorly articulated goals and objectives
• External evaluations:
  1. Lack of control over type and content of evaluation
  2. Lack of commitment to using results beyond statistics collected
  3. Funders and external evaluators do not accept responsibility for programs
  4. Poorly articulated goals and objectives (interviewees indicated that goals were often too lofty or unrealistic to produce a meaningful evaluation)

Evaluation Methodologies

• External, professional evaluations were discussed in the report and although they can be very beneficial in some circumstances, they cannot constitute the primary method of ongoing PLEI evaluation.
• Informal methods discussed include: peer review, staff "think tanks" and community networks which involve other PLEI and non-PLEI organizations that share or overlap in terms of clientele or research areas.

Godin, J. (1994). More than a crime: A report on the lack of public legal information materials for immigrant women who are subject to wife assault. Research Section, Department of Justice Canada.

---

• Previous Page |
• Table of Contents |
• Next Page