Department of Justice
www.justice.gc.ca
Common menu bar links
Evaluation of Public Legal Education And Information: An Annotated Bibliography
4. Evaluation Reports
4.1 PLEI Evaluations
The reports annotated in this section are actual evaluations, both internal and external, of recent PLEI initiatives. Based on the selection, it can be noted that PLEI organizations that perform comprehensive evaluations do so in a methodical and consistent manner. The approach, documentation and analysis in many of these reports can be applied to various PLEI programs in other jurisdictions.
Burtch, B. & Reid, K. (1993). Discovering barriers to legal education: First-generation immigrants in greater Vancouver. Vancouver: The People's Law School.
General Overview
This evaluation report was produced for the People's Law School ("PLS") in Vancouver, British Columbia. The objectives of this assessment were three-fold: (i) to determine who is using the PLEI programs provided by the school; (ii) what barriers exist for first generation immigrants to accessing legal information resources and how PLS can overcome these barriers; and (iii) to measure the role played by PLEI in immigrant settlement and adaptation. The primary method of PLEI provision addressed is the school's free law classes that are offered as part of the Cultural Minorities Program. The study used a control group approach whereby participants were divided on the basis of whether or not they had attended PLS free law classes. The study is intended to improve the delivery of services offered by PLS and to provide "information of direct relevance to immigrant-serving agencies in B.C., as well as people concerned with policy development and the administration of justice in B.C."
Much of this report may be useful to organizations that encounter first generation immigrant populations in their work. However, the actual questionnaire and Interviewer's Manual may be very helpful to a wider range of PLEI organizations. In particular, the "rules for interviewing" provide some simple directions for eliminating bias in questioning and prompting. These instructions are especially pertinent where consistency concerns arise due to multiple interviewers. Furthermore, the background information and legal needs questions may be easily adapted to accommodate different populations and other topics of legal information.
Common Components of Evaluation
- External evaluation
- Four objectives for the study are clearly articulated in the document and are linked to the conclusions
- Recommendations include a number of suggestions for further research/assessment projects
The Challenges of PLEI Evaluation
- Evaluators found it difficult to generate a population sample. Most of those approached were suspicious of the law, often because of political repression experienced in their country of origin, or thought that the data would be used for government purposes. Moreover, many participants perceived the questions as dealing with private matters and were hesitant to discuss income levels and attitudes toward policies.
Evaluation Methodologies
- The study was conducted by way of semi-structured interviews that followed a lengthy questionnaire. The questionnaire was divided into two parts: (i) 24 questions regarding background information (personal characteristics); and (ii) 18 questions regarding legal information needs.
- The questionnaires in all five languages were pre-tested and modified where necessary.
- Interviewers were recruited from the five ethnic communities surveyed. All interviewers were bilingual and attempts were made to match genders between the interviewer and interviewee. Interviewers participated in a training session on the questionnaire and were given an Interviewer's Manual that included general rules for interviewing as well as specific background information and probing techniques for most of the questions.
- Hypothetical legal scenarios were used to test knowledge differentials for those who have attended free law classes in each of the five cultural groups.
Curtis, C. K. & Meehan, G. (2001). Courtlink auto crime prevention program: Evaluation report. Vancouver: Law Courts Education Society of British Columbia.
General Overview
This 13-hour program was designed for senior elementary classes and alternative schools in British Columbia and was developed by the Law Courts Education Society of B.C. The goals of the evaluation were two-fold: (i) to identify reactions of experiences with the program; and (ii) to assess the degree to which the program met its two stated goals of increasing knowledge and promoting positive attitudes. Different evaluation tools were used to measure the two goals. This report exhibits an extremely thorough understanding of evaluation methodologies and included a detailed analysis of findings. At every stage of the report the authors explain why particular choices were made in the process and offer suggestions as to effective methods for developing PLEI evaluations for different populations. Similar to other formal, external evaluations, the methodologies sections provide useful information for subsequent PLEI evaluations. The steps taken in developing both the survey research and the pre-test/post-test instruments are explained in detail and may be adapted to other educational programs. Research questions and various assessment instruments that were developed are all appended to the report and would be a good starting point for any school-based PLEI programs.
Common Components of Evaluation
- External evaluation
- Built into the project design and conducted pre-program, concurrently and post-program
- Goals of the evaluation and the educational program were articulated in the report and evaluation methods were matched to different goals.
- One of the recommendations suggested an annual review and update of materials used by teachers to ensure that they are time sensitive. This is a common way that projects can perform ongoing monitoring without incurring great expenses of hiring external evaluators.
The Challenges of PLEI Evaluation
- Open-ended questions are often desirable to elicit emotional, psychological and intuitive responses, however, it is important for PLEI providers to rank the preferred or "more desirable" answers to questions in order to clarify intended outcomes.
- Limitations placed on feedback surveys may encourage consistency in reporting. It is important to offer an opportunity to discuss "factors" that affect the administration of the program somewhere on the questionnaire. For example, it is not sufficient to ask whether a particular activity was useful or effective without knowing how much time was allocated to that activity or if it was appropriate to the particular students involved.
- One draw back of pre-testing is that participants have knowledge of the study which may threaten the validity of the tests.
Connections between Goals of PLEI and Evaluation
- Goals of PLEI programs should be realistic and sensitive to target audiences and environments. For example, if students show possession of prior knowledge about a particular topic, facilitators should not expect to see "substantial gains in knowledge" which may be a stated goal.
Evaluation Methodologies
- Questionnaires were constructed for three groups (facilitators were given two different questionnaires to complete) to measure the experiences of the program.
- Pre- and post-tests were given to students to measure change in knowledge and attitude
- All tests were pilot-tested and modifications were made accordingly
- Tests included some scenario questions that asked for responses as to what should be done and what action would likely be taken
- Evaluators also sought oral feedback from students and teachers regarding what they thought and how they felt about the program.
Ellis, D. (1997). Program review. Vancouver: The People's Law School.
General Overview
This study summarizes a comprehensive review of the People's Law School ("PLS") PLEI programs. It involved a range of PLEI stakeholders in an attempt to view PLS services from a local and provincial perspective. Thirty conclusions were drawn, which were grouped into eight categories of inquiry, as well as five recommendations directed specifically toward the staff and Board of PLS. Although the "findings and learnings" collected for each population or method of evaluation were intended for the staff and Board of PLS, other PLEI organizations will benefit from reviewing these findings. The evaluation is clearly written and the structure of the report allows for easy access to relevant information, especially the bulleted section that follows each question entitled "What do we/can we learn from these findings?" The tools used in the various evaluation components are appended to the report and may be useful to PLEI providers in designing evaluations. These include samples of the interview guide, client feedback sheet, resource/pamphlet feedback sheet and focus group guides with topics and questions.
Common Components of Evaluation
- External evaluation
- Conducted post-program
- Program review had clear, but broad objectives:
"to analyze the efficiency and effectiveness of the present programs of the People's Law School with regard to current and emerging needs for PLEI and to consider the Society's role in relation to other PLEI providers in the province"
(p.8). - No comment on further or ongoing evaluations
The Challenges of PLEI Evaluation
- It is difficult to assess whether there is overlap between programs for a number of reasons:
- PLEI providers believe that need is so great that there can never be enough PLEI
- No commitment to sharing information in order to decrease duplication
- Not enough trust between organizations to effectively coordinate initiatives
- Belief that innovation is more likely to occur with a larger number of smaller organizations than with larger amalgamations
Challenges of Funders' Evaluation Requirements
- Despite the reticence to discuss overlap, funders are highly concerned with duplication of efforts and program overlap between PLEI providers and want this type of analysis included in evaluations.
- Suggestions for better coordination between core PLEI providers came primarily from funders even though there is reluctance from PLEI organizations.
Evaluation Methodologies
- Identified methodology was a qualitative, participatory approach
- Two focus groups: (i) staff at PLS and (ii) Board of PLS
- Materials/pamphlet feedback from PLEI distributors
- Interviews with funders, groups with unmet PLEI needs, co-sponsors of PLS programs and observer-experts in the PLEI field all conducted in informant's native language
- Client feedback surveys (684 were returned)
Law Courts Education Society of British Columbia. (2000). Journeys of justice First Nations research project: Final report. Vancouver: Law Courts Education Society of British Columbia.
