Community Justice Help and Quality

• Previous Page
• Table of Contents
• Next Page

Scholarly studies in the last 10 – or even 20 years – have evaluated and often compared the effectiveness of assistance provided by various authorized providers of legal services, focusing on the effectiveness of services delivered by lawyers, paralegals, licensed immigration consultants (Ontario), licensed legal technicians (US), law students, and court staff or staff working out of courthouses.

But there is not a large body of academic literature that investigates the quality and effectiveness of law-related assistance provided by not-for-profit organizations, particularly with respect to robust, individualized assistance that may skirt the boundary between legal information and legal advice. As noted earlier in this report, community organizations that provide critical law-related assistance, and do not accept direct payments for their services, are generally not yet acknowledged – by regulators, bar associations, and other groups led by legal professionals – as important partners in the provision of justice services.297

What do we know, then, about the effectiveness of community justice help? How does it compare to the services from a lawyer or paralegal? This section considers, at some length, the small body of relatively recent academic literature relevant to these questions.

In this section, we discuss:

• the small body of literature that considers the quality of law-related assistance provided by not-for-profit community-based organizations;
• a small number of reports that assess the quality of legal services provided by licensed legal professionals who are not lawyers; and
• a selection of other significant research that explores the roles of lawyers and non-lawyers, and “where lawyers matter”.

We include the latter two points because, in light of the dearth of academic literature on the quality of law-related assistance provided by not-for-profit community-based organizations, we believe that literature that considers the quality of other forms of non-lawyer assistance, and looks at “where lawyers matter”, helps to shed light on the question of where assistance from a highly trained and licensed legal professional is not critical for the protection of the public.

Evaluation of law-related assistance by not-for-profit community organizations

In the previous section, we gave examples of organizations and programs that provide community justice help, noting program evaluations where publicly available. This section considers the academic literature that evaluates community justice help – law-related assistance provided by not-for-profit community-based organizations.

Study in England and Wales

In 2003, researchers conducted an in-depth study298 of the difference in lawyer and non-lawyer activity and outcomes in England and Wales, using a range of quantitative and qualitative methods. The study compared service provision by three different groups of solicitors as well as one group of not-for-profit agencies. It found that the services provided by the not-for-profit agencies were as high quality as the services provided by the groups of solicitors.

The study considered the provision of advice and assistance in welfare benefits, debt, housing, and employment cases, including cases that did not involve a legal proceeding.299 It looked at three types of assessments to determine the relative quality of lawyers and non-lawyers: assessments of client satisfaction, of peer reviewers, and of outcomes. It found that, “NFP [not-for-profit] agencies had clients with slightly higher satisfaction ratings and got significantly better results, and their work on cases was more likely to be graded at higher levels of quality by experienced practitioners working in their field”.300

The study found that, “taken as a group, non-lawyers perform to higher standards than lawyers”.301 And, even taking into account the possibility that there were factors that the researchers could not analyze or control, they found that the “results show that it is possible for non-lawyer agencies to perform at the same or higher levels of quality than lawyers, and that in itself undermines a key claim of the profession to exclusive knowledge”.302 Echoing an earlier foundational study in the UK,303 the researchers emphasized the importance of specialization as a predictor of quality.304

US evaluation research

Rebecca Sandefur, a lead researcher in the access to justice field, has conducted several research initiatives that explore, in the US context, what a “legal” problem looks like. Her reports frequently note the intersection of the law in almost every aspect of people’s lives, including housing, work, finances, benefits, and the care of dependent family members. She cautions against framing problems of this nature – with multifaceted aspects – as solely “legal”; doing so means that the focus of the possible responses to the problem are similarly legal solutions. Her research points out that, for many of these problems, legal solutions are not necessarily the only, or best, response.

On a related theme, Sandefur, with Thomas Clarke, has conducted research that evaluates the range of lawyer and non-lawyer services that people access for assistance with their justice-related problems. Much of their evaluation work focuses on services provided by people who are authorized to provide legal services who are not licensed as lawyers. This research classifies programs staffed by people who are carrying out work traditionally available only through lawyers – but who are not, in fact, fully qualified lawyers – as “Roles Beyond Lawyers” programs, or “RBL” programs.305

Sandefur and Clarke propose a three-part evaluation framework306 to support future assessments of programs that rely on new roles being performed by non-lawyers. The framework involves three broad categories for evaluation: appropriateness, efficacy, and sustainability; in other words, “[e]ssentially, researchers want to know if a program does the right thing, does it effectively, and is capable of doing it into the future”.307

The framework can “both make a material difference” and be “competently performed by someone without full legal training”. Programs that are efficacious are both “competently performed” and “positively impactful”. Programs that are sustainable are those that have durable models of training, supervision, and regulation to ensure consistent quality, and are able to maintain legitimacy among a range of stakeholders.308

This framework was applied by Sandefur and Clarke in their 2019 evaluation of three pilot navigator projects offered through New York City courts.309 The study finds that one of the navigator programs overseen by a local not-for-profit agency – the only program that provided ongoing, “for the duration” assistance – had a significant impact on case outcomes, compared to outcomes where no assistance was provided.

The navigators in the three pilot navigator programs assessed in the study provided assistance to people who came to the court. This included providing information about the legal process; assisting them with the organization of their documents; helping them to access and complete court forms (accompanying them to “do-it-yourself” kiosks and helping them to use software to complete court forms, many of which were standard forms that included codified law, for example, a list of defenses in a housing case); and accompanying them to meetings, negotiations, and into the courtroom (answering factual questions the judge may address to them).

All of the navigators in the three programs received training and were supervised, but the extent of training and supervision, and the nature of the supervision (provided by a lawyer or someone else) varied. All received training on the difference between legal information and legal advice, including what they should not do so as to avoid crossing the line into giving legal advice. All of the navigators provided assistance to tenants arriving in their courts on housing and eviction issues; one also provided assistance with debt and consumer issues.

Of the three navigator programs, one – the Access to Justice Navigators Project – was court-operated, staffed by volunteers overseen by court staff or court lawyers, and gave only in-court “navigator for the day” services. A second – the Housing Court Answers Navigators Project – also gave only in-court “navigator for the day” services and was staffed by volunteers; it was overseen by a local not-for-profit, Housing Court Answers.

The navigators in the third program – operated by another local not-for-profit, University Settlement – were paid employees at University Settlement, and were “navigators for the duration”, handling cases from intake through to resolution and beyond, distinguishing their assistance from the other two navigator programs. The University Settlement navigators program coordinated its intake activities with the Housing Court Answers project, taking on cases that had already been screened for eligibility and had characteristics to suggest that additional navigation assistance could have an impact on case outcomes. In addition to in-court assistance, they provided a range of ongoing support outside of court, such as assistance in accessing benefits and other services.310 The aim of the program was to prevent tenants from getting evicted – an ambitious goal not shared by the other two navigator-for-the-day programs.

In their evaluation study, Sandefur and Clarke find that, across the three navigator programs, all met the measures of “appropriateness” as set out in their evaluation framework. With respect to efficacy – assistance that is competently performed and has a positive impact – and particularly with respect to case outcomes, the findings vary across the three pilot sites, reflecting the different goals and activities of the programs. Both the Housing Court Answers and University Settlement navigator programs were intended to affect case outcomes on non-payment of rent cases, a goal that was not in place for the Access to Justice Navigators Program.

The evaluation also considers the sustainability of the programs, including markers relating to legitimacy, perceived value by navigators and stakeholders, and financial sustainability. Here, the researchers find that the three navigator pilot programs have the potential for sustainability, but cannot be expanded without greater financial investment, even though their costs are much less than providing legal aid lawyers to the same people.

Key findings relating to case outcomes, per the three pilot projects, include the following:

• The tenants who received in-court and ongoing out-of-court assistance from the University Settlement navigators program were much more likely to be assigned a court interpreter.311 The tenants assisted by these navigators, who first were assisted by Housing Court Answers navigators, raised significantly more defenses than unrepresented tenants, were more likely to have their defenses recognized, and were more likely to see the judge order repairs.312 Their assistance is “associated with statistically significant differences in case outcomes”: zero percent of the tenants assisted by University Settlement navigators were evicted from their homes, compared to one formal eviction for about every nine non-payment cases filed in the city.313
• The tenants who received assistance from the Housing Court Answers navigators program raised significantly more defenses in their answers than unassisted tenants and were more likely to have their defenses recognized by the court. They were also more likely than unassisted tenants to see the judge order repairs.314
• The cases that received assistance from the court-operated Access to Justice navigators were significantly more likely to end with formal judgments than cases that were not assisted, but otherwise the navigator program did not have an impact on case outcomes.315 (As noted above, this program was not intended to affect case outcomes.)

A recent US study conducted by Mary McClymont for The Justice Lab at Georgetown Law Centre surveyed 23 navigator programs run out of state courts.316 The survey was conducted through outreach and interviews with more than 60 informants involved in the creation, oversight, or management of the navigator programs. The report defines navigators as people with legal credentials or training who assist people who are not represented by lawyers with their civil law problems. The navigators do not operate as part of a lawyer-client relationship and are not part of a formal program or institution that provides specialized training.

About half of the navigator programs were overseen by not-for-profit organizations. Unlike the navigation services provided by University Settlement in the New York City navigators study, the navigator programs surveyed by McClymont did not extend beyond same-day assistance.

The purpose of McClymont’s survey was to canvass the features of the range of programs, including types of cases handled by the programs, the range of tasks performed by the navigators in the various programs, the topics and methods of training, the range or extent of supervision, and the availability of back up and support. Although the survey was not intended to look at the outcomes or impacts of the various programs, it does provide data about the informants’ perceptions of impact, and weaves in data from several user satisfaction surveys conducted by a few of the programs.

The survey indicates that:

• navigators help to improve the accuracy and completion of forms;317
• people felt that they were better prepared on what to do next, and their anxiety was reduced, after receiving assistance from navigators;318 and
• people appreciated that speaking with a navigator enabled them to tell their stories and to be heard.319

The report notes that the leaders of the navigator programs are well aware of the admonition against navigators providing legal advice and indicates that the navigators are carrying out their roles without running afoul of the rules against the unauthorized practice of law. Like Sandefur and Clarke’s study of navigators in New York City, the study raises a strong concern about the longer-term institutionalization and sustainability of many of the navigator programs.320

Canadian evaluation research

Canada does not have a body of academic literature that evaluates the quality or impact of law-related assistance provided by staff or navigators at not-for-profit community-based organizations.321 Some academic research, though, speaks to the effectiveness of such services.

For example, research conducted by Suzanne Bouclin, at the University of Ottawa Faculty of Law, describes the success of Ottawa’s Ticket Defence Program (TDP) in assisting street-involved people.322 Using a case study approach, the study discusses the TDP – a program that, between 2003 and 2007, saw trained and mentored frontline volunteers provide law-related assistance to street-involved people charged with minor provincial or municipal offences.

The TDP liaisons and agents provided information and resources to enable street-involved people to defend tickets themselves; referred them to other programs and to “sympathetic lawyers” when a street-involved person had a legal question beyond the scope of the TDP’s mandate or the agent’s training; and helped street-involved people navigate the process for responding to or defending their receipt of a ticket. They assisted with first appearances and, in a minority of cases, went to trial, working with lawyers and law students to develop legal arguments.

The report describes the “high rate of success” of the TDP in having tickets dismissed or charges withdrawn. The TDP was also successful in advocating for changes in the law’s treatment of “panhandling”, and in discouraging the issuance of tickets for passive panhandling.323 Despite its successes, the TDP was forced to close its doors because the TDP liaisons and agents were not licensed as lawyers or paralegals – even though, according to the research, for a number of years, street-involved people in Ottawa could not “afford paralegals in any event”,324 and legal aid and pro bono legal services were generally not available.325

Evaluation of legal services provided by non-lawyer licensed legal professionals

In what follows, we canvass a small number of reports that consider the quality and effectiveness of services provided by paralegals or legal technicians, authorized in Canada and in the US to provide defined services in limited areas.

Canadian evaluation research

As noted in Section 4 of this report, Ontario’s rules permit paralegals to provide services, without requiring supervision by a lawyer, in certain areas of law, for defined activities, and before specific adjudicative bodies. In 2012, the LSO retained Strategic Communications to research and report on Ontario’s experience with paralegals over the first five years in which they were operating pursuant to the LSO’s paralegal licensing regime. Based on results from a survey of 1001 paralegal user respondents, the research found that there was a “general degree” of satisfaction with the paralegal services.326

A recent Canadian study by Lisa Trabucco explores the reality of the provision of legal services by non-lawyers in Canada and indicates that non-lawyers already provide a range of legal services in a variety of settings. Trabucco reviews studies that consider the effectiveness of legal services provided by non-lawyers, including paralegals in Ontario and licensed legal technicians in US states, and looks at available information relating to misconduct or sanctions against non-lawyer representatives. Based on data from the LSO, covering about two and a half years, she finds that,

The number of paralegal professional misconduct matters compared to lawyer misconduct matters at hearings was proportionately no greater than the number of paralegal licensees relative to lawyer licensees. Further, the range of matters was similar for both lawyers and paralegals. This suggests that the professionalism and competence of regulated paralegals is about equal to, and certainly no less than, that of lawyers, or at least that paralegal conduct is the subject of disciplinary hearings to roughly the same extent that lawyer conduct is.327

Other research focuses less on the quality of services provided by paralegals and more on their impact on the ability of people to access trained legal assistance. One case study328 on the prevalence of paralegals in Ontario’s Landlord and Tenant Board proceedings indicates that the use of paralegals has increased, but only as substitutes for experienced non-lawyer representatives on the landlord side of proceedings; the cost point for paralegal services is not low enough to be accessible for many if not most tenants.

US evaluation research

As noted in Section 4, a few US states have begun to establish programs that authorize Licensed Limited Legal Technicians (LLLTs) to provide limited legal services in specified areas of law. One of the first programs, established by the Washington Supreme Court in 2012, enabled LLLTs to provide services relating to family law. In mid-2020, the Washington Supreme Court decided to sunset the program, citing the costs of sustaining the program and the small number of program applicants.329

A couple of years earlier, Rebecca Sandefur and Thomas Clarke conducted a preliminary evaluation of the program. According to the 2017 evaluation, clients reported that LLLTs were able to provide competent assistance, and that using the services of an LLLT improved their legal outcomes.330

Other studies relating to lawyers and non-lawyers, including “where lawyers matter”

US and UK research

Several US studies in the last 10 years have reviewed the evidence about the effectiveness of legal services provided by lawyers, particularly in comparison with non-lawyers. Many of these studies include a discussion of “where lawyers matter”, drawing from data that compares outcomes achieved by legal services provided by lawyers and non-lawyers.

In research published very recently,331 Rebecca Sandefur reviews the empirical evidence on three questions relevant to the consideration of the provision of legal advice by non-lawyers:

• evidence relating to the consumer demand for legal advice;
• evidence relating to the quality of legal advice provided by non-lawyers; and
• evidence of harms that result from the current restrictions on the provision of legal advice by non-lawyers.332

Although Sandefur’s review does not specifically consider legal advice provided by staff working at community-based not-for-profit organizations, her review and analysis of the empirical evidence relating to the quality of legal advice provided by non-lawyers is relevant. On this question, she summarizes the evidence as follows:

The available evidence reveals that non-lawyers can provide competent and effective legal advice. Non-lawyer sources of assistance make mistakes, but when working on appropriate matters usually make no more mistakes than lawyers do. As this research review shows, non-lawyers do a great deal of effective and high-quality work […]. At the same time, non-lawyer assistance is not always sufficient. In some situations that raise complex issues of law, non-lawyers as currently trained may not be fully equipped to provide adequate service.333

The research and evidence reviewed by Sandefur spanned several indicators of the quality of non-lawyer legal advice, including consumer satisfaction, consumer complaints, case outcomes, and expert review. With respect to studies based on consumer satisfaction, Sandefur finds that “non-lawyers are highly rated, sometimes even more highly rated than attorneys”.334 Studies on consumer complaints were the source for her finding, mentioned in the quotation above, as to equivalency in frequency of mistakes between non-lawyers and lawyers.335 In terms of the body of research on case outcomes, while noting that lawyers may be more appropriate for contexts of legal complexity, Sandefur finds that:

If the measure is prevailing in some kind of case before a court or hearing body, the general finding is that non-lawyer advocates perform as well or better than lawyers when non-lawyers are specialized and experienced.336

Over recent years, several studies have analyzed the impact of lawyers by employing randomized control trials. Russell Engler, in “When Does Representation Matter?” reviews and examines a number of empirical studies relating to the impact of a lawyer on a client’s outcomes, in a number of different forums.337

In his review of the body of research, Engler points out that several factors affect the outcome of cases beyond whether a party is represented.338 Those factors include the quality of the representatives:

Where the representation is provided by lay advocates, including law students, the effectiveness of the representation will turn on whether the representatives have received specialized training for advocacy in the particular context.339

Rebecca Sandefur also conducted a meta-analysis of studies that investigate research relating to the impact of substantive and procedural law on case outcomes in adjudicative proceedings.340 Here she finds that people who are represented by lawyers at an adjudicative proceeding are more likely, on average, to win than those who are unrepresented. But she also observes, “how much more likely varies … widely across different kinds of civil justice problems and different studies of lawyers’ impact” and concludes that the one factor that seems to affect the variation in lawyers' impact is procedural complexity, including the complexity of documents and procedures.341

An earlier study in the UK – referenced frequently by Sandefur and others – looked at the will-writing services provided by solicitors as well as by will-writing companies. The 2011 UK study found that, although there were high overall levels of customer satisfaction of services provided by will-writing companies as well as by solicitors, there was also much “evidence of consumer detriment”.342 The poor quality of wills was of particular concern due to the potential harm this can cause. And the study found that, with respect to this concern about quality, “the same proportion of wills prepared by solicitors and will-writing companies were failed”,343 and the wills were almost as “likely to fail when the client had simple or complex circumstances”.344 Key problems included, “cutting and pasting of precedents; unnecessary complexity; and use of outdated technology”.345

Also worthy of note is an earlier landmark study conducted in the US context that is referenced in almost all subsequent literature that considers the quality of non-lawyers’ services. In 1998, in “Legal Advocacy: Lawyers and Nonlawyers at Work”, Herbert Kritzer analyzed and compared the differences in results achieved by lawyer and non-lawyer advocates in the US (where advocates are representing people before a tribunal, commission, or arbitrators) in four different venues.346 Drawing from his investigations, Kritzer found that expertise is a critical factor, and that formal legal training does not necessarily denote expertise. Instead, the main difference with respect to effectiveness was that advocates with specialized technical expertise tended to bring crucial insider knowledge to support presenting a client’s case.347

Kritzer also concluded that formal legal training played less of a role than substantial experience with the setting, and that lawyers could move across settings, including where the settings differed both substantively and procedurally.348

Canadian research

Canadian research published in 2011 examined the role of legal counsel in the outcomes of refugee determination cases.349 This research, conducted by Sean Rehaag at Osgoode Hall Law School of York University, was based on a review of over 70,000 refugee decisions from 2005 to 2009.350 The study compared outcomes in cases in which refugee claimants were represented by lawyers, immigration consultants, or pro bono representatives, focusing primarily on the services provided by lawyers and immigration consultants. (The study included categories for “none” or “other”, but they comprise a very small percentage of the cases.)

The study’s major finding is that “access to qualified and competent lawyers is a major factor in claimants successfully navigating the refugee determination process”;351 claimants represented by competent lawyers were much more likely to succeed and were less likely to withdraw their claims or have their claims declared abandoned than those represented by immigration consultants. The research also found that refugee claimants are successful more often when they are represented by immigration consultants than if they are unrepresented.352

In addition, the study found that claimants’ success rates were higher when they were represented by experienced counsel:

• Lawyers who represented significant numbers of claimants during the period of the study had higher success rates than lawyers who represented a smaller number.353
• Therefore, it would appear that what matters at Refugee Protection Division hearings is not merely the ability of claimants to secure lawyers to serve as counsel, but also the ability to secure lawyers experienced in refugee law.354

Community justice help – what the evaluation literature tells us: summing up

The body of academic literature – small though it is – indicates that community workers, working in a not-for-profit setting, are able to provide good quality assistance to people with their law-related problems. The studies suggest that the nature of the law-related problem – for example, its procedural complexity – and the nature of the services that are provided – for example, services provided “for the duration” – may make a difference.

In summary, the studies reviewed in this section suggest the following:

• not-for-profit community organizations, with trained specialized staff, are able to provide as high-quality law-related assistance as lawyers within the specialized scope of their organizations’ services;
• law-related assistance provided by not-for-profit community organizations, including individualized services provided one-on-one, can have a positive impact on case outcomes;
• non-lawyers who provide law-related assistance are not subject to more professional misconduct or sanctions than lawyers;
• specialization is a key ingredient of quality with respect to law-related assistance, and community workers are able to develop that specialization;
• trained frontline community workers, situated to support highly marginalized communities, and trained and mentored to ensure high-quality services, can have high rates of success in effectively dealing with their clients’ law-related problems; and
• an experienced lawyer may be important where legal matters are complex, or where “relational expertise” – familiarity with a court or tribunal setting and the personnel in the court – comes into play.

The body of evaluation literature – program evaluative literature as well as academic evaluative literature – is thin, particularly in Canada. It would be helpful to have more Canadian research in order to identify and support good practices in the provision of community justice help (including potential areas for improvement), and to demonstrate effectiveness. This recommendation is picked up in the next section.

• Previous Page
• Table of Contents
• Next Page