Access to Justice in Both Official Languages Support Fund
4. Evaluation Findings (cont'd)
4.3. Project Funding
The third component of the Support Fund is project funding, the aim of which is to raise awareness in the official language communities or increase an organization's capacity to meet community needs in relation to access to justice in both official languages. Using an effective selection process that the recipient organizations clearly understand, the Department is able to fund a number of projects relating to Support Fund objectives. These findings are described in greater detail below.
Profile of Projects Funded and Activities Carried Out
Since the Support Fund was created in 2003, the Department of Justice has received 147 applications for project funding. To date, 41 of those files are still active and 106 have been concluded (whether or not they received funding).
Table 7 provides an overview of the applications for funding of projects that have been funded by the Support Fund.
Table 7: Overview of Projects Funded by the Support Fund (n=106)
Projects funded to date include workshops for lawyers on language rights, development of precedents for pleadings, legal terminology workshops, awareness-raising programs for young people about the legal profession, and online databases of lawyers who practice in the minority language.
The Department of Justice encourages recipients to seek financial or other support for their projects from additional sources; 48% of the projects received financial support from another source, and 42% received non-financial support. The type of non-financial support that recipients obtained most often involved human resources for carrying out their activities (23%) or equipment loans/room rentals (7%). Table 8 identifies the most common additional sources of non-financial and financial support.
Table 8: Sources of Non-financial and Financial Support
| Percentage of Projects | … that received non-financial support from a: |
|---|---|
| 55% | Community organization |
| 32% | Educational institution |
| 25% | Provincial government |
| 2% | Other federal department or fund |
| Percentage of Projects | … that received financial support from: |
|---|---|
| 37% | Registrations/dues/sales |
| 31% | Other federal department or fund |
| 24% | Provincial government |
| 22% | Community organization |
| 14% | Educational institution |
| 2% | Private enterprise |
In general, the activities that were planned for projects funded were carried out as planned. Only 6% of agreements had to be extended and 1% cancelled. Although slight changes were sometimes made to the activities and/or timetables, those changes were generally minor and did not affect the final outcomes. In addition, recipients generally kept Support Fund officials well informed about any change made. The Department of Justice provides careful oversight for the projects, but a bond of trust has developed between recipients and Support Fund officials. Funding recipients generally feel comfortable speaking openly with the Department about their concerns or the problems they are dealing with.
Funding Application and Project Selection Process
The Department of Justice has established funding application and project selection processes for the Support Fund. Eligible organizations submit an application using the general form provided by the Department for all funding applications. That application is then reviewed by a selection committee. Overall, the stakeholders involved feel that they are provided with good information and support throughout the application and project selection processes.
The Department uses several methods to inform eligible organizations about the Support Fund. Information about selection criteria, eligibility and objectives and the process to follow for making an application is provided at meetings of the Subcommittee and the F-P-T Working Group. As well, Support Fund officials have held information sessions to present this information to various groups. The Department's Web site also contains useful information about the application process, and organizations are invited to contact Support Fund officials if they have any question about their funding application.
The Department has made a variety of sources of information about the Support Fund funding process available to potential recipients. As shown in Table 9, the recipients consulted were informed about the possibility of submitting a funding application to the Support Fund either by the Department directly or through another organization, or by a combination of sources.
| From a colleague/another organization | 38% |
|---|---|
| We received a letter/e-mail from the Department of Justice | 31% |
| Through the Department of Justice Web site | 31% |
| At meetings with/presentations given by the Department | 31% |
| At meetings of the Subcommittee or F-P-T Working Group | 15% |
| Through our own research | 8% |
| Other | 15% |
While the Support Fund's selection criteria are connected with the needs identified in the environmental scan, they are still relatively broad. On one hand, this characteristic means that the Support Fund is able to fund a range of activities dealing with the needs identified. On the other hand, it can lead to a degree of ambiguity regarding what may and may not be funded by the Support Fund. Some applicants who were consulted found it difficult to determine the extent to which their project application met the selection criteria, so they had to contact the Department for clarification. In this regard, the evaluation findings indicate that Support Fund officials were available promptly to answer questions of that nature.
In addition to the direct support provided by Support Fund officials throughout the funding application process, the Department offers a guide to preparing a funding application and a standard funding application form on its Web site. Recipients generally do not experience major difficulties in submitting their applications, and believe that the process is effective and is designed and managed in a way that meets their needs.
The funding application review process is especially expeditious. When an organization submits its funding application, the selection committee, which is composed of four members (two members of the Programs Branch and two members of the Office of Francophonie, Justice in Official Languages and Legal Dualism), reviews the application to determine whether the organization and its project are eligible for funding under the Support Fund. The selection committee meets on an as-needed basis, i.e. as applications are received. The average time an applicant can expect to wait between the date an application is submitted and the date it is reviewed by the selection committee is 15 days. Once the application has been reviewed, the selection committee informs the applicant of the decision or any clarification needed. The average time between the date when an application is reviewed by the selection committee and the date when a formal decision is given is 24 days, although in more than 40% of closed files the formal decision was given within 10 days. Often, Support Fund officials inform an applicant of the decision by telephone, even before sending a letter.
Although in some cases the selection committee has gone back to applicants to clarify some aspects of the application, that process has not resulted in any significant delay. Among closed files, clarifications were requested in 48% of cases. Of those, nearly 90% of applications were subsequently accepted. When an application is rejected, Support Fund officials always inform the applicant of the reason for the rejection. Reasons for rejection of an application include:
- the application was outside the purview of the Support Fund (85%),
- the project was a duplication of services (20%),
- the project was not under federal jurisdiction (10%), and
- there were not enough partnerships with Francophone organizations (10%).
The survey of recipients confirmed that the support offered by the Department met their needs to a large extent. As shown in Table 10, nearly all respondents had a favourable opinion of the information and support provided by the Department.
| Very satisfied | Satisfied | |
|---|---|---|
| Clarity of information about the application process | 54% | 38% |
| Support provided by the Department of Justice during the application process | 77% | 23% |
| Criteria used to determine eligibility for assistance | 62% | 38% |
| The Department's explanation of the decision | 46% | 46% |
Objectives and Relevance of Project Funding
Because the project funding selection criteria relate directly to the needs identified in the environmental scan, the projects funded are generally considered to be relevant responses to needs in relation to access to justice in both official languages. For example, the objectives of the projects funded are closely tied to the objectives of the Support Fund (see Table 11).
| Increase capacity of Department's partners | 41% |
|---|---|
| Raise awareness in official language minority communities | 21% |
| Develop partnerships | 18% |
| Improve access to justice in French | 17% |
| Raise awareness among the general public | 10% |
| Raise awareness in the legal profession | 9% |
As well, the projects funded are selected based on the extent to which they correspond to the needs identified. Table 12 identifies the needs addressed by projects funded to date.
| Legal tools and resources | 31% |
|---|---|
| Promotion/awareness-raising in communities | 26% |
| Promotion/awareness-raising in the legal profession | 19% |
| Professional development for lawyers | 15% |
| Networking among stakeholders | 10% |
| Promotion/awareness-raising among the general public | 4% |
Through the awareness-raising activities carried out, the projects funded make the issue of access to the law in both official languages more visible. The evaluation findings indicate that, to date, these activities have raised awareness among lawyers, the OLMCs and the general public, not only of language rights in relation to access to justice, but also of the importance of requesting and offering justice services in both official languages.
Activities focussing on providing training for lawyers and developing legal tools in both official languages also contribute to increasing the capacity of lawyers to offer services in the minority official language and thus increase access to justice services in both official languages.
Various organizations have participated in developing jurilinguistic tools and providing legal training, including AJEFs. The organizations most active in developing legal tools for lawyers practising in the minority language are the four jurilinguistic centres: the Centre for Legal Translation and Documentation at the University of Ottawa, the Centre de traduction et de terminologie juridiques at the Université de Moncton, the Institut Joseph-Dubuc and the Research Centre of Private and Comparative Law at McGill University.
Surveys of lawyers indicate that the jurilinguistic resources offered by the four jurilinguistic centres are not always well known or widely used in the legal community. When they are used, however, they are generally regarded as useful. If the visibility and promotion of these resources were increased, it would doubtless make it possible to address the needs of a larger number of minority official language practitioners.
As illustrated in Table 13, a significant proportion of respondents were not familiar with or did not use the jurilinguistic resources available at the three Francophone centres. Because the purpose of this evaluation is not to evaluate each project separately, we can only speculate on these trends. What it is important to note, however, is the fact that more than 9 out of 10 respondents who use the resources produced find them useful or very useful.
Table 13: Knowledge, use and usefulness of jurilinguistic resources
As indicated in Table 14, the same trends emerge among English-speaking respondents in Quebec.
Table 14: Knowledge and use of jurilinguistic resources offered by McGill University and level of usefulness of the resources
| Fairly well | 4% |
|---|---|
| Little | 30% |
| Not at all | 59% |
| Rarely | 24% |
|---|---|
| Never | 76% |
| Very useful | 50% |
|---|---|
| Useful | 33% |
| Not useful | 17% |
Source : Survey of English-speaking jurists
Training sessions funded by the Support Fund and offered by AJEFs and some of the jurilinguistic centres are generally well regarded. The training deals with a variety of subjects, including language rights, legal terminology in French, and issues relating to the practice of law (for example, dispute resolution). The lawyers surveyed reported that training dealing with legal terminology and precedents for legal instruments in the minority language is particularly relevant. Training relating to specific areas of law is also relevant, but to a lesser extent, mainly depending on the area in which the lawyers practice.
Tables 15 and 16 set out the survey data relating to the relevance of the professional development activities offered.
| Very Relevant | Relevant | |
|---|---|---|
| Use of French in criminal proceedings | 33% | 16% |
| Use of French in family proceedings | 25% | 17% |
| Use of French in civil proceedings | 40% | 24% |
| Legal terminology in French | 65% | 26% |
| Access to model legal instruments in French | 58% | 24% |
Source : Survey of French-speaking jurists
| Very Relevant | Relevant | |
|---|---|---|
| Use of English in criminal proceedings | 7% | 4% |
| Use of English in family proceedings | 7% | 7% |
| Use of English in civil proceedings | 15% | 30% |
| Legal terminology in English | 30% | 41% |
| Access to model legal instruments in English | 41% | 33% |
Source : Survey of English-speaking jurists
Overall, the evaluation findings indicate that activities funded by the Support Fund are relevant and appropriate in terms of achieving the objectives identified and, to some extent, meeting the needs identified in relation to access to justice. According to about half the French-speaking lawyers consulted in the survey, significant progress has been made in the last three years in:
- the professional development activities offered that relate to the delivery of legal services in French (46%); and
- access to legal tools in French (54%).
As well, about 50% of respondents noted that in the last three years, more of their clients had requested legal services in French.
The vast majority (85%) of English-speaking lawyers in Quebec could not answer the question about progress that had been made in the last three years in professional development activities and in access to legal tools. The reasons that might explain why progress has not been as significant, or at least has not been as visible, in the Quebec Anglophone community include:
- There seem to be few organizations working in the area of justice in English in Quebec which would suggest that offering relevant professional development activities and developing legal tools in English in Quebec is of less importance than comparable French-language activities are in Francophone communities outside Quebec.
- The vast majority of English-speaking lawyers surveyed (78%) could not identify the type of support offered by the Barreau du Québec for lawyers who practice in English in Quebec, and nearly all of the English-speaking lawyers consulted (96%) said that there were no other organizations, or they knew of no other organizations, that offered legal support for lawyers who practice in English in Quebec.
- The Support Fund does not seem to have the same level of visibility in Quebec as elsewhere in the country. This might mean that fewer professional development activities and activities relating to the development of legal tools are carried out by anglophone organizations in Quebec, because they are not using funding provided by the Support Fund. The Support Fund's lack of visibility in Quebec can be explained partially by the fact that Anglophone organizations in Quebec have not participated in large numbers in the activities of the advisory committees.
- The needs of the Anglophone legal community in Quebec might not be the same as the needs of the Francophone legal community outside Quebec. Because the political and legal realities are different from the situation for Francophone communities outside Quebec, issues relating to the practice of law in English in Quebec might also be different.
Despite the contribution of activities funded by the Support Fund to achieving the objectives and meeting the needs associated with the delivery of legal services in the minority language, the following findings highlight some of the limitations of the Support Fund:
- Needs in relation to access to justice in both official languages go beyond just criminal law and matters under federal jurisdiction. Stakeholders expected that these additional needs could be addressed under the Support Fund if the goal is to improve access to justice in both official languages in the broader sense.
- For better access to justice in both official languages to be achieved, the institutional capacity to offer those services must be improved. Since the administration of justice is a matter under provincial jurisdiction, it is expected that the provinces and territories will be more involved in this issue and will be given adequate support.
- Because new needs have emerged over the years and some of the needs identified earlier, while still relevant, may have changed, it is expected that the Support Fund will be sufficiently flexible to respond to the new demand.
Performance Measurement
Procedures have been put in place for measuring the performance of projects funded by the Support Fund, but they are still incomplete. To receiving funding from the Support Fund, an organization must show that it has an evaluation plan that includes a method of measuring the extent to which anticipated outcomes have been achieved. A majority of recipients surveyed (69%) indicated that they had a process in place for measuring the impact of the activities funded by the Support Fund. From the review of project files, it appears that performance is measured in 55% of cases, generally using the following methods:
- informal discussion/feedback from stakeholders (24%),
- survey of satisfaction among participants/users of the service (17%),
- quality of deliverable (14%),
- number of participants in an activity carried out (7%), and/or
- level of use of a product or service (6%).
Although recipients of funding under the Support Fund seem to be clear about the importance of measuring performance, the types of performance measurement used suggest that there is still some confusion regarding measurement of outcomes as compared to measurement of outputs of the activities carried out.
In addition to evaluation reports, funding recipients must submit financial reports and activity reports to the Department of Justice. More than three quarters (77%) of recipients surveyed reported that the Department's expectations regarding data collection are clear.
The data collected are generally used for evaluation and audit purposes, and for the Support Fund's future planning, that is:
- data are entered in the Grants and Contributions Information Management System (GCIMS),
- the Department makes information available about the activities carried out under the projects funded, and the outcomes achieved, on its Web site, and
- The data provide Support Fund officials with information about trends, gaps and new needs in relation to access to justice in both official languages. Support Fund officials also use the data to involve Advisory Committee and Subcommittee members in Support Fund planning discussions.
4.4. Evaluating Future Impacts of the Support Fund
This evaluation provides a partial picture of the outcomes of the Support Fund. As was noted in the introduction, the evaluation covers three of the five years of Support Fund funding and part of the fourth year. This approach was necessary so that the evaluation could meet the reporting requirements attached to the initial approval of the Support Fund. Obviously, there will be further activities initiated between now and the end of fiscal 2007-2008, the impact of which should be considered in order to obtain a complete picture of the Support Fund's achievements.
The value of measuring future impacts of the Support Fund arises from both results-based management and the more comprehensive process under way in relation to the Official Languages Program. Since introducing its management framework, Results for Canadians, in 2000, the federal government has adopted a results-based management process that necessarily involves performance measurement and evaluation processes for new initiatives such as the Support Fund. It is necessary, therefore, to determine how it will be possible to evaluate the long-term outcomes of Support Fund activities. That evaluation is also of considerable value to the Support Fund in that it coincides with an anticipated outcome of the Official Languages Program of the Government of Canada: that Canadians have "improved access to justice in both official languages."
[12] The extent to which the Department of Justice will be in a position to measure the impact of its
official languages initiatives will largely dictate the extent to which the federal government will be able, by extension, to measure the achievement of that anticipated outcome of its Official Languages Program.
This evaluation largely covers the short-term outcomes of the Support Fund and examines, in part, the anticipated outcome in the medium term. While the picture is incomplete (for the reasons stated earlier), the methodology used to conduct the evaluation can be used to measure the progress achieved to date in terms of the three short-term outcomes of the Support Fund. It is important to note, however, that this process is not intended to evaluate each project funded by the Support Fund on an individual basis. This distinction is important, and it arises in every evaluation that deals with a grants and contributions program. The Department will, of course, have to report on the projects it funds, particularly in relation to the extent to which they conform with the goals and objectives of the Support Fund. However, the Department is not expected to measure the net outcome of each activity funded for each project, out of the hundred or so projects that it has funded to date. Overall, the outcomes of the Support Fund that have been measured to date can be used to determine the extent to which the activities of the Support Fund contribute to achieving its anticipated outcome—to offer improved access to services in both official languages in the justice system—has been achieved. Those outcomes, however, cannot be used to obtain a precise measurement of the relative contribution of the Support Fund to that outcome.
Measuring the long-term outcomes of the Support Fund would require that all other initiatives (federal, provincial, municipal and community) that might contribute to them be measured, and this would be as laborious a process as it would be improbable. The two long-term outcomes identified in the Support Fund logic model specify the long-term objectives toward which the Support Fund's efforts are directed. Identifying those long-term outcomes does not automatically imply a corresponding obligation to measure the extent to which they are achieved, and then determine the relative contribution of the Support Fund to the achievement (or non-achievement) of those outcomes in the long term. In other words, if the Department does not measure the extent to which the long-term outcomes that the Support Fund wishes to contribute to are achieved, this does not, in itself, mean that they cannot be included in its logic model, since the logic model is used for purposes beyond just the evaluation process. That tool is one of the methods used to express the actual logic of a program; it is a very useful process for management purposes, and no less valid simply because not all of its components are systematically evaluated.
In practical terms, the Department must determine how the evaluation process can best support its management structure. With an initiative like the Support Fund, in which a range of stakeholders are involved, the Department of Justice has to play a relatively specific role. It must ensure that the decisions it makes regarding management of the Support Fund are consistent with the terms and conditions of the program, and thus that they support the objectives on which the Support Fund is based. In these circumstances, the distinction between what might be worthwhile to measure and what must be measured takes on particular importance. The Department of Justice is accountable for the outputs and short-term outcomes of the Support Fund, and must be able to explain how the Support Fund contributes to achieving the medium-term outcome. It is doubtful, however, that it is necessary to measure the achievement of long-term outcomes to ensure that the Support Fund is being properly managed.
Measurement of the impacts of the Support Fund is intrinsically tied to, and largely dependent on, continuous performance measurement. Even if the Department does not measure the achievement of the Support Fund's long-term outcomes, and does not evaluate each project funded, it may still consider it important to get a better understanding of how the short-term and medium-term outcomes identified in the Support Fund logic model relate to one another. The success of that analysis depends largely on the quality of the data collected through the continuous performance measurement. This evaluation indicates that Support Fund managers and funding recipients understand the importance of results-based management and the efforts that must be made to document the impact of Support Fund activities.
To date, evaluation of the Support Fund has depended largely on collecting new data as part of formal evaluation processes (e.g. document review, interviews and surveys), with the limitations inherent in that process. While it is sometimes thought that it is "too soon" to evaluate the impact of activities such as those undertaken by the Support Fund, it is important to remember that the mere passage of time does not solve any of the problems associated with measuring results. In fact, the passage of time very often only accentuates the problems. It would therefore be prudent for the Department of Justice to specify the anticipated outcomes of the Support Fund that it wishes to document and measure, and adjust its reporting requirements accordingly.
[12] Government of Canada. Canada's Linguistic Duality: A Framework to Manage the Official Languages Program. (2005) Ottawa, p. 12.
- Date modified: