Development of An Access to Justice Index for Federal Administrative Bodies
3.0 Collecting the Data
Upon completion of a set of questions with weights for each of the four categories, the Questionnaire was edited for accessible or plain language, but with an understanding that the intended audience were the administrative bodies themselves. The final questions with their respective weights can be found in Section 3 of this report.
The Questionnaire was programmed into FluidSurveys and sent to the participating administrative bodies for completion where the project lead coordinated responses from different work units as appropriate. Respondents were instructed to answer the questions from the perspectives of the parties. All questions were mandatory. Space was provided for comments after each question so that respondents could clarify an answer to indicate if they believed that the question was not applicable to their body. During the analysis of the data, any such comments were taken into consideration and will be considered in the final analysis of the pilot project before moving forward. All questions were deemed applicable for both administrative bodies.
3.1 Calculating the Index
Each question in the Questionnaire is weighted using the discrete values of 10, 5, and 1. This weighting approach, with three basic values signaling contrasting levels of importance, was selected for ease of understanding and usability.Footnote 65 The weights are assigned based on the relative importance of the question to the overall category, as well as the number of questions relating to a particular issue. These three values were chosen over a sliding scale to ensure adequate differentiation. A “yes” response to a question received the total points possible, while a “no” response received zero points.Footnote 66
A standardized score out of 100 was calculated for each category by dividing the score that the administrative body achieved by the total score possible and then multiplying this number by 100.
To produce the Composite Index, each of the index scores for the four categories are added together and then divided by four, according an equal weight of 25% to each category’s set of indicators. Table 5 below shows the calculations. The higher the score, the better the administrative body does on facilitating access to justice for parties. A perfect score is 100. For this pilot project, no “pass” or “fail” score was identified; nor was a minimum score identified on any of the sub-categories. Rather, the score provides a baseline measurement and a way to document change and improvement.
Table 5 below shows the number of questions in each sub-category, the total weight accorded to each sub-category and the calculation of Index category scores, as well as the calculation of the Composite Index. In the Index Calculation column, the weighted score earned by each administrative body is represented by the number of the sub-category (e.g. 1a, 1b, 2a, 2b, etc.).
| Composite Access to Justice Index Score | |||
|---|---|---|---|
| Sub-category | # of Questions | Total Weight Possible | Index Calculation (weighted score/total weight)*100 |
| Category 1 – Access to the Administrative Body | |||
| 1a Physical access | 14 |
52 |
1a/52 * 100 = INDEX1a |
| 1b Access through Technology | 16 |
87 |
1b/87 * 100 = INDEX1b |
| Totals Category 1 | 30 |
139 |
(1a + 1b)/139 * 100 = INDEX1 |
| Category 2 – Processes | |||
| 2a Procedural | 20 |
105 |
2a/105 * 100 = INDEX2a |
| 2b Representation | 8 |
41 |
2b/41 * 100 = INDEX2b |
| 2c Interpersonal | 5 |
31 |
2c/31 * 100 = INDEX2c |
| 2d Informational | 20 |
80 |
2d/80 * 100 = INDEX2d |
| Totals Category 2 | 53 |
257 |
(2a+2b+2c+2d)/257 * 100 = INDEX2 |
| Category 3 – Costs | |||
| 3a Service Charges | 11 |
70 |
3a/70 * 100 = INDEX3a |
| 3b Intangible Cost Supports | 3 |
7 |
3b/7 * 100 = INDEX3b |
| Totals Category 3 | 14 |
77 |
(3a+3b )/77 * 100 = INDEX3 |
| Category 4 – Outcomes | |||
| 4a Distributive justice | 8 |
24 |
4a/24 * 100 = INDEX4a |
| 4b Functionality | 5 |
25 |
4b/25 * 100 = INDEX4b |
| 4c Transparency of outcomes | 4 |
20 |
4c/20 * 100 = INDEX4c |
| Totals Category 4 | 17 |
69 |
(4a+4b+4c)/69 * 100 = INDEX4 |
| Access to Justice Composite Index |
|
|
(INDEX1 + INDEX2 + INDEX3 + INDEX4) /4 * 100 = COMPOSITEINDEX |
3.2 Limitations
The indicators in the Index together form a framework on what access to justice in the administrative law context can look like. The Index does not pretend to fully define what access to justice can mean to parties, nor is it the only access to justice framework possible.
The data collection was completed by the administrative body itself. There may be variation across the country, between regional offices in terms of services available to parties (e.g. physical access to Commission offices) or staff (e.g. access to training opportunities). It is hoped that discussion around the results of this exercise will stimulate additional work to understand access to justice in its own specific context, including implementation of many of the items included in the Index and additional follow-up on other items. As results from these studies become available, they will be incorporated into future iterations of the Index to make it an even stronger tool for change.
The Index and its Questionnaire were developed with consultation from at least four federal administrative bodies, academics and several other organizations. It is recognized that the Index was only piloted with two administrative bodies, the CHRC and the Competition Tribunal, and that these two bodies are quite distinct in terms of mandate, and the parties they serve. It is hoped that additional federal administrative bodies will be interested in participating in the Access to Justice Index project enabling more feedback on the value and utility of this tool.
The difference between the mandate of the CHRC, and consequently its structure, and other tribunals may render cross-tribunal comparisons slightly more difficult. The Index was designed to accommodate these differences with the goal of making such comparisons possible. Though the CHRC does not presently hold hearings, it does nevertheless interact with parties to a complaint. The principled expectations that a party may have at a hearing should also apply in other contexts in which there is direct interaction. Lower scores of certain type of tribunals over others may be more a reflection of legislative provisions rather than administrative practices.
Legislative provisions may inherently limit access. In such circumstances, a tribunal may only be able to get a certain score. For example, an administrative body that is limited by its legislation in the type of remedies it can provide, may only score 70. From a scoring perspective, this may equate to a “B-“; whereas, from an operational point of view, this may be considered an “A+”. This may ultimately diminish the comparative value of the Index. Ultimately, however, the Index is not intended to be used without the specific context of each administrative body clarified.
By negotiating the scores, a regression toward the means may have occurred resulting in the majority of items being identified as having a weight of 5. Greater variation in the weights may be desirable. A table showing the number of questions at 1, 5, and 10 may help illustrate the above point. It may also be important to note that as mandatory service standards come into force, the weight of various questions may change. Finally, it may be worth validating the questions among actual users/parties of administrative bodies.
- Date modified: