Annex A – Methodology

Background

The Department launched the first cycle of the CFS in 2006, which concluded in 2009. Subsequent CFS cycles include Cycle II (2009-12), Cycle III (2016-2019) and Cycle IV (2020-22). Cycle IV of the CFS was launched in October 2020 with the Tax Law Services Portfolio. Next, the survey was conducted across the Indigenous Rights and Relations Portfolio in February 2021, the Central Agencies Portfolio in June 2021, the Business and Regulatory Law Portfolio in November 2021, and finally, the Public Safety, Defence and Immigration Portfolio in May 2022.

Following the conclusion of each survey cycle, an extensive review takes place in an effort to fine-tune the CFS administration process and make improvements where possible. With the guidance of the Statistical Consultation Group at Statistics Canada, the Department has developed the CFS questionnaire and methodology for collecting client feedback on the degree to which the delivery of legal services is meeting the needs and expectations of client departments. Over the years, Statistics Canada has played an important role by reviewing and challenging the proposed approach throughout the survey design and implementation stages, vetting the analyses of survey data and the presentation of findings contained in CFS reports.

Survey Administration

For this cycle, invitations to participate in the CFS were intended for employees at the EX-minus-1 level and above in the National Capital Region (NCR) and the EX-minus-2 level and above in the regions. From across 43 client departments and agencies, 4,598 service users reported having received Justice Canada legal services in the 12 months prior to the survey.

Interpreting Results

The survey collected feedback from clients, in the form of satisfaction ratings, using a 10-point Likert scaleFootnote 16 with two anchors: not at all satisfied (1) and completely satisfied (10). Feedback was sought along three key dimensions of service quality as per the Department’s Service Standards (see Annex H): accessibility/responsiveness, timeliness, and usefulness (the latter includes legal risk management, which is presented separately in this report). Each service dimension is composed of a number of individual elements pertaining to client satisfaction, many of which relate directly to the Department’s Service Standards for legal services. Furthermore, service users were asked to rate their level of satisfaction with the overall quality of legal servicesFootnote 17 and overall considerations elements. Specifically for Service Standard elements, in addition to surveying client satisfaction, the questionnaire asked service users to rate the importance of these elements, again using a 10-point scale. Finally, service users were asked to rate their knowledge of Justice Canada’s Service Standards.

It should also be noted that the use of a weighting strategy, adopted for the first cycle of the CFS, was discontinued at the recommendation of Statistics Canada prior to the start of the second cycle. Consequently, any references to previous survey results for elements of satisfaction now refer to comparable unweighted data. Due to this change in methodology, some results may differ from previously circulated values featured in the Cycle I results report.

Margins of error are presented in this report in the form of rating (± margin of error). The magnitude of the margin of error is generally affected by the extent of variability in respondent feedback, the overall size of the respondent group and the confidence level chosen by the survey team. This range of values is called the confidence interval and for the purpose of the CFS, a 95% confidence intervalFootnote 18 is used. As an example, in this report, overall quality of Litigation Services results are presented as 8.5 (±0.1), which implies that the 95% confidence interval for the mean rating of the overall quality of Litigation Services obtained from this survey is from 8.4 to 8.6.

The CFS used a targeted census approachFootnote 19 in which invitations to participate in the survey were sent to all potential users of legal services at the EX-minus-1 level and above in the NCR and the EX-minus-2 level and above in the regions. This approach was chosen because departmental rosters are limited in identifying actual users of legal services and potential sources of error associated with sampling can be avoided. With a census approach, margins of error account for variability related to non-response to the invitation to complete the questionnaire. That is, the respondents to the CFS were treated like a random sample from all potential legal services users, assuming that the respondents were representative of the population of interest. Had all potential users responded to the survey, there would have been no variability and the margins of error would have all been zero, as all ratings/perceptions would have been accounted for. The Finite Population Correction (FPC) FactorFootnote 20 was also applied as part of the calculation of margins of error in order to take into account the number of potential users and number of survey respondents. In the absence of the FPC, the margins of error would be overstated when survey respondents comprised more than 5.0% of the potential users.

In order to compare ratings between current and past surveys as well as various categories of service users, two independent sample t-tests were undertaken. All t-tests conducted were based on the null hypothesis of equality of two mean ratings against the alternative hypothesis of mean ratings not being equal. In other words, by identifying which of the two hypotheses one fails to reject, one is able to determine whether the difference between the two mean ratings is statistically significant or not.