Department of Justice Canada Client Feedback Survey
Survey Results – Cycle III (2016-2019)

Full PDF Version

Annex A – Methodology

Introduction

With the support of Statistics Canada, Justice Canada developed a standardized questionnaire and methodology for collecting client feedback on the degree to which the delivery of legal services is meeting the needs and expectations of client departments and users of legal services. Statistics Canada played an important role by reviewing and challenging the proposed approach throughout the survey design and implementation stages, vetting the analyses of survey data and reviewing and commenting upon the presentation of findings contained in this report.

Survey Administration

The Department launched this cycle of the survey in November 2016 with the Aboriginal Affairs Portfolio (AAP). Next, the survey was conducted with the Central Agencies Portfolio (CAP) in September 2017, the Tax Law Services Portfolio (TLS) in February 2018 and then the Public Safety, Defence and Immigration Portfolio (PSDI) in October 2018. The survey of the Business and Regulatory Law Portfolio (BRLP) concluded in June 2019. In addition, the Department obtains feedback specific to the National Litigation Sector as well as Public Law and Legislative Services Sector.

Potential legal service users received invitations to complete a standardized questionnaire designed to collect data pertaining to the Legal Advisory, Litigation, Legislative Drafting, and Regulatory Drafting Services provided by the Department in the 12 months prior to having received the survey. In an effort to extend the response from clients to potential lower level users of legal services, and subsequently gain a more accurate portrayal of client perceptions of legal services, invitations to complete the questionnaire for this cycle have been extended down one full level to now include employees at EX-minus-2 level and above in the NCR and EX-minus-3 level and above in the Regions.

The Department uses a census approachFootnote 13 for the CFS and administers the survey via a web-based questionnaire. In total, 53,230 invitations to complete the questionnaire were sent to potential users of legal services across all five Portfolios. Of this population, 17,729 respondents completed the questionnaire. In turn, 5,545 respondents reported having used departmental legal services in the twelve months preceding the administration of the survey. Unless otherwise noted, all of the results presented in this report are based on the feedback from these (or a subset of these) 5,545 legal services users.

Interpreting Results

The survey collected feedback from clients using a 10-point Likert scaleFootnote 14 with two anchors: not at all satisfied (1) and completely satisfied (10). Feedback was sought along three key dimensions of service quality as dictated by Justice Canada’s Service Standards (see Annex L): accessibility/responsiveness, usefulness (this includes legal risk management, which is evaluated separately in this report), and timeliness. Each service dimension is composed of a number of individual elements/questions pertaining to client satisfaction, many of which relate directly to the Department’s Service Standards for legal services. Further to this, respondents were asked to rate their level of satisfaction with the overall quality of legal services.Footnote 15 In addition the questionnaire asked respondents to rate the importance of these particular elements, again using a 10-point scale.

Margins of Error

In reviewing the results presented throughout this report, it is important to remember that survey results represent estimates of client perceptions of service delivery by the Department. As such, there is an important construct to bear in mind, namely the calculated margin of error. Margin of error traditionally reflects the sample-to-sample variability in the use of a sampling methodology. The magnitude of the margin of error is generally affected by the extent of variability in respondent feedback, the overall size of the respondent group and the confidence level chosen by the survey team. For the purposes of the CFS, a 95% confidence level was adopted.Footnote 16

The CFS used a census approach in which invitations to participate in the survey were sent to all potential users of legal services. Margins of error account for variability related to non-response to the invitation to complete the questionnaire – that is, the respondents to the CFS are treated like a random sample from all potential users of legal services, assuming that the respondents are representative of the population of interest, which is all potential users of the legal services. Had all service users responded to the survey, there would have been no variability and the margins of error would have all been zero, as all opinions would have been accounted for. The Finite Population Correction FactorFootnote 17 has been applied as part of the calculation of margins of error in order to take the size of the total number of potential users into account; otherwise, the margins of error would be overstated.