3. Evaluation Methodology

The evaluation was guided by an evaluation matrix (evaluation questions, indicators, and data sources), which was developed through the evaluation scoping and design process. Appendix A contains the list of evaluation questions.

An Evaluation Working Group (EWG) composed of Justice Canada representatives from the LSB and the Finance LSU provided advice during the design and implementation of the evaluation. The EWG coordinated access to documents and data, and provided feedback on evaluation products (i.e., the evaluation questions and indicators, preliminary findings and the draft evaluation report).

Multiple lines of evidence were used to address the evaluation questions: a process mapping exercise; a review of documents; a review of administrative data; key informant interviews; and two electronic surveys.

3.1 Process Mapping

To ensure that the evaluation was based on an accurate understanding of the design and delivery of legislative services, process mapping sessions were held as part of the methodology design process. These sessions involved selected representatives from the LSB sections and from the Finance LSU.

The goal of these sessions was to gain a better understanding of the workflows and processes of legislative services. Draft process maps and associated descriptions were developed based on perspectives shared during the sessions and on currently available documentation. Once finalized, these process maps and descriptions were used to inform all other data collection methods.

3.2 Document Review

The document review was initiated during the design phase of the evaluation, and continued throughout the evaluation process, as additional information became available. It provided descriptive information on the various processes and tools used by the LSB and Finance LSU to deliver their services. The range of documents consulted included:

• Program documents: Key background documents provided by the LSB and the Finance LSU, such as drafting guides and reference material (e.g., project handbook, orientation guide, description of roles and responsibilities, and internal process flowcharts, directives and policies, etc.);
• Publicly available departmental and other government documents: Departmental documents were reviewed (e.g., last evaluation of Legislative Services (2013), and audit reports (2012 and 2017)), along with relevant documents on federal priorities (e.g., Departmental Plans, mandate letters, etc.);
• Survey: Justice Canada’s Client Feedback Survey is administered by the Corporate Planning, Reporting, and Risk Division as part of its overall performance management agenda. The purpose of the Survey was to obtain feedback on the degree to which Justice Canada legal services respond to the needs of client departments and agencies. Qualitative feedback from Cycle IV (2019-20 to 2022-23) were available for legislation and regulatory services.

3.3 Administrative Data Review

The administrative data review included information obtained from Justice Canada’s Departmental Business Analytics System (i.e., Explore). Data was extracted from Explore’s Data Warehouse via Tableau, which included data from iCase, LEX, the Integrated Financial and Material System (IFMS), and PeopleSoft. iCase and LEX data were extracted during July 2022. The data review focused on files to which LSB and Finance LSU timekeepers recorded time between FY 2017-28 and FY 2021-22. The data review considered the number of hours that LSB timekeepers recorded by file type, client name, activity (i.e., professional development), and risk and complexity rating. For Finance LSU, the data review considered the number of hours that Finance LSU timekeepers recorded by file type, client name, and risk and complexity rating.^{Footnote 10}

Administrative data was reviewed from the Justice Laws Website between FY 2019-20 to FY 2021-22. The data review focused on website usage, including the number of visits and time spent per page in a fiscal year.

3.4 Key Informant Interviews

Semi-structured interviews with key informants contributed to the in-depth understanding of legislative services, and the contribution of the LSB and the Finance LSU to that process. A total of 51 interviews were conducted for LSB and Finance LSU involving 109 individuals. Of those, 20 were individual interviews, while 31 were group interviews. A brief description of interviews conducted is provided below and additional details can be found in Appendix B. The following groups were consulted through these interviews:

• Legislation Section;
• Regulations Sections;
• Specialized Legislative Services Section;
• Finance LSU;
• Legal counsel from LSUs who have participated in the legislative process;
• Representatives from sponsoring departments and agencies;
• Other representatives from Justice Canada; and,
• Canada Gazette.

3.5 Electronic Surveys

Two electronic surveys were administered as part of the evaluation.^{Footnote 11}

• A survey involving middle manager and junior level counsel from the LSB; and,
• A survey of representatives from sponsoring departments and agencies who have received legislative services within the previous year. For the purpose of the survey, this group also included legal counsel from LSUs assigned to these sponsoring departments and agencies.

For each of the two surveys, a questionnaire was developed in both English and French, based on the relevant evaluation issues and questions. It included mainly closed-ended questions, with a few open-ended questions.

The survey involving representatives from the LSB proceeded with a sample of 163 individuals who received an invitation to participate. A total of 107 individuals completed the survey questionnaire, for a response rate of 66%.

The survey involving representatives from sponsoring departments and agencies proceeded with a sample of 80 individuals who received an invitation to participate. A total of 47 individuals completed the survey questionnaire, for a response rate of 59%. This included 25 respondents from sponsoring departments and agencies, 21 respondents from LSUs, and one respondent from a sector of Justice Canada.

3.6 Consideration of GBA Plus and Diversity and Inclusion

In accordance with the Policy on Results, the evaluation included considerations related to Gender-based Analysis (GBA) Plus. In particular, the evaluation issues and questions, and the data collection process, explored issues related to the accessibility of federal legislation, covering both the approach to drafting legislative text (vocabulary used) and the accessibility of the Justice Laws Website.

3.7 Constraints, Limitations and Mitigation

Table 2 describes the main limitations related to the proposed methodology, along with the mitigation strategies that were implemented. Overall, the evaluation process did not encounter constraints or limitations that prevented its ability to adequately address all evaluation issues and questions.