PACP Committee Report

Departmental Performance Reports (DPRs) provide a mechanism through which federal departments and agencies are held to account by Parliament. The financial and non-financial performance information in DPRs is intended to support parliamentary consideration of spending plans and the approval of annual appropriations in the Main Estimates. DPRs also support government accountability more generally since they show how departmental activities make a difference to citizens. They provide Canadians with key information about how departments and agencies performed against the expected results and plans set out in their Reports on Plans and Priorities (RPPs).

Improving the overall quality of DPRs continues to be a priority for the Government. In 2007, the Treasury Board Secretariat implemented a strategy to improve the content and form of these documents. The strategy was developed in response to comments made by parliamentarians and the Office of the Auditor General (OAG), and was designed to ensure a more concise, results-focused approach to reporting. As part of the strategy, the Secretariat augmented its annual outreach efforts with federal departments to ensure adherence to the new approach. The Secretariat also began producing a Good Practices Handbook for performance reporting, a key reference guide for departments and agencies in developing a balanced and credible picture of performance.

The Treasury Board Secretariat assesses DPRs annually against commonly accepted public reporting principles through the Management Accountability Framework (MAF) process. Since 2007, MAF results have demonstrated steady progress in the quality of departmental reporting.¹  Departments and agencies have indicated that the new concise, results-focused approach has allowed them to become more effective in telling their performance stories. Library of Parliament researchers, Committee clerks and the OAG have indicated that the new approach is a step in the right direction. 

The Secretariat recently commissioned the CCAF-FCVI Inc. (formerly known as the Canadian Comprehensive Auditing Foundation) to conduct a study of performance reporting practices across a variety of jurisdictions. The study (released in March 2010) used the following criteria: reliability, validity, comparability, consistency, user-friendly reporting (understandability), relevance and innovation. It ranked Canada second of the five reporting jurisdictions examined, which included the United Kingdom, the United States, Australia and New Zealand.

On September 20, 2010, the Standing Committee on Public Accounts tabled Report 15 entitled Selected Departmental Performance Reports for 2008-2009 – Department of Industry, Department of Transport. As required, the Government has developed a comprehensive response to the seven recommendations made in Report 15. This response was developed collaboratively by the Treasury Board Secretariat, Industry Canada, Transport Canada and the Privy Council Office.

RECOMMENDATIONS 1 AND 2:

Recommendation 1

That Industry Canada and Transport Canada set clear and concrete performance targets that are directly related to the expected performance of their programs

Recommendation 2

That Industry Canada and Transport Canada clearly demonstrate in their Departmental Performance Reports the link between departmental activities, expected results, and actual performance.

The recommendations are accepted.

The Treasury Board Secretariat’s Policy on Management, Resources and Results Structures (MRRS) supports the development of a common, government-wide approach to the collection, management, and reporting of financial information and non-financial information on program objectives, performance and results. Since the Policy on MRRS’s implementation in 2005, departments and agencies have worked with the Secretariat to develop and maintain a current inventory of their programs, demonstrate how these programs are aligned to each other and to strategic outcomes through Program Activity Architectures (PAAs), provide program expenditures according to the PAA, and track and report program performance against concrete performance targets according to this structure through Performance Measurement Frameworks (PMFs). Departments and agencies continue to improve and stabilize their PAAs and PMFs as a consistent basis for reporting performance in their DPRs. Over time, this will facilitate trend analysis and benchmarking.

Industry Canada and Transport Canada recognize the importance of using clear performance measures, identifying targets and reporting performance against expected results in their DPRs, as defined by the Policy on MRRS. However, as with other organizations, their PAAs and PMFs continue to mature and stabilize. Furthermore, some departments and agencies have experienced delays in complying with certain aspects of the policy – for example, in defining outcomes that can reasonably be attributed to a particular program. The Secretariat continues to provide leadership, guidance and outreach to departments and agencies on MRRS implementation to support departments and agencies in addressing these challenges.

A summary of improvements made by Industry Canada and Transport Canada in this area since 2008-09, as well as plans for future progress, is provided below.

Industry Canada

Industry Canada’s 2009-10 RPP was the department’s first report to contain targets for every strategic outcome and the outcome of every program activity. The corresponding 2009-10 DPR reported on the degree to which these targets were met. As an example of how Industry Canada set clear and concrete targets directly related to expected results, the program activity “Competition Law Enforcement and Advocacy” had an expected result of “Competitive markets and informed consumer choice”. One of the supporting performance indicators used was the “Percentage of economy subject to market forces”. The target identified in the 2009-10 RPP was to “Increase or maintain current percentage (approximately 82% of GDP)” and the performance status reported in the 2009-10 DPR was “Met all” with an actual result of 82%.

In the 2009-10 DPR, Industry Canada made increasing use of performance summary text and explanatory notes to make linkages between expected results and performance more explicit. It should also be noted that Industry Canada continues to endeavour to provide explanations to account for any significant variances in performance. 

Industry Canada has implemented a new review process of its Performance Management Framework (PMF) that will allow the department to better understand the linkages between elements such as departmental activities, actual performance, expected results and targets. This process is expected to form the basis for ongoing improvements. The review process involves completion of templates which require that the reason for choosing a specific target be articulated. As well, the templates used in this process result in an explicit understanding of variances in indicator results as well as the methodology used to measure the indicator. This and other information required for the templates will lead to a better understanding in the department of the strengths and weaknesses of each element, as well as of the linkages between elements and of other external factors that may impact results. 

Transport Canada

In 2007-08, Transport Canada’s PAA was redesigned. The adoption of a clear, mandate-driven PAA has facilitated the work on the department’s PMF and, as such, will assist with the adoption of a set of clear and concrete performance indicators and targets.

Since the redesign of the PAA, Transport Canada has made ongoing improvements to its PMF to address the requirements of the Policy on MRRS, the feedback received from the Treasury Board Secretariat through the MAF process, and the recommendations of various internal and external reviews. In December 2009, Transport Canada undertook an extensive review of the departmental strategic outcome level and program activity level performance indicators. This exercise resulted in more meaningful, specific and measurable measures for the department.²  Ongoing refinements to the performance indicators at lower levels of the PAA will continue in 2010-11.³

Transport Canada’s Enterprise Performance Framework (EPF), which is expected to be completed in 2011-12, provides a means to standardize the approach for performance management (i.e., performance indicators, performance targets, performance accountability and performance monitoring and reporting) across the organization. It addresses both internal and external planning and performance management requirements of Transport Canada in a consistent manner to ensure the coordinated and effective use of performance information to achieve expected results. The resulting performance plan will address each facet of management that is typically employed in business planning (Plan/Program Management, Risk Management, Project Management and Resource Management).

RECOMMENDATION 3

That Industry Canada and Transport Canada provide more detailed explanations in their departmental performance reports about the methodology used to present results.

This recommendation is accepted.

It is important for departments and agencies to provide parliamentarians with accurate and credible performance information, including information on the methodology used to present results. At the same time, DPRs should maintain a strategic focus and remain concise. Departments and agencies are encouraged to provide web links to the more detailed, technical information that is relevant to understanding performance on their websites. The Treasury Board Secretariat will therefore explore how departments and agencies could use departmental websites to present, where useful for clarity and where feasible, indicator methodologies for the 2013-14 DPR.

The perspectives of Industry Canada and Transport Canada on the inclusion of methodology in their DPRs are provided below.

Industry Canada>

As recommended by the Treasury Board Secretariat, Industry Canada has used electronic links to publicly available sources in its DPR to allow the reader to verify the data quality and to understand the methodology behind the data. 

Industry Canada began collecting methodology information on all performance statements in 2009-10 and summarized this, where applicable, in the 2009-10 DPR through text and explanatory notes. For example, for the strategic outcome, “Science and Technology, Knowledge, and Innovation are Effective Drivers of a Strong Canadian Economy”, one of the indicators in the 2009-10 DPR is Canada’s rank on the Innovation Index. Information presented explains to the reader the purpose of the index and its sub-components, and a link to the data source for further information is provided. Complete methodology information is available upon request for all performance indicators. In relation to performance status (i.e., exceeded, met all, mostly met, etc.), an explanation was provided in the “How to Read This Report” section of the 2009-10 DPR, in accordance with instructions from the Treasury Board Secretariat. 

As Industry Canada continues to improve its Parliamentary Reporting, the new templates and processes in place will ensure that the department has explicit methodologies available for each indicator before the DPR process begins. The DPR will also continue to provide readers with information on trends against historical performance to indicate whether performance has improved, declined, remained stable or was unavailable.

Transport Canada

On a department-wide basis, Transport Canada is in the process of implementing a comprehensive methodology to present results in its DPR. Components of this methodology to improve performance reporting are described below.

Based on the revised 2007-08 PAA, Transport Canada completed a review of its PMF at the strategic outcome and program activity levels during the previous fiscal year and it is currently embarking on an assessment of its PMF at the lower program levels. It is anticipated that this extensive review will result in a stable PMF for upcoming years that will facilitate benchmarking and/or the provision of baseline data. Meanwhile, the department has adopted an approach in its 2009-10 DPR whereby trend analyses are presented where data is available, and qualitative proxy measures have been included so as to provide more comprehensive and accurate results-based information.

Moreover, the recommendations, guidelines and templates provided by Treasury Board Secretariat have proved very useful in improving results-based reporting. For example, following the Standing Committee on Public Accounts’ hearing on May 13, 2010, Transport Canada decided to include a performance target, status and summary of performance for each program activity in its 2009-10 DPR. The summary also includes a brief explanation to justify the department’s performance assessments.

Regarding the methodology used in the development of performance indicators, it should be noted that many of the data sources are publicly available. For example, the average age of transportation infrastructure is available from Statistics Canada and the number of aviation accidents per 100,000 flying hours is available from the Transportation Safety Board. However, in some cases, it may be difficult to present the methodology of a certain performance indicator, or its corresponding target and result in the DPR due to the intricacy and complexity of the methodology in question.

RECOMMENDATION 4

That Industry Canada and Transport Canada provide credible and balanced performance reports by clearly indicating where program performance was less than expected, explaining why, and discussing what steps were taken to modify program design, and delivery in order to improve performance.

The recommendation is accepted.

It is important that DPRs serve as an accurate depiction of a department or agency’s successes and shortcomings over the course of the reporting period. It is equally important that departments and agencies use the information at their disposal to make in-year changes to plans, where feasible.

The Treasury Board Secretariat strongly supports the practice of balanced reporting. Its 2008-09 DPR guidance included the following instructions on how to achieve balance in reporting: “Report whether the expected results were met or not met and identify lessons learned. Both successes and performance gaps must be addressed. If plans did not unfold as expected, explain why and how this affected performance. Outline any corrective actions or strategies that will be pursued in light of the results achieved”. The Secretariat will expand this guidance for the 2010-11 DPR and place additional emphasis on the need to discuss the steps taken, where warranted and supported through evidence, to modify program design and delivery in order to improve performance. 

Industry Canada and Transport Canada agree to continue to work towards producing more balanced, credible reports. Summarized below is a discussion of progress made by both departments since 2008-09, as well as what can be expected in terms of future improvements.

Industry Canada

Balanced reporting in Industry Canada’s 2009-10 DPR is evidenced in the reporting of results that did not meet targets specified in the 2009-10 RPP, along with explanations as to why these targets were not met, where applicable. For example, Industry Canada did not meet its target for a 10 percent “Increase in number of clients using the Canada Business Network (CBN) website over the previous year”. The explanation provided for this is that a new methodology was introduced in 2009 to count website visits. Furthermore, a new CBN website was launched in October 2009, reducing the number of websites from 14 to 1. This resulted in a reduction in referrals to the website by external search engines, which is typical, but temporary, following major changes to a website.

Industry Canada has taken steps beyond those included in Treasury Board Secretariat guidance. In an effort to present a balanced performance story, for example, the department included trends against historical results information for indicators, where available, in the 2009-10 DPR, which helped to establish year-over-year comparisons. The department provided an online layer of content that gives users access to performance information at lower levels of the PAA, as well as additional information on lessons learned.

Where applicable, Industry Canada has included lessons learned in its DPRs to explain how adjustments were made during that fiscal year where a need for an adjustment was identified. As an example of an adjustment made, the Community Futures Program risk assessment process was altered as a result of an audit performed on that program. This was shown as a lesson learned that included an in-year change in the 2009-10 DPR. However, there is a limited scope in the DPR for discussion of future program design changes since the document’s purpose is to explain to parliamentarians what departments and agencies accomplished with authorized resources within that fiscal year. 

Industry Canada will place further emphasis on explaining risk information and providing more lessons learned for program activities in the 2010-11 DPR.

Transport Canada

The extensive review of Transport Canada’s PMF that is continuing in 2010-11 will result in more balanced and credible reporting, as well as more meaningful, specific and measurable departmental indicators. A stable PMF for the upcoming years will also facilitate benchmarking and the provision of baseline data. The revised indicators at lower levels of the PAA emerging from the review process will be available for the 2012-13 DPR. Until 2012-13, Transport Canada will be reporting on a PMF that is still a work in progress in terms of meaningful measures and data to support the reporting of results. Despite this, the department has adopted an approach whereby qualitative proxy measures are reported so as to provide more comprehensive and accurate results-based information. (This was the approach used in the 2009-10 DPR.)

RECOMMENDATION 5

That the Treasury Board of Canada Secretariat provide departments and agencies with a template for the preparation of their departmental performance reports, so the reports are structured in a similar fashion in order to be more use-friendly.

This recommendation is accepted.

The Treasury Board Secretariat currently provides departments and agencies with guidelines that include a suggested template for DPRs. The Secretariat will develop a new common structure that will provide a standardized index and template for all departments and agencies to follow in the preparation of 2010-11 DPRs. Departments and agencies will continue to have the flexibility to report content as suited to their unique context, mandate and results.

RECOMMENDATION 6

That, where feasible, departments and agencies provide detailed financial and performance information about programs on their websites.

This recommendation is accepted.

The Policy on MRRS provides the basis upon which the Estimates documents are structured. In their DPRs, departments and agencies are required to provide financial and non-financial performance information at the strategic outcome level and program activity level. This structure ensures that the DPR remains a concise, results-focused document that enhances the reader’s ability to understand a federal department’s core mandate.

Departments and agencies normally table their DPRs in Parliament in the fall of every year⁴.  Where feasible, annual electronic (i.e., web-based) reporting of lower level program results would complement the information provided in the DPR. The Secretariat will develop and implement a standardized and streamlined approach to annual electronic reporting on lower level programs for the 2013-14 DPR, which will allow departments and agencies the necessary time to improve and stablize their PAAs and PMFs as a consistent basis for reporting.

Various departments have already begun posting certain elements of lower level program performance information on their websites. For example, Industry Canada’s DPR provides links to further performance information that is available online. It should also be noted that Transport Canada’s evaluations of various program areas are available on its publicly-accessible website. This allows the department to demonstrate to parliamentarians and the public the linkages between its performance indicators, its expected results and its actual results.

RECOMMENDATION 7

That, if feasible, all House of Commons Standing Committees review the DPRs referred to them in order to ensure that they contain the information needed to allow parliamentarians to hold the government to account for achieving meaningful results for Canadians in an economical, efficient and effective manner.

It would not be appropriate for the Government to respond to this recommendation, as this is the prerogative of Parliament.