PACP Committee Report
If you have any questions or comments regarding the accessibility of this publication, please contact us at accessible@parl.gc.ca.
INTRODUCTION Of course, it is not simply a matter of spending money but of achieving results for Canadians with those expenditures. While formerly governments primarily managed according to inputs, such as amounts spent, governments are increasingly focusing on what results they expect to achieve with those inputs, that is, how they intend to improve the lives of Canadians. As part of this trend towards managing for results, each federal department releases a report on plans and priorities (RPP) in the spring.[1] These reports are intended to supplement the information provided in the Main Estimates by linking financial information to performance expectations. RPPs set out each organization’s planned spending and human resources for the next three years, priorities for the coming fiscal year, the results it intends to achieve, and how it will measure its performance. In the fall, approximately six months after the end of the preceding fiscal year, each department releases a departmental performance report (DPR). DPRs set out actual spending and human resources used during the fiscal year and demonstrate the results achieved as compared to the priorities and performance expectations outlined in the corresponding RPP. The role of Parliament, then, is to scrutinize the performance objectives outlined in the RPPs in order to assess whether they are appropriate and reasonable, and then examine the DPRs to determine whether actual performance met expectations. These documents should help Parliament hold the government to account for achieving meaningful results for Canadians. However, parliamentarians have not given much attention to the reports, in part because they have been dissatisfied with their quality. The performance objectives contained in the RPPs are often not specific or concrete and parliamentarians find the results statements in the DPRs lacking in credibility and objectivity.[2] The Office of the Auditor General of Canada (OAG) has conducted a number of audits of performance reports, beginning in 1988 and continuing with further audits in 1992, 1997, 2000, 2003, and 2005. Despite the fact that performance reports have been produced in one form or another since 1981, the OAG continually has found that progress in improving these reports was disappointing. As the House of Commons Standing Committee on Public Accounts (the Committee) has had a long-standing interest in the quality of DPRs, in February 2008 it requested that each year the OAG audit two randomly selected DPRs.[3] The OAG declined to conduct the audit due to resource constraints.[4] On 5 May 2009, the Committee decided to conduct its own study of DPRs, with two departments selected at random—Industry Canada and Transport Canada. The Committee held its hearing on the 2008-2009 DPRs of Industry Canada and Transport Canada on 13 May 2010.[5] From the OAG, the Committee met with: Neil Maxwell, Assistant Auditor General and Tom Wileman, Principal. From Treasury Board of Canada Secretariat, the Committee met with: Alister Smith, Assistant Secretary, Expenditure Management Sector and Amanda Jane Preece, Executive Director, Results Based Management Division. Industry Canada was represented by: Richard Dicerni, Deputy Minister; Kelly Gillis, Chief Financial Officer, Comptrollership and Administration Sector; and Ron Parker, Assistant Deputy Minister, Industry Sector. Transport Canada was represented by: Yaprak Baltacioglu, Deputy Minister and André Morency, Assistant Deputy Minister, Corporate Management and Crown Corporation Governance, Corporate Services. RATING MODEL AND REPORTING PRINCIPLES
Using this model, the OAG assessed DPRs as basic, fair, good, very good, or excellent for each criterion. Overall, in 2005 the OAG found that most of the performance reports provided a good overview of departments’ organizational context and planned strategic outcomes, but performance expectations were not always clear and concrete, information did not always focus on results, reported results were not always balanced, and the reports provided little evidence that performance information was used to improve program results. In the opinion of the OAG, there are five factors that have inhibited the quality of performance reporting. They are:
The Treasury Board of Canada Secretariat (the Secretariat) assists departments in the preparation of DPRs by providing guidance, templates, samples, information sessions, and a good practices handbook. [9] The Secretariat also outlines four Government of Canada reporting principles. They are:
EXAMINING SELECTED PERFORMANCE REPORTS A. Providing a performance story The Committee found some reasonable practices in the DPRs of Transport Canada and Industry Canada. Both provided a performance analysis for their program activities, including performance indicators and expected results. They also included quantifiable information related to their performance indicators. However, the reports did not set specific targets for performance and did not include baseline data from which to make comparisons of progress. Moreover, even if there were specific targets and baseline data, explanations would need to be provided in order to link departmental activities to results. Industry Canada, for example, provides a general expected result for its various program activities, a performance indicator and its trend, and outlines various activities undertaken over the year. However, there is little explanation or linkage provided between the activities and the indicator or results. If the indicator is trending downwards, it is not clear why, or what changes need to be made, if any. That is, it is not clear how the activities have contributed positively or negatively to the indicator and expected results. This matters because in order to understand performance, it is necessary to have a clear desired objective, a proposed trajectory of how to get there, an outline of actions that will be taken to achieve the objective, and an indication of how these actions will help achieve the objective. Undertaking this approach would allow departments to provide an annual assessment of progress towards achieving the objective and explain any modifications that might need to be taken in order to improve performance. As they currently stand, Industry Canada’s and Transport Canada’s DPRs lack a clear perspective of where Canada is heading, of year-over-year progress, in their respective policy areas, and what contributions they are making. They do not provide a logical connection between activities, outputs, immediate outcomes, intermediate outcomes, and strategic objectives. In other words, the DPRs do not provide a performance story. They do not provide a story of what results the departments expect to achieve and how their programs did or did not contribute to achieving those results. The Committee recommends:
B. Methodology Transport Canada assessed its performance status in various areas as “mostly met” or “met all.” However, it is not at all clear from the DPR how the department interprets “mostly met,” what exactly was “mostly met,” and, by extension, what was not met. A performance status should provide meaningful information. Without further explanation, it is impossible for the reader to understand what is meant by “mostly met,” or to determine whether or not this is a fair and reasonable presentation of program performance. Both Industry Canada and Transport Canada need to make improvements in how they explain the methodology used to present results. They need to provide explanations about their indicators, targets, and assessments of performance. The Committee recommends:
C. Balanced and Credible Reporting When asked about this issue, officials from Industry Canada pointed to indicators in their DPR that were trending downwards. They indicated that Industry Canada’s future DPRs would include specific targets and more information from evaluations.[10] Also, both departments have discussions in their DPRs on “lessons learned.” Providing quantitative indicators and specific targets can help create a more balanced report because they give a clear, objective picture of results, making it more difficult to soften the presentation of unfavourable results through carefully chosen language. The Committee has observed this in the OAG’s DPR, which contains a number of quantifiable performance indicators and specific targets. For the past few years, the OAG has struggled to meet its targets for completing audits on time and on budget. As a result, the Auditor General has been asked questions by Committee members about how the OAG will improve in this area. However, quantifiable indicators and specific targets are not sufficient to achieve balance because the indicators may not be clearly related to departmental activities, as was noted earlier. If an indicator is not linked to departmental activities, then it becomes difficult to explain how departmental performance was less than expected or what steps need to be taken in order to improve results with respect to the indicator. Industry Canada’s DPR did contain examples of declining indicators, but the DPR did not explain the reasons for the decline or what would be done to improve program design and delivery in order to address results that were less than expected. While both DPRs contained discussions of lessons learned, these were not well incorporated into the overall presentation of program performance. Lessons should be learned when program performance does not meet expectations and changes need to be made. Yet, it was not clear from Industry Canada’s and Transport Canada’s DPRs how program performance was less than expected and what lessons were learned as a result. As the OAG noted in its rating model, the DPRs should demonstrate how program managers have used performance information to improve their programs. Doing so would provide a more balanced presentation and thereby make the reports more credible. Thus, the Committee recommends:
TEMPLATE However, this can mean that it is difficult to find similar types of information in different DPRs. For example, Transport Canada clearly identifies its risk analysis in its table of contents; whereas, Industry Canada’s risk analysis lies buried within its departmental overview. Moreover, each department has a different approach to risk analysis. Transport Canada’s DPR discusses risks to the transportation sector, but, for the most part, does not link these risks to departmental performance. Industry Canada, on the other hand, discusses internal corporate risks, such as recruitment and retention, as well as indicating that the department works with various industry sectors to help them deal with changes in the economy. In order to be more user-friendly, the Committee believes that DPRs should be structured in a similar fashion. A common layout would make it easier to find information, as well as interpret the information that is presented. As it currently stands, parliamentarians must learn how to navigate each DPR separately, which can make it more time consuming to read the reports, and more difficult to make comparisons between organizations. Consequently, the Committee recommends:
E-REPORTING In an effort to streamline the information presented to Parliament, the Secretariat encouraged departments to use a concise reporting format for their reports tabled in 2009. The Secretariat advises departments that their DPRs should be approximately 30 pages in length. While the Committee understands and supports the move to a concise reporting format, it is concerned that streamlining the documents means that a lot of information is left out and thus not reported publicly. Under the Treasury Board’s Management, Resources and Results Structure Policy, departments must develop a “program activity architecture,” which outlines their program activities, sub-activities, sub-sub-activities, etc. The goal of the program activity architecture is to link resource allocations to intended results. Departments subsequently report financial and performance information as it pertains to their program activities, but not lower levels, in the DPRs. However, most people understand government activities through specific programs, and not program activities, which are collections of numerous, sometimes disparate, programs. Indeed, this is how the government manages its work and budget announcements are usually made with respect to programs and not program activities. Likewise, many parliamentarians are interested in how much has been spent on a particular program, not on a program activity. Reporting on program activities, rather than programs, makes it difficult to connect departmental activities to results, a problem that was noted earlier. On the other hand, it is only by consolidating information into program activities that departments can prepare concise reports.
COMMITTEE REVIEW However, most policy committees do not review DPRs, in part because they are occupied with other studies, such as legislative reviews and extensive policy work, and because they do not find the information in the DPRs useful for their needs. Yet, committee review of DPRs is essential for improving the quality of the reports. It is only by examining the DPRs and questioning departmental officials about the information DPRs do and do not contain that parliamentarians can provide feedback about what changes they would like to see. As noted earlier, one of the five factors identified by the OAG that has inhibited the quality of performance reports is that parliamentary committees have not given much scrutiny to or shown interest in performance reports. Departments have little incentive to improve their DPRs if the documents receive little attention or scrutiny. The Deputy Minister of Transport Canada welcomed the Committee’s study. She said, “I congratulate the committee for this process. I think it gives us additional energy to focus further on our reporting documentation. We really appreciate that.”[13] Parliamentarians need to take responsibility for the information they receive, and push departments to improve their DPRs in ways that meet the needs of parliamentarians. While this Committee can make some observations about the quality of specific DPRs in terms of general criteria, the reports are better assessed in light of the policy considerations they are meant to address; that is, the individual policy committees are better placed to assess whether specific DPRs provide meaningful information about departmental performance in addressing policy issues in a given area. Consequently, the Committee recommends:
CONCLUSION The Committee also thinks that it should be easier to find information by adopting a template for DPRs and by putting more detailed financial and performance information on departmental websites. Ultimately though, it is up to parliamentarians and committees to encourage the government to provide reports that they find useful in holding the government to account for achieving results for Canadians. The following quote from the Secretariat sums up the importance of credible performance reporting: [1]In this report, the term “department” is meant to include departments and agencies. [2] The Canadian Comprehensive Auditing Foundation outlines legislators’ concerns with public performance reports in Users and Uses, September 2005. [3] House of Commons Standing Committee on Public Accounts, Departmental Performance Report, 8th Report, 39th Parliament, 2nd Session, February 2008. [4] The OAG’s response was contained in a letter to the Committee dated 20 February 2009 from the Auditor General, which stated, in part, “I would therefore like to inform the Committee that, in order to make the best use of our current resource level, we will not be able to conduct this particular audit as requested.” [5] House of Commons Standing Committee on Public Accounts, 40th Parliament, 3rd Session, Meeting 15. [6] Auditor General of Canada, April 2002 Report, Chapter 6—A Model for Rating Departmental Performance Reports. [7] Auditor General of Canada, May 2003 Report, Chapter 1—Rating Department Performance Reports and April 2005 Report, Chapter 5—Rating Selected Departmental Performance Reports. [8] An output is a product or service from a department’s activities (for example, a pamphlet, research study, or training), and an outcome is an external consequence attributed to a department’s program or initiative. [9] The Treasury Board of Canada Secretariat’s primary documents for preparing DPRs are: Guide to the Preparation of Part II of the 2008-2009 Estimates and Performance Reporting--Good Practices Handbook 2009. [10] Meeting 15, 10:25. [11] Meeting 15, 9:55. [12] Ibid., 10:05. [13] Meeting 15, 9:00. [14] Meeting 15, 9:00. [15] Ibid., 10:55. [16] Treasury Board of Canada Secretariat, Performance Reporting: Good Practices Handbook, 2008, page 4.
|