PACP Committee Report

INTRODUCTION
            One of Parliament’s fundamental roles is to review and approve the government’s spending plans and subsequently hold the government to account for its expenditures. The government sets out its spending plans in the Estimates documents and actual spending is outlined in the Public Accounts.

            Of course, it is not simply a matter of spending money but of achieving results for Canadians with those expenditures. While formerly governments primarily managed according to inputs, such as amounts spent, governments are increasingly focusing on what results they expect to achieve with those inputs, that is, how they intend to improve the lives of Canadians.

            As part of this trend towards managing for results, each federal department releases a report on plans and priorities (RPP) in the spring.^[1] These reports are intended to supplement the information provided in the Main Estimates by linking financial information to performance expectations. RPPs set out each organization’s planned spending and human resources for the next three years, priorities for the coming fiscal year, the results it intends to achieve, and how it will measure its performance.

            In the fall, approximately six months after the end of the preceding fiscal year, each department releases a departmental performance report (DPR).  DPRs set out actual spending and human resources used during the fiscal year and demonstrate the results achieved as compared to the priorities and performance expectations outlined in the corresponding RPP.

            The role of Parliament, then, is to scrutinize the performance objectives outlined in the RPPs in order to assess whether they are appropriate and reasonable, and then examine the DPRs to determine whether actual performance met expectations. These documents should help Parliament hold the government to account for achieving meaningful results for Canadians. However, parliamentarians have not given much attention to the reports, in part because they have been dissatisfied with their quality. The performance objectives contained in the RPPs are often not specific or concrete and parliamentarians find the results statements in the DPRs lacking in credibility and objectivity.^[2]

            The Office of the Auditor General of Canada (OAG) has conducted a number of audits of performance reports, beginning in 1988 and continuing with further audits in 1992, 1997, 2000, 2003, and 2005. Despite the fact that performance reports have been produced in one form or another since 1981, the OAG continually has found that progress in improving these reports was disappointing. As the House of Commons Standing Committee on Public Accounts (the Committee) has had a long-standing interest in the quality of DPRs, in February 2008 it requested that each year the OAG audit two randomly selected DPRs.^[3] The OAG declined to conduct the audit due to resource constraints.^[4] On 5 May 2009, the Committee decided to conduct its own study of DPRs, with two departments selected at random—Industry Canada and Transport Canada.

            The Committee held its hearing on the 2008-2009 DPRs of Industry Canada and Transport Canada on 13 May 2010.^[5] From the OAG, the Committee met with: Neil Maxwell, Assistant Auditor General and Tom Wileman, Principal. From Treasury Board of Canada Secretariat, the Committee met with: Alister Smith, Assistant Secretary, Expenditure Management Sector and Amanda Jane Preece, Executive Director, Results Based Management Division. Industry Canada was represented by: Richard Dicerni, Deputy Minister; Kelly Gillis, Chief Financial Officer, Comptrollership and Administration Sector; and Ron Parker, Assistant Deputy Minister, Industry Sector. Transport Canada was represented by: Yaprak Baltacioglu, Deputy Minister and André Morency, Assistant Deputy Minister, Corporate Management and Crown Corporation Governance, Corporate Services.

RATING MODEL AND REPORTING PRINCIPLES
            In order to conduct audits of DPRs, as well as help departments and others assess the quality of DPRs, in 2002 the OAG developed a rating model for DPRs.^[6] This model was further refined and put into practice through audits in 2003 and 2005. ^[7] The OAG’s model for good quality DPRs encompasses the following five criteria:

1. Organizational context and strategic outcomes are clear. Reports should clearly situate the department’s work within a broader context, and describe the department’s operating environment, its mission and mandate, external partners, and risks faced. They should also communicate how a department’s activities are organized and present a logical sequence from program activities to key results. Reports should clearly identify what business the department is in and what it does for Canadians.
2. Performance expectations are clear and concrete. Performance expectations are initially provided in a department’s RPPs and should be expressed as outputs or outcomes,^[8] specify the direction of planned changes, and the time frame in which those changes will be made. If expectations are not clearly stated, it is hard to assess whether performance results met expectations. In other words, departments should clearly state their commitments to Canadians in a way that allows them to be held to account for meeting those commitments.
3. Key results are reported against expectations. Reports should state whether the department achieved its performance expectations in terms of outputs and outcomes and explain any gaps in performance. Reports should clearly state whether the department met its commitments to Canadians, and if not, explain why not.
4. Performance information is credible and balanced. Reporting should be based on reliable information and credible data. The reporting of good results should be balanced with the reporting of shortcomings. Parliamentarians and Canadians should be able to read a performance story that is credible, balanced, unbiased, and complete.
5. Use of performance information is demonstrated. Managers should use performance information to improve program design, implementation, and results. Thus, performance reports should highlight lessons that have been learned and how corrective action was taken to improve performance.

            Using this model, the OAG assessed DPRs as basic, fair, good, very good, or excellent for each criterion. Overall, in 2005 the OAG found that most of the performance reports provided a good overview of departments’ organizational context and planned strategic outcomes, but performance expectations were not always clear and concrete, information did not always focus on results, reported results were not always balanced, and the reports provided little evidence that performance information was used to improve program results.

            In the opinion of the OAG, there are five factors that have inhibited the quality of performance reporting. They are:

1. The basic principles of good reporting are not well understood and applied.
2. As performance reporting takes place in a political environment, there is a reluctance to report shortcomings.
3. There are no incentives for good reporting or sanctions applied for poor reporting.
4. Many departments do not consider performance reports to be a high priority.
5. Parliamentary committees have not given much scrutiny to or shown interest in performance reports.

            The Treasury Board of Canada Secretariat (the Secretariat) assists departments in the preparation of DPRs by providing guidance, templates, samples, information sessions, and a good practices handbook. ^[9] The Secretariat also outlines four Government of Canada reporting principles. They are:

1. Focus on the benefits for Canadians, explain the critical aspects of planning and performance and set them in context. Information in a RPP and a DPR should be relevant to Members of Parliament and to Canadians and should explain the benefits for Canadians that the department aims to achieve through its strategic outcomes and program activities.
2. Present concise, credible, reliable, and balanced information. It is important that a coherent, sound, and balanced picture of performance be presented in the DPR. This requires the department to acknowledge where performance did not meet expectations, provide the necessary explanations as to why, and describe the corrective actions that will be taken in the future.
3. Associate performance with plans, priorities, and expected results, explain changes, and apply lessons learned. The core elements of a public performance report are descriptions of what an organization planned to accomplish, how the actual results compare with those plans, explanation of any significant differences, and identification of lessons learned.
4. Link resources to results. Accountability from a public reporting perspective means explaining what has been accomplished with the resources entrusted to a department in relation to what was planned and demonstrating whether performance represents a responsible, efficient, and effective use of public funds.

EXAMINING SELECTED PERFORMANCE REPORTS
            In light of these criteria and reporting principles, the Committee identified in its examination of the DPRs of Industry Canada and Transport Canada a number of areas where improvements can be made, which are outlined below.

A. Providing a performance story
            In order to be able to report meaningfully on performance, it is important that departments present a logical sequence from program activities to key results and set clear and concrete performance expectations. If departments do not clearly link program activities to results, it is difficult to determine what difference they are making, and if they do not make clear commitments, then it is difficult to hold them to account for their performance.

            The Committee found some reasonable practices in the DPRs of Transport Canada and Industry Canada. Both provided a performance analysis for their program activities, including performance indicators and expected results. They also included quantifiable information related to their performance indicators. However, the reports did not set specific targets for performance and did not include baseline data from which to make comparisons of progress. Moreover, even if there were specific targets and baseline data, explanations would need to be provided in order to link departmental activities to results. Industry Canada, for example, provides a general expected result for its various program activities, a performance indicator and its trend, and outlines various activities undertaken over the year. However, there is little explanation or linkage provided between the activities and the indicator or results. If the indicator is trending downwards, it is not clear why, or what changes need to be made, if any. That is, it is not clear how the activities have contributed positively or negatively to the indicator and expected results.

            This matters because in order to understand performance, it is necessary to have a clear desired objective, a proposed trajectory of how to get there, an outline of actions that will be taken to achieve the objective, and an indication of how these actions will help achieve the objective. Undertaking this approach would allow departments to provide an annual assessment of progress towards achieving the objective and explain any modifications that might need to be taken in order to improve performance.

            As they currently stand, Industry Canada’s and Transport Canada’s DPRs lack a clear perspective of where Canada is heading, of year-over-year progress, in their respective policy areas, and what contributions they are making. They do not provide a logical connection between activities, outputs, immediate outcomes, intermediate outcomes, and strategic objectives. In other words, the DPRs do not provide a performance story. They do not provide a story of what results the departments expect to achieve and how their programs did or did not contribute to achieving those results. The Committee recommends:

RECOMMENDATION 1
That Industry Canada and Transport Canada set clear and concrete performance targets that are directly related to the expected performance of their programs.

RECOMMENDATION 2
That Industry Canada and Transport Canada clearly demonstrate in their departmental performance reports the link between departmental activities, expected results, and actual performance.

B. Methodology
            Both Industry Canada and Transport Canada provide a number of performance indicators in their DPRs, which give readers some objective, quantifiable information. However, the Committee found that the DPRs did not  discuss why the particular indicators were used, whether and why changes were made to the indicators from year to year, what the strengths and weaknesses of the chosen indicators are, and how those indicators fit into an overall context of performance. The indicators also only provided a one-year snapshot of performance rather than a broader year-over-year perspective.

            Transport Canada assessed its performance status in various areas as “mostly met” or “met all.” However, it is not at all clear from the DPR how the department interprets “mostly met,” what exactly was “mostly met,” and, by extension, what was not met. A performance status should provide meaningful information. Without further explanation, it is impossible for the reader to understand what is meant by “mostly met,” or to determine whether or not this is a fair and reasonable presentation of program performance.

            Both Industry Canada and Transport Canada need to make improvements in how they explain the methodology used to present results. They need to provide explanations about their indicators, targets, and assessments of performance. The Committee recommends:

RECOMMENDATION 3
That Industry Canada and Transport Canada provide more detailed explanations in their departmental performance reports about the methodology used to present results.

C. Balanced and Credible Reporting
            One of the most common criticisms about DPRs is that they are not credible because they are not sufficiently balanced. Reporting on performance should be part of a larger initiative to manage for results and thus performance information should be useful for program managers, and not just for external reporting.

            When asked about this issue, officials from Industry Canada pointed to indicators in their DPR that were trending downwards. They indicated that Industry Canada’s future DPRs would include specific targets and more information from evaluations.^[10] Also, both departments have discussions in their DPRs on “lessons learned.”

            Providing quantitative indicators and specific targets can help create a more balanced report because they give a clear, objective picture of results, making it more difficult to soften the presentation of unfavourable results through carefully chosen language. The Committee has observed this in the OAG’s DPR, which contains a number of quantifiable performance indicators and specific targets. For the past few years, the OAG has struggled to meet its targets for completing audits on time and on budget. As a result, the Auditor General has been asked questions by Committee members about how the OAG will improve in this area.

            However, quantifiable indicators and specific targets are not sufficient to achieve balance because the indicators may not be clearly related to departmental activities, as was noted earlier. If an indicator is not linked to departmental activities, then it becomes difficult to explain how departmental performance was less than expected or what steps need to be taken in order to improve results with respect to the indicator. Industry Canada’s DPR did contain examples of declining indicators, but the DPR did not explain the reasons for the decline or what would be done to improve program design and delivery in order to address results that were less than expected.

            While both DPRs contained discussions of lessons learned, these were not well incorporated into the overall presentation of program performance. Lessons should be learned when program performance does not meet expectations and changes need to be made. Yet, it was not clear from Industry Canada’s and Transport Canada’s DPRs how program performance was less than expected and what lessons were learned as a result. As the OAG noted in its rating model, the DPRs should demonstrate how program managers have used performance information to improve their programs. Doing so would provide a more balanced presentation and thereby make the reports more credible. Thus, the Committee recommends:

RECOMMENDATION 4
That Industry Canada and Transport Canada provide credible and balanced performance reports by clearly indicating where program performance was less than expected, explaining why, and discussing what steps were taken to modify program design and delivery in order to improve performance.

TEMPLATE
            Over 90 federal organizations prepare DPRs. The Secretariat produces a guide to preparing the DPRs and a good practices handbook. The guide contains various tables and charts that departments are to include in their DPRs. The good practices handbook elaborates on the Secretariat’s four reporting principles, discussed earlier, and provides examples of good practices found in various DPRs. Departments are free to design the presentation of their DPR as they see fit, as long as they follow these general guidelines.

            However, this can mean that it is difficult to find similar types of information in different DPRs. For example, Transport Canada clearly identifies its risk analysis in its table of contents; whereas, Industry Canada’s risk analysis lies buried within its departmental overview. Moreover, each department has a different approach to risk analysis. Transport Canada’s DPR discusses risks to the transportation sector, but, for the most part, does not link these risks to departmental performance. Industry Canada, on the other hand, discusses internal corporate risks, such as recruitment and retention, as well as indicating that the department works with various industry sectors to help them deal with changes in the economy.

            In order to be more user-friendly, the Committee believes that DPRs should be structured in a similar fashion. A common layout would make it easier to find information, as well as interpret the information that is presented. As it currently stands, parliamentarians must learn how to navigate each DPR separately, which can make it more time consuming to read the reports, and more difficult to make comparisons between organizations. Consequently, the Committee recommends:

RECOMMENDATION 5
That the Treasury Board of Canada Secretariat provide departments and agencies with a template for the preparation of their departmental performance reports, so the reports are structured in a similar fashion in order to be more user-friendly.

E-REPORTING
            When preparing their DPRs, departments must sort through a considerable amount of information in order to present a concise document to Parliament. Transport Canada’s Deputy Minister told the Committee that her department has over 180 outcomes.^[11] According to Industry Canada’s Chief Financial Officer, their 2006-2007 DPR was over 150 pages long. ^[12]

            In an effort to streamline the information presented to Parliament, the Secretariat encouraged departments to use a concise reporting format for their reports tabled in 2009. The Secretariat advises departments that their DPRs should be approximately 30 pages in length.

            While the Committee understands and supports the move to a concise reporting format, it is concerned that streamlining the documents means that a lot of information is left out and thus not reported publicly. Under the Treasury Board’s Management, Resources and Results Structure Policy, departments must develop a “program activity architecture,” which outlines their program activities, sub-activities, sub-sub-activities, etc. The goal of the program activity architecture is to link resource allocations to intended results. Departments subsequently report financial and performance information as it pertains to their program activities, but not lower levels, in the DPRs.

            However, most people understand government activities through specific programs, and not program activities, which are collections of numerous, sometimes disparate, programs. Indeed, this is how the government manages its work and budget announcements are usually made with respect to programs and not program activities. Likewise, many parliamentarians are interested in how much has been spent on a particular program, not on a program activity. Reporting on program activities, rather than programs, makes it difficult to connect departmental activities to results, a problem that was noted earlier. On the other hand, it is only by consolidating information into program activities that departments can prepare concise reports.

           
In order to continue to present concise reports to Parliament, as well as provide more detailed information on programs, the Committee believes that departments should present financial and performance information about specific programs on their websites, which is sometimes referred to as “e-reporting.” This would allow parliamentarians and Canadians to have access to detailed and meaningful financial and performance information, when it is desired, as well as to have a more general overview of the department as is presented in DPRs. At present, only the latter is available, and those who are interested in more detailed information are unable to obtain it. While Industry Canada has many electronic links within its DPR, these lead to general websites, program descriptions, or media releases and not to the kind of financial and performance information found in the DPR, but at a more specific program level. As this kind of information would be useful for both parliamentarians and Canadians, the Committee recommends:

RECOMMENDATION 6
That, where feasible, departments and agencies provide detailed financial and performance information about programs on their websites.

COMMITTEE REVIEW
            The main vehicle for parliamentary review of DPRs is through committees. Committees can review DPRs when they come out in the fall, or they  can incorporate review of the DPRs into their consideration of the Main Estimates, as is done by the Public Accounts Committee, since past performance can be used to assess the reasonableness of current performance expectations and requests for annual appropriations.

            However, most policy committees do not review DPRs, in part because they are occupied with other studies, such as legislative reviews and extensive policy work, and because they do not find the information in the DPRs useful for their needs. Yet, committee review of DPRs is essential for improving the quality of the reports. It is only by examining the DPRs and questioning departmental officials about the information DPRs do and do not contain that parliamentarians can provide feedback about what changes they would like to see. As noted earlier, one of the five factors identified by the OAG that has inhibited the quality of performance reports is that parliamentary committees have not given much scrutiny to or shown interest in performance reports. Departments have little incentive to improve their DPRs if the documents receive little attention or scrutiny. The Deputy Minister of Transport Canada welcomed the Committee’s study. She said, “I congratulate the committee for this process. I think it gives us additional energy to focus further on our reporting documentation. We really appreciate that.”^[13]

            Parliamentarians need to take responsibility for the information they receive, and push departments to improve their DPRs in ways that meet the needs of parliamentarians. While this Committee can make some observations about the quality of specific DPRs in terms of general criteria, the reports are better assessed in light of the policy considerations they are meant to address; that is, the individual policy committees are better placed to assess whether specific DPRs provide meaningful information about departmental performance in addressing policy issues in a given area. Consequently, the Committee recommends:

RECOMMENDATION 7
That, if feasible, all House of Commons Standing Committees review the DPRs referred to them in order to ensure that they contain the information needed to allow parliamentarians to hold the government to account for achieving meaningful results for Canadians in an economical, efficient, and effective manner.

  CONCLUSION
            The Deputy Minister of Transport Canada acknowledged that her department’s performance report could be improved and is a work in progress. ^[14] However, the Assistant Auditor General noted that the DPRs have been the responsibility of departments since 1981, or almost 30 years.^[15] The Committee would have hoped that more progress would have been made by now. The Committee found that the DPRs of Industry Canada and Transport Canada need to: include clear and concrete performance targets that are directly related to the expected performance of their programs; demonstrate the link between departmental activities, expected results, and actual performance; provide more detailed explanations about the methodology used to present results; and indicate where program performance was less than expected, explain why, and discuss what steps were taken to modify program design and delivery in order to improve performance. The Committee believes that taking these actions would greatly improve the usefulness of DPRs.

            The Committee also thinks that it should be easier to find information by adopting a template for DPRs and by putting more detailed financial and performance information on departmental websites. Ultimately though, it is up to parliamentarians and  committees to encourage the government to provide reports that they find useful in holding the government to account for achieving results for Canadians. The following quote from the Secretariat sums up the importance of credible performance reporting:
Without clearly communicated plans, priorities, and expected results, coupled with the reporting of results against those plans and performance expectations, the ability of parliamentarians to hold the government to account is impeded. Without effective public reporting, accountability and transparency in government are diminished and trust in public institutions suffers.^[16]