PACP Committee Report
If you have any questions or comments regarding the accessibility of this publication, please contact us at accessible@parl.gc.ca.
INTRODUCTION The importance of evaluation in the federal government has been emphasized through recent legislative and policy changes. The 2006 Federal Accountability Act introduced an amendment to the Financial Administration Act that requires all non-statutory grants and contributions programs to be reviewed every five years for relevance and effectiveness. [1] Under Treasury Board’s Policy on Evaluation, which was updated in 2009, large departments must ensure that all direct program spending is evaluated every five years.[2] In order to meet these requirements, it is vital that departments have adequate capacity, support, and systems in place to complete the evaluations required by legislation and Treasury Board policy. The Office of the Auditor General (OAG) included in its Fall 2009 Report an audit on how evaluation units in six departments (Agriculture and Agri-Food Canada, Canadian Heritage, Citizenship and Immigration Canada, Environment Canada, Fisheries and Oceans Canada, and Human Resources and Skills Development Canada) identified and responded to various needs for effectiveness evaluation, and whether the departments had developed the required capacity to respond to those needs.[3] The OAG also examined the oversight and support role of the Treasury Board of Canada Secretariat (the Secretariat) in monitoring and improving the evaluation function in the federal government. As the Public Accounts Committee has long supported the development of effectiveness evaluations in the federal government, it held a hearing on this audit on 4 May 2010.[4] From the Office of the Auditor General the Committee met with: Sheila Fraser, Auditor General of Canada; Neil Maxwell, Assistant Auditor General; and Tom Wileman, Principal. From the Treasury Board of Canada Secretariat the Committee met with: Michelle d'Auray, Secretary of the Treasury Board of Canada, and Alister Smith, Assistant Secretary, Expenditure Management Sector. The Committee also met with representatives from two of the departments audited. From Citizenship and Immigration Canada the Committee met with: Neil Yeates, Deputy Minister, and Elizabeth Ruddick, Director General, Research and Evaluation. From Environment Canada the Committee met with: Ian Shugart, Deputy Minister, and William Blois, Associate Director, Audit and Evaluation Branch. ANNUAL REPORT
The Committee decided to meet with officials from Environment Canada and Citizenship and Immigration Canada, as it would have been too unwieldy to invite representatives from all six departments audited. These two departments provided action plans to the Committee prior to the hearing on how they intend to address the OAG’s recommendations, and Agriculture and Agri-Food Canada and Fisheries and Oceans Canada provided action plans after the hearing. The Committee trusts that these departments will implement these plans and work to improve their effectiveness evaluation capacity. However, the Committee did not receive action plans from Canadian Heritage or Human Resources and Skills Development Canada. Thus the Committee recommends:
The audit findings also raise the concern that effectiveness evaluation needs to be improved in all departments and agencies and not just the ones audited. Within the federal government, it is a central agency—the Treasury Board of Canada Secretariat—that is responsible for monitoring the evaluation function across the government. The Secretariat developed the updated Policy on Evaluation, which was intended to address gaps in the coverage and nature of evaluations, the governance and independence of the evaluation function, and the lack of clarity around the collection of data. The Secretariat monitors the evaluation function through a number of means, such as an annual capacity assessment survey, periodic reviews of the quality of evaluation reports, and Management Accountability Framework (MAF) assessments of the quality and use of evaluations in departments. The Secretariat also maintains a Centre of Excellence for Evaluation (the Centre) that provides guidance and support to departments on evaluation. The audit found that the Centre had not made adequate progress in developing tools to help departments address the long-standing problem of insufficient data for the evaluation of effectiveness. At the time of the audit, the Secretariat also had not issued guidance to departments to help them implement the Policy on Evaluation.[6] The Secretary of the Treasury Board, Michelle d'Auray, told the Committee that the Centre has since issued draft guidance on developing departmental evaluation plans and performance measurement strategies. The Centre has also set up an evaluation community of practice for exchanges of best practices.[7] Alister Smith, Assistant Secretary, leads the Secretariat’s evaluation activities and indicated that these guidance documents should be finalized by the end of the fiscal year.[8] The Auditor General told the Committee that she was pleased to hear about the actions the government has taken since the completion of the audit.[9] Ordinarily, the Committee tracks departmental progress in addressing the OAG’s recommendations through status reports. In this instance, the Committee notes that the Secretariat has in the past published a report on the evaluation function, called “The Health of the Evaluation Function in the Government of Canada: Report for Fiscal Year 2004-2005.” According to the audit, this is the only such report published; though, the Secretariat continues to conduct an annual capacity assessment survey.[10] In her opening statement, the Secretary stated: It is not clear if the Secretariat intends to make this annual report available to the public. The Committee believes that this report should be made public as it would provide a valuable accountability tool for parliamentarians and Canadians to monitor progress in implementing the government’s new evaluation requirements. Thus, the Committee recommends:
PERFORMANCE INFORMATION The Deputy Minister of Citizenship and Immigration Canada, Neil Yeates, acknowledged that collecting performance information is a large challenge for his department, in part because data is often very expensive to collect.[13] The Deputy Minister of Environment Canada, Ian Shugart, stated that often the issue is not that there is no performance data, but that the data is not sufficient to cover all of the evaluation questions. His department has a performance measurement framework, but it is not complete.[14] The Policy on Evaluation requires departments to ensure that ongoing performance measurement is implemented so that sufficient performance information is available to support program evaluation.[15] The Secretariat monitors departmental management through the annual MAF assessments, which include a section on the “utility of the corporate performance framework.”[16] For the 2008 assessments, three of the six departments included in the audit–Citizenship and Immigration Canada, Canadian Heritage, and Agriculture and Agri-Food Canada—received the comment, “The performance indicators are not clear and cannot be used for data collection to provide reliable insight into program effectiveness.” Environment Canada and Fisheries and Oceans Canada received the comment, “The organization has developed a weak performance measurement framework.” And Human Resources and Skills Development Canada received the comment, “An incomplete or inadequate performance measurement framework has been developed.” Four of these departments received a rating of “acceptable” for the utility of the corporate performance framework, and two received a rating of “opportunity for improvement.” As a strong performance measurement framework is an important precondition for meaningful evaluations, the Committee recommends
It is clear that departments will have to make significant improvements to their performance measurement frameworks and subsequent data collection in order to enable meaningful effectiveness evaluations. The ongoing collection and assessment of reliable performance information is also important for managers to monitor program performance and for departments to be able to report publicly on the results their programs are achieving. While the Secretariat is monitoring this issue through its MAF assessments, these do not indicate the extent to which departments have sufficient performance information for the purposes of effectiveness evaluations. As the Committee would like to ensure that evaluations are not being hindered by a lack of performance information, it recommends:
MEETING LEGISLATIVE/POLICY REQUIREMENTS The audit found that departments had developed risk-based evaluation plans during the audit period.[17] However, the proportion of program expenses evaluated between 2004-2005 and 2007-2008 ranged from an annual average of 5% to 13%.[18] If departments are to achieve the requirement of evaluating all direct program spending over a five-year cycle, then they will have to have an annual average of 20%. While there is still time to increase evaluation capacity, departments have a substantial way to go before they will reach the target for the annual proportion of program spending evaluated. As the Committee would like to monitor progress to ensure that departments are on track to meet the evaluation requirements, it recommends:
STAFFING However, if all federal departments need to make a similar expansion in their evaluation units, then it may be difficult to find sufficient qualified and experienced evaluators. The audit noted that the shortage of experienced program evaluators in the federal government has been a longstanding concern, which has affected the ability of departments to hire the people they need and has led to evaluators being hired away by other evaluation units.[22] In order to ensure that they have sufficient staff, departments will have to prepare recruitment and retention strategies for their evaluation units. The Secretary of the Treasury Board told the Committee that there are currently over 500 evaluators working in the federal government,[23] and Alister Smith described how the Secretariat is working with universities and the Canada School of Public Service to help build up the community of qualified evaluators.[24] Nonetheless, the Committee remains concerned about the ability of departments to attract and retain qualified and experienced evaluators, as this will have a significant impact on their ability to complete the additional evaluations now required. The Committee would like to monitor this issue and recommends:
A further concern related to staffing raised by the audit was the heavy reliance on contractors. The audit found that about 90% of the evaluations completed during the audit period were wholly or partially conducted by contractors.[25] While there will always be an important role for contractors, as they can offer expertise that is not available internally and can provide supplementary capacity, an excessive reliance on contractors may inhibit the ability of internal evaluation units becoming subject-matter experts, instead causing them to spend most of their time managing contracts. The Committee trusts that this reliance on contractors will diminish as departments build up their evaluation units. Since the Committee would like to ensure that this is the case, it recommends:
FOLLOW-UP Given the current weaknesses in effectiveness evaluation identified by the audit and the possible difficulties in meeting the legislative and policy requirements, the Committee believes that it would be valuable to have a follow-up audit completed before the end of the first five year cycle in 2017-2018. The Committee notes that significant increases have been made in the financial and human resources devoted to the evaluation function within departments. The Committee believes that it is important that the government achieves value for money with these resources and that they are used in an economical and efficient manner. Thus, the Committee recommends:
The Committee also notes that the Policy on Evaluation requires the Secretariat to ensure that an evaluation of the policy is conducted every five years.[28] This evaluation should help determine whether the increased requirements for evaluations on program effectiveness are indeed effective. CONCLUSION [1] See section 260 of the 2006 Federal Accountability Act and section 42.1 of the Financial Administration Act. “Subject to and except as otherwise provided in any directives issued by the Treasury Board, every department shall conduct a review every five years of the relevance and effectiveness of each ongoing (grants and contributions) program for which it is responsible.” [2] Treasury Board of Canada, Policy on Evaluation, April 2009, section 6.1.8.a. [3] Auditor General of Canada, Fall 2009 Report, Chapter 1—Evaluating the Effectiveness of Programs. [4] House of Commons Standing Committee on Public Accounts, 40th Parliament, 3rd Session, Meeting 12. [5] Chapter 1, page 2. [6] Chapter 1, paragraphs 1.79-1.81. [7] Meeting 12, 9:10. [8] Ibid., 10:05. [9] Ibid., 10:50. [10] Chapter 1, paragraph 1.71. [11] Meeting 12, 9:10. [12] Chapter 1, paragraph 1.34. [13] Meeting 12, 9:35. [14] Ibid., 9:45. [15] Policy on Evaluation, section 6.1.10. [16] The Management Accountability Framework assessment scale and the results of assessments can be found on the Secretariat’s website at: http://www.tbs-sct.gc.ca/maf-crg/index-eng.asp. [17] Chapter 1, paragraph 1.22. [18] Ibid., exhibit 1.4. [19] Ibid., exhibit 1.7. [20] Meeting 12, 10:45. [21] Chapter 1, paragraph 1.91. [22] Ibid., paragraphs 1.52-1.54. [23] Meeting 12, 9:30. [24] Ibid., 10:30. [25] Chapter 1, paragraph 1.61. [26] Chapter 1, paragraph 1.91. [27] Meeting 12, 10:40. [28] Policy on Evaluation, section 7.4. |