PACP Committee Report

INTRODUCTION
            Effectiveness evaluation is a tool used by managers to assess the extent to which government programs are meeting their objectives. It involves the systematic collection and analysis of evidence on program outcomes. The information gathered through evaluations can be used to make judgments about the relevance, performance, and value-for-money of programs, and thereby support decisions on policy direction, program improvement, and expenditure allocations. Evaluations also help departments demonstrate to parliamentarians and Canadians that government programs are delivering their intended results. Sound information on program effectiveness is particularly important in light of recent government initiatives to freeze operational spending and conduct strategic reviews of government programs.

            The importance of evaluation in the federal government has been emphasized through recent legislative and policy changes. The 2006 Federal Accountability Act introduced an amendment to the Financial Administration Act that requires all non-statutory grants and contributions programs to be reviewed every five years for relevance and effectiveness. ^[1]  Under Treasury Board’s Policy on Evaluation, which was updated in 2009, large departments must ensure that all direct program spending is evaluated every five years.^[2] In order to meet these requirements, it is vital that departments have adequate capacity, support, and systems in place to complete the evaluations required by legislation and Treasury Board policy.

            The Office of the Auditor General (OAG) included in its Fall 2009 Report an audit on how evaluation units in six departments (Agriculture and Agri-Food Canada, Canadian Heritage, Citizenship and Immigration Canada, Environment Canada, Fisheries and Oceans Canada, and Human Resources and Skills Development Canada) identified and responded to various needs for effectiveness evaluation, and whether the departments had developed the required capacity to respond to those needs.^[3] The OAG also examined the oversight and support role of the Treasury Board of Canada Secretariat (the Secretariat) in monitoring and improving the evaluation function in the federal government.

            As the Public Accounts Committee has long supported the development of effectiveness evaluations in the federal government, it held a hearing on this audit on 4 May 2010.^[4] From the Office of the Auditor General the Committee met with: Sheila Fraser, Auditor General of Canada; Neil Maxwell, Assistant Auditor General; and Tom Wileman, Principal. From the Treasury Board of Canada Secretariat the Committee met with: Michelle d'Auray, Secretary of the Treasury Board of Canada, and Alister Smith, Assistant Secretary, Expenditure Management Sector. The Committee also met with representatives from two of the departments audited. From Citizenship and Immigration Canada the Committee met with: Neil Yeates, Deputy Minister, and Elizabeth Ruddick, Director General, Research and Evaluation. From Environment Canada the Committee met with: Ian Shugart, Deputy Minister, and William Blois, Associate Director, Audit and Evaluation Branch.

ANNUAL REPORT
            The OAG’s audit on effectiveness evaluation in the federal government found that:

• during the audit period, a relatively low proportion of program expenses were evaluated;
• many of the evaluations examined did not adequately assess program effectiveness due to inadequate data;
• departments have concerns about their capacity to meet the requirement to evaluate all direct program spending;
• the Secretariat has not provided sustained support for effectiveness evaluation; and
• with the exception of Environment Canada, departments do not regularly identify and address weaknesses in effectiveness evaluation.^[5]

            The Committee decided to meet with officials from Environment Canada and Citizenship and Immigration Canada, as it would have been too unwieldy to invite representatives from all six departments audited. These two departments provided action plans to the Committee prior to the hearing on how they intend to address the OAG’s recommendations, and Agriculture and Agri-Food Canada and Fisheries and Oceans Canada provided action plans after the hearing. The Committee trusts that these departments will implement these plans and work to improve their effectiveness evaluation capacity. However, the Committee did not receive action plans from Canadian Heritage or Human Resources and Skills Development Canada. Thus the Committee recommends:

RECOMMENDATION 1
That Canadian Heritage and Human Resources and Skills Development Canada provide the Public Accounts Committee with action plans by 31 December 2010 of how they intend to address the Office of the Auditor General’s recommendations included in Chapter 1 of the Fall 2009 Report.

            The audit findings also raise the concern that effectiveness evaluation needs to be improved in all departments and agencies and not just the ones audited. Within the federal government, it is a central agency—the Treasury Board of Canada Secretariat—that is responsible for monitoring the evaluation function across the government. The Secretariat developed the updated Policy on Evaluation, which was intended to address gaps in the coverage and nature of evaluations, the governance and independence of the evaluation function, and the lack of clarity around the collection of data. The Secretariat monitors the evaluation function through a number of means, such as an annual capacity assessment survey, periodic reviews of the quality of evaluation reports, and Management Accountability Framework (MAF) assessments of the quality and use of evaluations in departments. The Secretariat also maintains a Centre of Excellence for Evaluation (the Centre) that provides guidance and support to departments on evaluation.

            The audit found that the Centre had not made adequate progress in developing tools to help departments address the long-standing problem of insufficient data for the evaluation of effectiveness. At the time of the audit, the Secretariat also had not issued guidance to departments to help them implement the Policy on Evaluation.^[6] The Secretary of the Treasury Board, Michelle d'Auray, told the Committee that the Centre has since issued draft guidance on developing departmental evaluation plans and performance measurement strategies. The Centre has also set up an evaluation community of practice for exchanges of best practices.^[7] Alister Smith, Assistant Secretary, leads the Secretariat’s evaluation activities and indicated that these guidance documents should be finalized by the end of the fiscal year.^[8] The Auditor General told the Committee that she was pleased to hear about the actions the government has taken since the completion of the audit.^[9]

            Ordinarily, the Committee tracks departmental progress in addressing the OAG’s recommendations through status reports. In this instance, the Committee notes that the Secretariat has in the past published a report on the evaluation function, called “The Health of the Evaluation Function in the Government of Canada: Report for Fiscal Year 2004-2005.” According to the audit, this is the only such report published; though, the Secretariat continues to conduct an annual capacity assessment survey.^[10] In her opening statement, the Secretary stated:
The Auditor General also recommended that the Treasury Board Secretariat should do more to monitor and support departments to help them identify priorities for improvement. This is addressed under the new policy which calls on the Treasury Board Secretariat to provide functional leadership for evaluation across the government. This includes regular monitoring and annual reporting to Treasury Board on the health of the evaluation function and our first report will be issued before the end of 2010-11.^[11]

It is not clear if the Secretariat intends to make this annual report available to the public. The Committee believes that this report should be made public as it would provide a valuable accountability tool for parliamentarians and Canadians to monitor progress in implementing the government’s new evaluation requirements. Thus, the Committee recommends:

 RECOMMENDATION 2
That beginning in 2010-2011 the Treasury Board of Canada Secretariat prepare an annual report on the status of the evaluation function and make this report available publicly.

RECOMMENDATION 3
That the Treasury Board of Canada Secretariat include in the annual report on the status of the evaluation function, an update on its oversight and support activities for effectiveness evaluation.

PERFORMANCE INFORMATION
            In order to properly evaluate program effectiveness, it is necessary to have access to sufficient and reliable data on program performance. The OAG reviewed a sample of 23 effectiveness evaluations, and 17 of these evaluations (or 74%) explicitly stated that program performance information was lacking because data was unavailable or was not sufficiently reliable.^[12] In 9 of these cases, the evaluations were unable to properly assess program success and effectiveness due to data limitations.

            The Deputy Minister of Citizenship and Immigration Canada, Neil Yeates, acknowledged that collecting performance information is a large challenge for his department, in part because data is often very expensive to collect.^[13] The Deputy Minister of Environment Canada, Ian Shugart, stated that often the issue is not that there is no performance data, but that the data is not sufficient to cover all of the evaluation questions. His department has a performance measurement framework, but it is not complete.^[14]

            The Policy on Evaluation requires departments to ensure that ongoing performance measurement is implemented so that sufficient performance information is available to support program evaluation.^[15] The Secretariat monitors departmental management through the annual MAF assessments, which include a section on the “utility of the corporate performance framework.”^[16] For the 2008 assessments, three of the six departments included in the audit–Citizenship and Immigration Canada, Canadian Heritage, and Agriculture and Agri-Food Canada—received the comment, “The performance indicators are not clear and cannot be used for data collection to provide reliable insight into program effectiveness.” Environment Canada and Fisheries and Oceans Canada received the comment, “The organization has developed a weak performance measurement framework.” And Human Resources and Skills Development Canada received the comment, “An incomplete or inadequate performance measurement framework has been developed.” Four of these departments received a rating of “acceptable” for the utility of the corporate performance framework, and two received a rating of “opportunity for improvement.” As a strong performance measurement framework is an important precondition for meaningful evaluations, the Committee recommends

RECOMMENDATION 4
That Environment Canada, Fisheries and Oceans Canada and Human Resources and Skills Development Canada provide the Public Accounts Committee with a strategy by 31 January 2011 of how they intend to improve their performance measurement frameworks.

            It is clear that departments will have to make significant improvements to their performance measurement frameworks and subsequent data collection in order to enable meaningful effectiveness evaluations. The ongoing collection and assessment of reliable performance information is also important for managers to monitor program performance and for departments to be able to report publicly on the results their programs are achieving. While the Secretariat is monitoring this issue through its MAF assessments, these do not indicate the extent to which departments have sufficient performance information for the purposes of effectiveness evaluations. As the Committee would like to ensure that evaluations are not being hindered by a lack of performance information, it recommends:

RECOMMENDATION 5
That the Treasury Board of Canada Secretariat include in the annual report on the status of the evaluation function, information on the number of effectiveness evaluations, and in which departments, that had difficulty assessing program success due to insufficient performance information.

MEETING LEGISLATIVE/POLICY REQUIREMENTS
            As noted earlier, under recent legislative and policy changes, all federal departments must evaluate all grants and contributions programs and large departments must evaluate all direct program spending over a period of five years. The requirements of the Policy on Evaluation will come into effect following a period of transition. The first five-year cycle covers 2013-2014 to 2017-2018.

            The audit found that departments had developed risk-based evaluation plans during the audit period.^[17] However, the proportion of program expenses evaluated between 2004-2005 and 2007-2008 ranged from an annual average of 5% to 13%.^[18] If departments are to achieve the requirement of evaluating all direct program spending over a five-year cycle, then they will have to have an annual average of 20%. While there is still time to increase evaluation capacity, departments have a substantial way to go before they will reach the target for the annual proportion of program spending evaluated. As the Committee would like to monitor progress to ensure that departments are on track to meet the evaluation requirements, it recommends:

RECOMMENDATION 6
That the Treasury Board of Canada Secretariat include in its annual report on the status of the evaluation function, information on the percentage of direct program spending and grants and contributions programs covered by evaluations in the fiscal year, as well as the percentage evaluated over the five year cycle.

STAFFING
            The audit found that evaluation unit staffing had increased over the audit period.^[19] The evaluation unit of Citizenship and Immigration Canada had increased from 3.3 full-time equivalents in 2004-2005 to 12.5 in 2008-2009, and the Deputy Minister indicated that they plan to increase this to 20.^[20] The evaluation unit at Environment Canada had also expanded significantly from 4 full-time equivalents in 2004-2005 to 10 in 2008-2009, and Environment Canada may have to double this number in the next few years.^[21] Across large departments the increase has been from 176.3 full-time equivalents in 2004-2005 to 296.7 in 2008-2009.

            However, if all federal departments need to make a similar expansion in their evaluation units, then it may be difficult to find sufficient qualified and experienced evaluators. The audit noted that the shortage of experienced program evaluators in the federal government has been a longstanding concern, which has affected the ability of departments to hire the people they need and has led to evaluators being hired away by other evaluation units.^[22] In order to ensure that they have sufficient staff, departments will have to prepare recruitment and retention strategies for their evaluation units.

            The Secretary of the Treasury Board told the Committee that there are currently over 500 evaluators working in the federal government,^[23] and Alister Smith described how the Secretariat is working with universities and the Canada School of Public Service to help build up the community of qualified evaluators.^[24] Nonetheless, the Committee remains concerned about the ability of departments to attract and retain qualified and experienced evaluators, as this will have a significant impact on their ability to complete the additional evaluations now required. The Committee would like to monitor this issue and recommends:

RECOMMENDATION 7
That the Treasury Board of Canada Secretariat include in its annual report on the status of the evaluation function, information on the number of evaluators employed in large departments in the federal government.

            A further concern related to staffing raised by the audit was the heavy reliance on contractors. The audit found that about 90% of the evaluations completed during the audit period were wholly or partially conducted by contractors.^[25] While there will always be an important role for contractors, as they can offer expertise that is not available internally and can provide supplementary capacity, an excessive reliance on contractors may inhibit the ability of internal evaluation units becoming subject-matter experts, instead causing them to spend most of their time managing contracts. The Committee trusts that this reliance on contractors will diminish as departments build up their evaluation units. Since the Committee would like to ensure that this is the case, it recommends:

RECOMMENDATION 8
That the Treasury Board of Canada Secretariat include in its annual report on the status of the evaluation function information on the percentage of evaluations that are conducted by contractors.

FOLLOW-UP
            The legislative and policy requirements for effectiveness evaluation are considerable. The audit findings, such as a lack of performance information, insufficient capacity, and a shortage of experienced evaluators indicate that departments may have difficulty meeting these requirements. While there remains considerable time for departments to plan and prepare, the audit notes that earlier requirements for full evaluation coverage were never met.^[26] It is not clear what would happen should departments not be able to achieve full evaluation coverage, especially the legislated requirement to review all grants and contributions programs every five years. Departments may sacrifice evaluation depth in order to achieve full coverage. The Deputy Minister of Environment Canada told the Committee that, “We may be flexible in how we implement this policy. We may cover a broad range area of programs in one evaluation rather than each individual program, which would be an efficient way of doing it.”^[27] This approach raises some concerns because departments may achieve full coverage at the cost of in-depth, meaningful evaluations of specific programs, thereby undermining the original goal of providing managers with assessments of all direct program spending.

            Given the current weaknesses in effectiveness evaluation identified by the audit and the possible difficulties in meeting the legislative and policy requirements, the Committee believes that it would be valuable to have a follow-up audit completed before the end of the first five year cycle in 2017-2018. The Committee notes that significant increases have been made in the financial and human resources devoted to the evaluation function within departments. The Committee believes that it is important that the government achieves value for money with these resources and that they are used in an economical and efficient manner.  Thus, the Committee recommends:

RECOMMENDATION 9
That the Office of the Auditor General conduct a follow-up audit within five years of the initial audit on whether departments are able to meet the requirements for effectiveness evaluation, and whether they are conducting evaluations in an economical and efficient manner.

            The Committee also notes that the Policy on Evaluation requires the Secretariat to ensure that an evaluation of the policy is conducted every five years.^[28] This evaluation should help determine whether the increased requirements for evaluations on program effectiveness are indeed effective.

CONCLUSION
            Effectiveness evaluations provide managers with objective, evidence-based assessments of the extent to which government programs are achieving their intended results. As such, effectiveness evaluations are very important for making good, informed decisions about program design and where to allocate resources. The Committee has long encouraged the development of effectiveness evaluation within the federal government and is pleased that the government has strengthened the requirements for evaluation. However, the findings of the OAG’s audit demonstrate that there are a number of weaknesses that need to be addressed if departments are to meet these requirements. Through this report, the Committee has requested the publication of various pieces of information that will help it monitor the ability of government departments to address these weaknesses and build a robust evaluation function.