[Recorded by Electronic Apparatus]
Tuesday, June 20, 1995
.0908 
[English]
The Vice-Chairman (Mr. Dromisky): Good morning, everyone. I call the meeting to order.
Thank you very much for coming. The witnesses are here and we're ready to roll.
Our first witness this morning is from the Office of the Auditor General of Canada, Ms Maria Barrados. Richard Flageole, a principal with the Audit Operations Branch, is with her.
Thank you very much for appearing before us this morning. We're looking forward to an exciting morning.
We will begin with the presentation by Maria.
Ms Maria Barrados (Assistant Auditor General, Audit Operations Branch, Office of the Auditor General of Canada): Thank you, Mr. Chairman. It is a pleasure to appear before your committee. I'm here this morning to discuss measurement and accountability issues. Our office appeared before your committee last year in March to discuss the results of our audit work in citizenship and immigration.
Mr. Flageole, the principal responsible for this area of audit, is with me today and he will be able to respond to your questions related to these past audits and future audit plans.
Our objective today is to address general questions related to accountability, particularly questions related to measurement of results, program evaluation and audit. We have not done any audit work on the settlement program and therefore are not in a position to comment on this program in any way. Rather, our comments are based on the results of our value for money work on a number of different programs and our recent audit work on effectiveness measurement. I refer particularly to our audit of program evaluation reported in our 1993 report.
[Translation]
As you know, our office serves Parliament through its various committees. Our work focuses on the implementation of existing policies. From time to time, our reports may raise operational issues that potentially impact on policy and related political debate. Our purpose, however, is to focus on operations. It is the prerogative of government and Parliament to question policy.
.0910
We cannot raise additional material not contained in our public reports. We are, however, happy to elaborate on the matters we have raised and to discuss related observations.
A number of the questions forwarded to me by the clerk relate to setting out an accountability framework. I would like to offer some general observations that may be of assistance in your subsequent deliberations.
As Parliament's auditors, we seek to support effective accountability relationships between government and Parliament. These relationships are important because they provide the basis for obtaining assurance that responsibilities conferred and monies provided have been properly and wisely used - with the desired results achieved. The accountability relationship with Parliament requires the government to report on the activities undertaken and the results achieved.
[English]
Making accountability work in government has not always been easy. In the private sector, financial performance and market response to activities that are not financially viable provide the basis for accountability to shareholders. There is no natural equivalent in the public sector. The public sector works through more indirect processes, meaning that more effort has to be given to defining and measuring results, and methods to report on results.
There are three elements fundamental to establishing any accountability framework and I'd like to discuss them. The first is agreeing on what is to be done and achieved; second is having a common understanding on how it is to be done; and third is reporting what has been done and achieved. I've stated these simply, but they're very difficult to put into operation. I'd like to elaborate a little on these three.
First is agreeing on what is to be done and what is to be achieved. As auditors we expect to see program objectives that are clearly stated on what is to be done and a clear statement of what is to be accomplished. Often legislation and policies give very general objectives, but it is not sufficient to report against these in terms of an activity or process to fulfil this obligation. Rather, the challenge in the development of a program becomes taking the general direction and setting up more specific objectives. Specific, clear statements of objectives provide a more realistic basis for assessing performance, measuring results and forming the basis for accountability reporting. Even where managers feel it is unreasonable to set clear objectives, some form of indicators defining direction in the form of goals, targets or benchmarks supporting objectives should be set.
A clear statement of goals or objectives is required to develop measures of performance. Performance measurement is not only essential to good management, it is also very important in accountability reporting.
Just as a side comment, when we talk about performance measurement we generally differentiate two different types of measures. There's the single measure. This is an indicator of performance against some form of a benchmark, target or goal. It's distinguished from evaluation studies that deal with issues of overall impact and assess the difference made by the program. Both forms of measurement are required.
The second element is having a common understanding of how this program is to be delivered. As your committee is aware, there are many different ways to deliver a program. As part of the effort to reform government, government managers are being challenged to think of new and innovative ways to deliver government programs. Increasing emphasis is being placed on using program delivery alternatives, including delivery by third parties. These mechanisms include delivery through partnerships, contracting out, contracting for services, granting licences, or the use of the other enterprises, just to name a few.
.0915
Different types of program delivery have different requirements for reporting on performance as set out in government policy and guidelines. A contribution program, for example, is based on transfer payments that are subject to being accounted for and audited pursuant to a contribution agreement. This is written in the agreement.
A grant program, on the other hand, is not subject to these requirements for accounting and audit but must be assessed in terms of eligibility and entitlement.
For programs delivered directly by government, policies are in place requiring audit and evaluation of their effectiveness. However, for programs delivered by third parties, these requirements, or mechanisms providing equivalent information, need to be built into the agreement.
Whatever form of program delivery is used, or whether the program is delivered by the federal government directly or by others, an important consideration for you committee is how Parliament can be assured that money is wisely spent and the desired results are achieved.
The third element is reporting what has been done and what has been achieved. Part of meaningful reporting includes an accounting for the money spent and a description of what was done. The greatest challenge is reporting on achievements, particularly effectiveness.
As our audit of program evaluation found, it has been difficult for government systematically to measure the effectiveness of its own programs and activities. To obtain sound achievement information, clear objectives are required and a means to collect data needs to be put in place. Often the most cost-effective means of data collection results from building the data collection requirements into a program from its inception.
In Canada, for most new or substantially modified programs, there has been a requirement in submissions to Treasury Board that an evaluation framework be developed as early as possible in a program's design or implementation. An evaluation framework sets out a plan to collect and interpret baseline data, ongoing program performance indicators, and a tentative plan for future evaluation of the program.
We found in our audit in 1993 that departments are not consistently meeting the requirements set out about 12 years ago for implementing these evaluation frameworks.
We have recommended that government managers develop evaluation frameworks at the inception of a program. Under such a framework, possible measures are spelled out and data requirements are defined. Managers can then select the most important and build in the collection of data as part of program operations. In this way performance data can be collected from the beginning of a program and provide the basis for ongoing performance measurement as well as for subsequent program evaluation. Both the ongoing measurement and the eventual program evaluations would form important parts of accountability reporting.
Considerations of accountability frameworks and results measurement in a specific program pose very fundamental questions about the program itself that are often difficult to sort out.
I hope that these general comments will be of assistance in your deliberations. Mr. Flageole and I are at your disposal to provide further clarification or to answer your questions.
Thank you, Mr. Chairman.
The Vice-Chairman (Mr. Dromisky): Thank you very much.
Does Mr. Flageole have a presentation today?
Ms Barrados: No, he doesn't.
The Vice-Chairman (Mr. Dromisky): So you'll take part in the discussion?
Mr. Richard Flageole (Principal, Audit Operations Branch, Office of the Auditor General of Canada): Yes.
The Vice-Chairman (Mr. Dromisky): Fine. Thank you very much.
We'll begin our questioning session with our hon. member Mr. Nunez.
[Translation]
Mr. Nunez (Bourassa): I assure you that it will not be easy to ask questions, because your brief was like a lecture on how to manage public funds, whereas normally, we would expect auditors to tell us in what way the department has not managed its funds well and what the problems are. Is the department efficient? Have you identified problems?
Finally, I would like you to focus a little more on anything abnormal or irregular that you have found. Normally, the opposition closely scrutinises what auditors have to say about the management of government. However, here, all I see are general and vague comments; there's nothing concrete. I would like you to be a bit more specific.
For example, you said that accountability relationships between government and Parliament require the government to report on the activities undertaken and the results achieved. Is that not the case today?
.0920
Why did you say that? Could you not make a comparison with other departments? Are the results with respect to the work plan? What else did you critize?
[English]
The Vice-Chairman (Mr. Dromisky): I will just interject for a moment before you respond.
Mr. Nunez, I think you are going off on a tangent, and it's not the direction in which we were asked to guide our witnesses. I think the witnesses were given a task, and in general terms it was to be on their responsibilities. They are to report on the nature of the federal government's continuing role in immigrant integration, including the most appropriate and equitable distribution of settlement costs. Then they were given five questions, identifying specific areas the standing committee asked them to address.
[Translation]
Mr. Nunez: First of all, I do not know who decided to call these people, why they were called and what the objectives were. I did not receive a document. I never saw the document you read. Someone else may be able to ask more concrete questions, but in my opinion, this is not normally part of the consultations we have undertaken. In my case, it is difficult to ask questions.
[English]
The Vice-Chairman (Mr. Dromisky): Did you want to respond?
Ms Barrados: I'm not so sure I can respond to all the questions, but there are two points I can respond to.
The first is that in our role as Parliament's auditors we are concerned with the accountability relationships government has to Parliament. We see our work as supporting that relationship. In a lot of the audits we do look at the reporting that goes from a particular department to Parliament and the information that is provided. In our audits we also, as you say, look at the operations of programs and make reports about the operations of these programs to Parliament.
The member is quite correct that we have given a very general statement here. That was to assist the committee in addressing the general question.
The observations on specific aspects of measurement do in fact come from the audit we did in 1993 on effectiveness measurement, where we looked at the problems government-wide. That was a government-wide audit that did compare the departments in the Government of Canada.
I can provide more specific comments on any of those things if the member is interested.
The Vice-Chairman (Mr. Dromisky): Thank you.
[Translation]
Mr. Nunez: When the Auditor General appeared before the committee, I asked him how much time had passed since he last reported on the Department of Immigration, and he said five years. So it should happen soon. That is a problem for us, especially on this committee, because there has not been a specific audit of this department for five years.
[English]
Ms Clancy (Halifax): On a point of order, Mr. Chairman, we have a vote, and Ms Barrados and Mr. Flageole have come here.... I'm prepared to be corrected, but the mandate given to our clerk, which she has very well fulfilled, for this meeting and the meeting last week was to bring people before us who would brief us on what we are going to be doing and give general information on the area of settlement. I did not understand that we were to be here today to deal with the Auditor General's department in its overall role, whether it relates to the Department of Citizenship and Immigration or it relates to government. I understood that these people had come here to answer questions about the issue of settlement.
Am I incorrect on this?
.0925
The Vice-Chairman (Mr. Dromisky): I think you are right.
Ms Clancy: Members knew about this. Did we not give a mandate to our clerk to secure witnesses for us who would brief us generally in this area? And is it not unparliamentary for us to be rude to both our clerk and our witnesses?
The Vice-Chairman (Mr. Dromisky): It's a very valid point that has been raised. Mr. Nunez, do you have any further questioning?
[Translation]
Mr. Nunez: Everything that comes from this side is always very valid. But I will give the floor to my colleague.
[English]
The Vice-Chairman (Mr. Dromisky): All right. Mr. Hanger, please.
Mr. Hanger (Calgary Northeast): Mr. Chairman, I too received no line or questions that you referred to earlier, as my colleague Mr. Nunez pointed out. So I don't know specifically where it's all going. But I do have a question I would like to ask.
I noticed that the auditors cannot speak really directly on settlement because there was never any audit work done on the settlement program. I would like to know, if you were assigned to do such a thing, how would you approach that, given the fact that we are looking at, what, a $500 million transfer of money. I would appreciate knowing the approach you would take.
Ms Clancy: Good question.
Ms Barrados: I'll answer that in general and perhaps Mr. Flageole would like to add to these comments.
Our approach in any of these audits is this. Our first step is to try to understand as much as we can of what the program actually is. Now, in this case we have a situation where things are changing very rapidly, and that is a difficulty. So our audit approach would be to look at what has gone on in the past, as much as is documented, and to then look at what is actually going on, and we do that by speaking to officials, looking at files, looking at documents.
After that phase of examination, so that we feel we are familiar with an audit program, we pose to ourselves the questions of outstanding areas of significance that reflect our concerns and our mandate in terms of economy, efficiency, and effectiveness or expenditure of the money, and we would design specific audit programs to test any of those things. Then we would send the auditors in and then we would talk to people and test files, run specific tests, to draw conclusions in those areas.
I don't know, Richard, if you want to add anything to that.
Mr. Flageole: Mr. Chairman, I think Madame Barrados gave a very good summary of how we would proceed. We really have to look at what's significant in the program, what can go wrong, where the risks are in terms of expected results, use of resources, and all of that. That requires a very good knowledge of the program.
As Madame Barrados mentioned, the program is changing and there is quite a renewal initiative going on. So we'll have to take that into account when we plan the next audit of Immigration. It doesn't make it an easy situation to audit something that is constantly evolving. We really have to look at the timing of our work, and that's what we will be doing in the near future.
Mr. Hanger: You have transfer moneys that go to the provinces, and that money is distributed by the provinces. What access do you have when a third or second party distributes funds?
Ms Barrados: That's one of the issues I raised generally in the statement. We are restricted by the terms and conditions that are in those agreements. So if there is an agreement that requires an accounting back and a provision of information, we have access to that information. We are the federal auditors. We don't go into the operations of another jurisdiction. Now I'm going beyond your specific question of what access do we have - we have access there set in the terms and conditions of those agreements that are set and that's where we can audit.
Going on from that a little bit, we have the concern that in any of these agreements, when they are set, there is attention paid to the amount of information that the people in Parliament and the department would like to have when they made the agreements.
Mr. Hanger: The Canada-Québec Accord specifies transfers of funds, and I understand that an audit has never been done on any of those funds that have been moved through that agreement. Would you have any idea why?
.0930
Ms Barrados: That agreement hasn't been audited. It's a contribution agreement, so any audit work we would do would be subject to the terms and conditions of that contribution agreement. My understanding was that when the program and the areas of risk were examined the last time, the auditors chose not to look at it at that time. It is on the list of things they are still to look at.
Mr. Hanger: So you're saying it's incomplete?
Ms Barrados: Yes.
The Vice-Chairman (Mr. Dromisky): Ms Clancy.
Ms Clancy: I'll just continue with my point of order because I might have an apology to make to the members of the opposition. I want to make very clear that at our meeting when we decided to go forward with this, perhaps it's my fault as the most senior member of the committee that I did not specify when we talked to the clerk about who was coming as witnesses. I think we said generally or whoever rather than naming, and this may have gone past the opposition members. Included in this would of course be officials from the Auditor General's department.
But certainly I want to make very clear that the clerk was doing something that this committee, in my understanding, had asked her to do and was not at all going over the heads of Mr. Hanger or Mr. Nunez.
At any rate, having said that, flowing from Mr. Hanger's questions, in the area of settlement your job is particularly difficult because of the federal-provincial aspect. There are agreements with most but not all of the provinces, and some of those agreements are also in various stages of renegotiation. Even if you did have carte blanche to go behind those agreements, given the nature of settlement with the various provinces, would you target maybe one particular project in Toronto or Vancouver or something, or would you go in and look at anything? Is it a kind of spot audit or would you go across the board, leaving aside what I understand to be the problem of the agreements?
Ms Barrados: We are very much guided by those agreements and I can't emphasize how important we think it is for this committee to examine the question of how much information it feels it should be getting about moneys that are being spent here. This committee feels, as members of Parliament, that it should be getting information on money that is being voted by Parliament. That's the first comment I'd like to make.
Specifically in terms of how we audit, on the financial audit side we have both responsibilities. We're auditing for the public accounts as well as doing a value for money audit. On the financial audit side, we will pick specific transactions that are randomly chosen. So you do have the situation where a team of auditors could come in just to look at one or two transactions.
On the value for money side, the effort is made to take a fairly large cut of a program or activity so that there is a meaningful statement made about that whole area. So it would not likely be one particular case or one particular instance. We might draw attention to one case or instance to illustrate a more general point, but our effort would be to make a more general point.
Ms Clancy: Okay, this will be my last question. Given the agreements, are you not almost totally restricted from doing your job?
Ms Barrados: We have to abide by those agreements. If there is an agreement that is in the form of almost a transfer, there's only so much we can do. If there is a grant, and often these agreements are grants, then we can audit the front end of those agreements. Were the proper steps taken in the decision to give the money and did the right people qualify and get the money? That's as far as we can go. There is the question of whether Parliament feels this is sufficient control.
Ms Clancy: But there's more than that; there's also a constitutional question.
Ms Barrados: It depends on the area you're in.
Ms Clancy: But certainly it would not be something where one could dive in without due regard to the constitutional issue.
.0935
Ms Barrados: Absolutely. We abide very much by the agreements that are in place. I commented on the concerns about the amount of control that has to be set for us, and the rules of the game are set for us by Parliament.
Mr. Hanger: Apart from the agreement, are there other restrictions that prevent you from tracking funds? When you strike an agreement with a province, are there other restrictions that prevent you, apart from what's written in the agreement, from tracking those funds as they are distributed by that party?
Ms Barrados: I have to think carefully about your question. Our restrictions are set by the agreements and the protocols that are laid out by the Government of Canada. Can you think of any other instance on the financial side, Richard?
Mr. Hanger: It could literally open the door wide if it were stated in the agreement that you could follow through and track everything, regardless of where it may go.
Ms Barrados: Yes. There's another aspect to this that I would like to draw to your attention. The agreements don't necessarily mean you send the auditor in to audit everyone. There is the question of our own resources as well. We also have a limit on the resources and the amount of auditing we can do. We can't be driving through every agreement everywhere.
The other possibility is that parts of the agreements require reporting back, so there is some form of reporting back on the basis of what was accomplished as opposed to having the auditor follow to track the money.
Mr. Hanger: You would have to understand the program thoroughly to decide in your own mind whether that money was being spent wisely or not.
Ms Barrados: Are you talking about an approach to audit?
Mr. Hanger: Yes.
Ms Barrados: Our approach to audit is very much one of looking at that program and then almost setting up a series of testable questions. Based on what we see in the operations, we pose the questions and then decide what kind of evidence we should gather to answer those questions. That has to do with the proper expenditure of the money, the efficiency and the effectiveness. The most difficult part is the effectiveness.
Mr. Assadourian (Don Valley North): Have you had a chance to audit the agreement between Quebec and Canada, signed a few years ago, to see how well it is performing?
Ms Barrados: No, we have not.
Mr. Assadourian: Is it not in the agreement for the federal government to audit the money it transfers to Quebec?
Ms Barrados: My understanding is that it is not.
Ms Clancy: You talked about third-party program delivery through partners. How do we make partners accountable?
Ms Barrados: That's the interesting question. This is a new area, and I think we're all exploring our way.
The processes we have in place are in the agreements we make with the partners, and we do address the three elements I was talking about. We clearly specify what we expect them to do. In saying that it sounds easy, but it's really quite difficult when you think of general programs and what you really expect them to do. How do you expect them to do that without being directional on process? I'm not advocating tight control over process because I think we all agree we want to move away from that.
Third, what do you require from them in terms of an accounting back for the money spent and accounting back on the performance side? The more specific that can be in terms of the expectation, the better chance that something will come back.
.0940
But I have to add that although it sounds simple, in practice it has proven to be very difficult to do, even within the government when government managers had all the protocols, had all the things in their hands. It's a difficult enterprise.
Ms Clancy: You may disagree, but I think it's easier when we're talking about business partnerships - I'm thinking of the Canada-Nova Scotia Development Agreement or something like that - because in most cases the parameters are fairly straightforward.
Am I right in believing that when you're dealing with partnering, say, with non-governmental organizations in the standard form of NGOs, particularly in the area of settlement, because of the necessary flexibility that's what makes it difficult, as opposed to almost a built-in obsolescence?
Ms Barrados: Even in the case of business partnerships, there are difficult sets of questions. What comes to mind in terms of the areas we've audited in the past is Hibernia.
Ms Clancy: A perfect project.
Ms Barrados: That's the member's comment.
We can fairly easily get our hands on what kinds of money have been spent and how that money has been paid, but what remains on the table is really what the government accomplished by doing this, quite apart from the edifice that is built and the immediate activities. What were the economic benefits here? This is the kind of question we have been posing all the time, because we think that's an important question when assessing that program.
So, yes, there are elements of it that are easier, but there are the questions of why public money is being used for these purposes. That's often a difficult question to analytically -
Ms Clancy: Your answer sheds a lot of light on the relationship between your department and the government when you consider it, because clearly governments frequently - perhaps not frequently, but perhaps more often than the Auditor General's department would like - cannot be quite as obsessed with the bottom line in the general overall picture. That is another editorial comment.
Mr. Knutson (Elgin - Norfolk): You emphasized the importance of building accountability measures right from the start. Are you advising Citizenship and Immigration on how to oversee settlement renewal? Or would you have an opinion, say, on what percentage of their budget for settlement they should be spending on auditing their own contracts?
Ms Barrados: Specific advice that I would give here - and I haven't been asked to give specific advice to the department, although I usually will give this kind of advice to anyone who asks me - is that sufficient time be taken in the development of the program. That would mean including the people from the evaluation and audit group at the very beginning, because these requirements for measurement can be built in much more easily at the beginning than after the fact. Particularly in the instance of these programs, as I understand them, the expectation is to go into local areas and get the money to as many different people as you possibly can. So a lot of thought has to be given up front on that. I think that is worth the investment.
My experience has been that it becomes very expensive and often very difficult to do these kinds of measures after the fact, particularly if you have populations who are going to be using these benefits for short periods of time, which I understand can sometimes be the case; they're in and they're gone, and they're not necessarily staying in the same place, either.
So it has to be done up front.
Mr. Hanger: In paragraph 21, on an evaluation framework, your last sentence reads:
- We found that departments are not consistently meeting the requirements set out twelve years
ago for a framework.
.0945
Ms Barrados: This comment was made in our 1993 audit report. It refers back to a policy that was in place in 1981, and of course at that time the immigration programs were not in their own department; they were part of other departments. It was a general observation, and we did it by counting. We counted the number of these frameworks in specific programs we examined, and there were some; there were not very many. As I recall, and I can go back and check, I don't remember seeing any in Immigration. But I will check.
Mr. Hanger: So you're saying the statement that the department is not meeting those requirements doesn't apply to Immigration.
Ms Barrados: When that statement was made, there was no Department of Immigration.
Mr. Hanger: Okay. I wonder what the relevance is, then, to what we've been talking about here.
Ms Barrados: I think it's highly relevant. The comment was made on a government-wide basis, because it's difficult to do and it's often not done. What happens is that people who are formulating policy get into the policy ideas and they get into all the things they have to do, but they don't worry about the measurement side, and that's something we see as a general case. Particularly here there's a very good opportunity to start building these things in.
The Vice-Chairman (Mr. Dromisky): Mrs. Bakopanos, please.
Mrs. Bakopanos (Saint-Denis): I'd like to go back to the accountability. That seems to be the key word these days from both sides of the House of Commons. Considering the fact that if we continue along the lines of what we're thinking in terms of settlement programs and transferring, let's say, the responsibility to the local community groups or non-governmental organizations, considering the fact that often enough there are limited resources in terms of their keeping books and keeping the type of reports that would be necessary for accountability, would you have any recommendations along these lines?
You did mention earlier that the government should get in at the beginning and set the goals and set the framework for that type of accountability and performance measurements, if we want to call them that. Taking it one step down to the level of the organizations and the community groups, how could we set up a program, then, in which at the very beginning they had the tools in order to self-evaluate their own performance?
Ms Barrados: You pose a set of difficult questions.
Mrs. Bakopanos: I know. I apologize, but that is the type of question we'll be dealing with.
Ms Barrados: Yes, and I'm not sure I have easy answers.
Mrs. Bakopanos: It's your opinion basically I'm asking for.
Ms Barrados: Some of the experience that may be somewhat analogous is actually in the CIDA process, where what you have is a number of projects in different world settings that are highly variable, and one of the characteristics is that the people who are delivering these programs generally are not preoccupied with measurement issues and tend not to be terribly interested in getting measures for some form of report because they are interested in providing the service.
Our experience has been - and you may want to consider this in some of your deliberations - that some assistance has to be provided to these groups in the form of some kind of guidance explaining what is required; but tell them at the beginning. It should be simple; it should not be too complex. It's better if they can do it themselves, if they can be given some kind of assistance, and I'm not suggesting a big infrastructure here at all, but some way that they could be given some kind of assistance to meet these requirements, keep it simple and keep it to what really matters.
I think the experience has been to have other auditors or evaluators coming in afterwards. It really becomes quite onerous and quite heavy, particularly if you have many different small projects. I think audit and evaluation has to come in on a more selective basis, and if there could be general guidance that is simple and to the point that provides the information people seek, that would be the best approach.
.0950
Mrs. Bakopanos: When would your department be able to provide - I'm thinking of a questionnaire or a form in which there are benchmarks and performance measurements. I think your office could play a very good role in setting up some sort of framework, some sort of guide for the local community organizations to use in evaluating themselves every three months, every six months or perhaps every year. That would depend, of course, on the type of settlement program.
I can see how you can probably not only get in at the top but also in at that level by assisting those groups. They really don't have the resources to be able to set up some sort of performance self-evaluation.
Ms Barrados: There are a couple of things you mentioned here and there's a distinction I would like to make. A benchmark is a target that you would be giving these groups. These people delivering and designing the program would say, this is what we want you to accomplish and you will report against that.
That puts a considerable obligation on the people designing and delivering. The obligation of the groups is to respond to how well they did against that benchmark. If that thought is given to designing the benchmark by the people delivering the program, it makes it easier for the groups to focus on what they should report against. Your worry is a different kind of worry - that the information you are getting is reasonably reliable and that it was honestly collected.
Your other question was related to the assistance we could provide. We are always ready to assist and provide advice in terms of expert advice.
We're not really in a position to start designing things as part of government operations. There is a lot of experience in the department and in the government that would help, such as departments like CIDA, where there is experience in terms of working with a number of these different kinds of projects and getting assessment information and accountability information from them.
They are finding it difficult. These are not easy. You have to work at them.
The Vice-Chairman (Mr. Dromisky): Thank you very much for your clarification on some of the issues this morning. As well, we appreciate your very comprehensive report. Thank you for the information you shared with us.
Since there are no further questions and no other comments, the meeting is adjourned. Thank you.