INTRODUCTION

Statistical data are used by many different organizations: governments use them to support decision‑making and to provide information about program outcomes; private sector businesses and non-governmental organizations use them for planning and marketing purposes; and academics use them in their research.

 Statistics Canada, the federal government’s statistical agency, has two main objectives:

• to provide statistical information and analysis about Canada’s economic and social structure; and
• to promote sound statistical standards and practices.

In support of these objectives, Statistics Canada has developed a Quality Assurance Framework and Quality Guidelines, which encompass six dimensions: accuracy, timeliness, accessibility, coherence, relevance and interpretability.

In its Spring 2014 Report, the Office of the Auditor General (OAG) released a performance audit that examined whether Statistics Canada:

• assessed the quality of selected key information it generates on Canada’s socio-economic conditions;
• assessed whether resources are used efficiently to produce selected data products; and
• established, based on user needs, priorities for existing and potential new products.^[1]
• In its examination, the OAG focused on four data products:
• Consumer Price Index;
• Labour Force Survey;
• National Household Survey; and
• Survey of Employment, Payrolls and Hours.

The House of Commons Standing Committee on Public Accounts (the Committee) held a hearing on this audit on 30 October 2014.[2] From the OAG, the Committee met with Michael Ferguson, Auditor General of Canada; John Affleck, Principal; and Colin Meredith, Director. From Statistics Canada, the Committee met with Wayne Smith, Chief Statistician of Canada, and Stéphane Dufour, Assistant Chief Statistician, Corporate Services and Chief Financial Officer.

QUALITY FRAMEWORK

Statistics Canada’s quality framework is designed to ensure that its statistical products are of sufficient quality for their intended uses. The OAG examined whether Statistics Canada had met the standards of the six elements of its quality framework for the four data products examined.

A.   Accuracy

Statistics Canada defines accuracy as the degree to which its information correctly describes the phenomena it is designed to measure. The OAG found that Statistics Canada undertook steps to ensure accuracy for the sample of data releases examined. Statistics Canada was responsive to changes in the population sampled, implemented sampling plans, trained and supervised data collection staff, and created and used quantitative indicators of data quality.[3]

For example, in order to mitigate the risks to data quality of moving from the mandatory long-form census questionnaire in 2006 to the voluntary National Household Survey in 2011, Statistics Canada increased the sample size from 3 million to 4.5 million households.[4] It also took steps to follow-up with households that did not initially complete the survey and focused on geographic areas known to include high proportions of particular population subgroups, such as low-income earners.[5]

Michael Ferguson, Auditor General of Canada, explained the OAG’s conclusion on accuracy, saying, “They completed all of the steps to ensure the accuracy, so in that particular area we were satisfied. Again, we indicate as well that there were the changes to the national household survey, and that caused Statistics Canada to take some other measures.”[6]

Wayne Smith, Chief Statistician of Canada, told the Committee that response rates can have two effects on the quality of data: the statistical variability of the estimates and non-response bias. He acknowledged that while he would have preferred a higher response rate to the National Household Survey, adjusting the sample size, from 20% to 30% of Canadian households, resulted in a similar number of responses; in fact, Statistics Canada received approximately 200,000 more responses from the National Household Survey in 2011 than from the long-form census questionnaire in 2006. So, the statistical variability of the estimates was not a concern. With respect to non-response bias, Mr. Smith said, “Essentially the conclusion was that, of the various claims that had been made about all the groups that would be under-represented, we found no real evidence that had occurred. We did find a slight discrepancy with regard to certain income data for low-income families. We identified that in information we published at the time the data was released.”[7]

B. Timeliness

The timeliness of statistical information refers to the delay between the point in time to which the information pertains and the date the information is available. The OAG found that Statistics Canada announced the release dates for all four data products in advance.[8] The releases occurred on the planned dates for all products, except the third release for the National Household Survey, which was postponed for four weeks due to a data processing error.[9]

With respect to timeliness, Mr. Ferguson commented, “We were satisfied with the time limits aspect of their quality framework. The one item we identified was that one release date was missed, but it was by only four weeks.”[10]

C. Accessibility

Accessibility is the ease with which statistical information can be obtained by the user. The OAG found that Statistics Canada makes data accessible and is planning further improvements. The OAG also noted that Statistics Canada is moving away from publishing its data in reports toward disseminating it on the Internet.[11] However, users told the OAG that Statistics Canada’s website is challenging to navigate–user satisfaction with the website averages 67%. Statistics Canada is working to increase user satisfaction by developing a new dissemination model to make its data easier to find.[12]

Mr. Ferguson summarized the OAG’s findings, “We found that they did make the data accessible. They were using the Internet more. They had centres for researchers. They stopped charging for some products. But, again, this is the area we talked about in which some users said the website was challenging to navigate.”[13]

Mr. Smith provided the Committee with more detail about users’ concerns with Statistics Canada’s website, stating:

[T]here is a lot of dissatisfaction with our search engine. People feel that if they type in words they don't feel it gets them directly to the kinds of information they are looking for. They find that the structure to drill down into the website is awkward and it's using terminology that's not familiar to them. This information is currently being used to try to do a comprehensive redesign of our website that will make the flow much easier.[14]

He later added, “Right at the moment we have several different databases that people have to consult distinctly. The idea is to merge them into one; to have one simple, straightforward interface; and to make the search engine more responsive to people's needs.”[15]

D. Coherence

The coherence of statistical data reflects the degree to which it can be brought together with other statistical information within a broad analytic framework. It reflects the comparability of related data for the same point in time, of the same data for different points in time and of Canadian and international data. The OAG found that Statistics Canada assessed the coherence of its data, and it adhered to international standards, definitions and classifications.[16] It also compared data from multiple sources to assess the coherence of its data products.[17]

Statistics Canada determined that the change in method from a mandatory census to a voluntary survey reduced the comparability of data from the 2006 Census long-form questionnaire to the 2011 National Household Survey.[18] As a result, Statistics Canada informed users that they should use caution when making comparisons using those data.[19]

Mr. Smith maintained that the National Household Survey contained useful information, telling the Committee that:

While there has been some erosion of data quality due to lower response rates, the National Household Survey nonetheless produced a massive data base of robust information at all levels of geography and for many small populations. To seek, as some have, to dissuade Canadians from using this rich and powerful data source does them a terrible disservice.[20]

E. Relevance

The relevance of statistical information reflects the degree to which it meets the needs of clients. The OAG found evidence of extensive use by diverse users of the data products examined.[21] Statistics Canada uses a variety of mechanisms to obtain user input and feedback on its statistical products, such as advisory and consultative committees, and bilateral relationships and meetings with users.[22] The OAG noted that Statistics Canada primarily consults with the federal, provincial and territorial governments and less with the private sector, municipalities and non-governmental organizations.[23] The OAG recommended that Statistics Canada obtain, document and analyze feedback from the full range of its data users.[24]

In response to the recommendation, Mr. Smith told the Committee that “Statistics Canada will in future systematically document this outreach process and demonstrate clearly how it is taken into account in the agency's program planning.”[25] He also noted that he meets regularly with business organizations.[26]

The OAG also observed that Statistics Canada’s subject matter advisory committees tended to be from academia with little representation from the private or non-governmental sectors and many of the committee members had long tenure on the committees.[27] The OAG recommended that Statistics Canada implement a formal and routine process to ensure the full range of users is represented on advisory committees and to ensure turnover in membership.[28]

Mr. Smith informed the Committee that Statistics Canada’s advisory committees play two roles. Some committees provide expert advice on methodology and other committees provide advice on the design of their programs, specific surveys and data gaps.[29] Statistics Canada is looking to broaden the representation on the latter type of committees. Mr. Smith noted that Statistics Canada has begun to formalize the renewal process for the agency's numerous advisory committees, to ensure they are properly constituted given their mandates, thus helping the agency remain aware of and responsive to evolving needs and statistical methods.”[30] Statistics Canada is also implementing a three-year term limit on membership.[31]

With respect to the National Statistical Council, which provides advice on priorities, Mr. Smith said:

To be advised effectively on priorities—on data priorities, on program priorities generally—I need a set of voices that are representative of the stakeholders in the statistical system. To have a committee that is dominated, as the committee has been to some extent in the past, by academics, gives the discussion a flavour that is somewhat biased in terms of certain interests and certain priorities. By bringing in more business members, by bringing in representatives from municipal governments, I get a more balanced view. I hear more views and more voices. The result, therefore, is more balanced in terms of the interests of all Canadians.[32]

The OAG observed that the needs for data from small geographic areas and subpopulations were not being fully met. For example, data from the Survey of Employment, Payrolls and Hours is only released at the national, provincial and territorial levels.[33] Thus, it is not possible to determine from the data on job vacancies where in a province or territory those jobs are located.[34] Additionally, Statistics Canada withheld data from the National Household Survey for 25% of census subdivisions due to quality concerns; data for an additional 15% of subdivisions were not released for confidentiality and other reasons.[35] Thus, reliable data was unavailable from the National Household Survey for 3% of the population, compared to less than 1% of the population for the 2006 Census.[36] Users outside the federal government told the OAG that data is of limited use when it is not available for their location.[37] The OAG recommended that Statistics Canada assess the feasibility of more fully addressing user needs for data from small areas and subpopulations.[38]

In response to concerns about the inability to access reliable data for some census subdivisions, Mr. Smith observed that the issues primarily related to small and remote communities. He said:  

The quality concerns really occur in provinces and in areas where there are very small municipalities of this nature, or of even a few hundred people. Certainly all of it, or almost all of it, is in municipalities below 5,000, and most of it is below 1,000. In provinces where they've done amalgamations of their municipalities into larger municipalities, we didn't really have a significant problem. In provinces where there are still large numbers of very small municipalities, that's where we encountered the difficulty. Saskatchewan, for example, was very badly affected by this.[39]

He also acknowledged that Statistics Canada was unable to get information from households on 36 Aboriginal reserves because they refused access to Statistics Canada employees.[40] Additionally, the costs associated with the number and geographic location of First Nations reserves make it difficult to publish and keep up-to-date data on the employment rate on reserves.[41]

Mr. Smith maintained that the data could still be used, but in an aggregated way. He commented, “To be clear, the data for those areas is included in the national picture and in the provincial picture. It's included in the census division picture. It's quite possible by combining several areas to generate an area for which we can provide data. That's been done.”[42] He added, “I don't want to trivialize the fact that a significant number of municipalities that would have expected to receive data didn't. It's not something I wanted to happen, and it's something I'm going to seek to avoid to the greatest extent possible in 2016.”[43]

To address the need for data for small areas and small populations, Statistics Canada is working to augment its coverage by using new methodologies and data sources. Mr. Smith said:

Statistics Canada is making great strides by developing and exploiting state-of-the-art techniques, such as micro-data simulation and smaller re-estimation, to extract the maximum value from existing datasets. Our growing use of administrative data is also increasing our ability to provide data for small areas and populations. These techniques produce, at affordable cost, estimates at a detail beyond that of a single survey, without imposing an additional response burden on Canadians and their businesses.[44]

In order to address the need for job vacancy information for smaller regions, Statistics Canada is working with Employment and Social Development Canada to develop a new survey. Mr. Smith informed the Committee that:

We have designed a survey that will allow us to publish data by occupation, with wages, at a fairly detailed geographical level. That survey should largely meet the kinds of needs that have been expressed in terms of information. It represents, as Minister Kenney indicated in a public statement, an investment of about $14 million, which is why Statistics Canada could never have undertaken it on its own.[45]

Statistics Canada anticipates collecting data for this survey at the beginning of 2015 and pushing data later in that year.[46] Mr. Smith also told the Committee that Statistics Canada is planning an annual national wage survey, with the intention of publishing data by occupation at areas below the provincial level.[47]

F. Interpretability

Interpretability refers to the need to give users sufficient information to interpret statistical data and thereby understand and use it properly. The OAG found that Statistics Canada makes information available to help users interpret data; however, the guidance was not always current and complete.[48] The OAG recommended that Statistics Canada ensure that its guidance is comprehensive and up to date.[49]

In its action plan provided to the Committee, Statistics Canada indicated that it launched a review of its program documentation and by January 2015, it will identify which programs require a more detailed and updated technical user guide.[50]

EFFICIENT USE OF RESOURCES

Statistics Canada is working to reduce its ongoing operating costs with minimal cuts to statistical and analytical programs. The OAG noted that Statistics Canada had identified a number of initiatives to make more efficient use of resources.[51] Of the 37 initiatives planned for 2010–2011 to 2012–2013, 15 had been implemented, resulting in savings of $6.7 million, with the remaining 22 initiatives planned for later implementation.[52] While savings are monitored at the corporate level, they are not tracked at the level of individual statistical programs.[53]

Mr. Ferguson told the Committee that, “Statistics Canada had identified and implemented a number of approaches that resulted in more efficient use of resources. These included using common corporate services and identifying alternate data sources.”[54] He later added, “We were satisfied to see that they had a plan to put in place some efficiency initiatives. They had made progress on that. They had more to do.”[55]

IDENTIFICATION AND IMPLEMENTATION OF PRIORITIES

In order to maintain the relevance of its work, Statistics Canada must identify and respond to emerging needs for socio-economic data. The OAG found that Statistics Canada had developed a 10-year plan, which includes investments in business continuity and quality maintenance.[56] Statistics Canada responded to emerging needs, for example, by redesigning the Consumer Price Index and Labour Force Survey and by conducting feasibility studies to understand needs for First Nations financial data.[57]

Statistics Canada also undertakes cost-recovery work in response to requests from other federal departments and levels of government. The OAG observed that Statistics Canada had continued some previously cost-recovered activities through its base funding, but it did not systematically assess whether it should continue these activities.[58] The OAG recommended that Statistics Canada systematically assess whether to continue cost-recovered work when external funding ceases.[59]

In response to the recommendation, Mr. Smith informed the Committee that:

The agency has now put in place a mechanism, in the course of its annual planning process, to formally review cost recovery programs that are losing funding, in order to determine whether it is necessary to provide, or to try to secure, permanent funding. In the absence of new external funding, maintaining a cost recovery program using existing funding would, of course, mean discontinuing some other programs.[60]

PROGRESS REPORT

Statistics Canada has developed an action plan in response to the OAG’s recommendations. In order to ensure that Statistics Canada implements its action plan as outlined and continues to produce high-quality statistical products in an efficient manner, the Committee recommends:

RECOMMENDATION

That, by 30 April 2015, Statistics Canada provide the Standing Committee on Public Accounts with a report outlining its progress in implementing the Office of the Auditor General’s recommendations in Chapter 8 of the Spring 2014 Report.

CONCLUSION

Statistics Canada provides a vitally important service to governments, businesses, non-governmental organizations and researchers by generating statistical information about Canada’s demographics, society and economic conditions. Statistics Canada has developed a strong reputation for the quality of its data, methodologies and its efficiency. In order to maintain that reputation, Statistics Canada continuously needs to ensure that its quality framework is up to date and is being implemented.

In its audit, the OAG concluded that Statistics Canada had applied its quality assurance framework to ensure the quality of its statistical programs. Mr. Ferguson summarized the audit’s findings, stating:

Overall, in many of those we found a lot of good things going on. The places where we had some concerns were on relevance and interpretability, in particular having some more up-to-date guidelines for people to understand the data and to have some fuller conversations with the users, as has been talked about already today, to identify some of the needs that the users were identifying.[61]

Statistics Canada has committed to make improvements in response to the OAG’s recommendations, and the Committee expects to be able to monitor these steps as Statistics Canada moves forward and implements its action plan.