What Happened When Canada Stopped Counting Its Numbers

Five years after leaders killed the long-form census, researchers can't tell you what direction the country is going in. 

As published by Al Jazeera America, April 2, 2015.

TORONTO — When a major Western country stops counting its numbers, bad things can happen. 

In June 2010, the Canadian government unveiled a grand experiment in data collection. In the name of privacy, Prime Minister Stephen Harper ended the mandatory long-form census for the country and swapped it out with a voluntary survey.

Five years later, there is a mass scramble to make sense of a rapidly changing country. Despite an explosion of corporate data-mining in most nations, researchers interested in tracking poverty, immigration and public health in Canada know less and less about the country as time progresses. They’re not, for example, entirely sure if income inequality is accelerating, stagnant or closing. Across the nation there is a loud, collective uneasiness among them.

Munir Sheikh was Canada’s chief statistician when the long form was killed. He publicly resigned in protest the next day.

He’s still not a fan. Running a country without hard data on its citizens, he said, is “like flying a plane without any guess as to where you’re going.”

“We’re not sure” has become the rallying cry of researchers, urban planners, health officials, policymakers and businesses. With its long-standing yardstick gone, the fogginess of new data means less and less is known about one of the world’s major economies as each year passes.

While Canada is one of the world’s wealthiest countries, it is faltering when measured against comparable nations. Modern economies rely on hard benchmarks against which data are measured. Even countries that do not hold a regular census, such as Germany, have measures in place to gain the crucial data needed to act as a standard. With crude oil prices in free fall, determining population and employment numbers is imperative for areas of Canada that depend on lofty prices per barrel. 

But even today, the government maintains that its voluntary survey has the integrity of a census, and there is no will in the party to reinstate its long-form predecessor.

The background

Since 1971, every five years, each Canadian household has been mailed a census. Of these households, 80 percent received a short questionnaire with eight questions. The remaining 20 percent were sent the long-form version and were legally compelled to answer an additional 53 questions about income, education, housing and family structure.

In June 2010, the Conservative Party argued that the long-form census was too personally invasive. This argument was buttressed by the alleged thousands of complaints received by the former Industry Minister Maxime Bernier (a claim that was later withdrawn). A subsequent investigation by The Globe and Mail, one of Canada’s largest newspapers, revealed formal complaints against the census had been steadily decreasing over the years.

As a remedy to perceived government overreach into private life, the Conservative government kept the mandatory short questionnaire in place but ended the mandatory long-form part and replaced it with a voluntary questionnaire they called the National Household Survey (NHS).

Foreseeing trouble in achieving the same response rates as a mandatory survey, additional copies were distributed to fill the anticipated response gap. In 2006 the response rate for the long-form census was 93.5 percent. In 2011 the NHS response rate plunged to 68.6 percent. To Sheikh, the reason is simple.

“If you tell somebody, ‘Please fill out a survey,’ a lot of people will ignore that. If you tell them ‘Please fill out this survey, it’s the law of the land, you’re going to break the law,’ they complete it,” he said.

The government, he noted, is gifted with a singular power: the authority to compel citizens to answer a survey.

“Anybody can make a voluntary survey, but you can’t force anyone to respond to your questions. Whatever you get will be flawed because it doesn’t represent the country.”

While Sheikh is perhaps one of the most vocal in his frustration over the state of data in the country, he is by no means alone.

A generation of data gone 

Economist Sheila Block suspected that, despite being a diverse country that prides itself on a policy of multiculturalism, Canada still has race issues that permeate the workplace. As she and her colleague Grace-Edward Galabuzi sorted through reams of employment data from the 2006 national census, they found results that dismayed them. 

“We have a lot of evidence from previous censuses and other work that we have racism and discrimination in the labor market in Canada,” said Block, a senior economist at the Canadian Institute for Policy Alternatives.

In 2011, using data from the 2006 national census, Block and Galabuzi produced “Canada’s Colour Coded Labour Market,” examining income and employment discrimination. They found that visible minorities were earning less and often were disproportionately employed in higher-risk fields. Furthermore, visibly minority immigrants consistently earned less than other immigrants. For an ethnically diverse economic powerhouse such as Toronto, this is a problem. In 2006, 47 percent of residents in the city identified as being a visible minority.

In 2014, Block was eager to reprise the report, using data from the 2011 census. As a number cruncher, she was interested in understanding the effects of the 2009 recession on workplace discrimination.

However, this time around, there was only a voluntary survey to work with. Because of unavailable data, she and her colleague found themselves describing why the report could not be finished. With little consistency between the NHS results and previous censuses, they could not determine which aspects of the data were reliable and which were not.

For Block, there is an urgency to her project. “We need to understand how labor-market discrimination operates in order to address it,” she said.

Because there is no way to determine how pervasive the issue is becoming, estimates are nothing more than best guesses made with shaky assumptions.

Barring a change of government in the upcoming October federal elections, the next time quality data on par with the 2006 census will be available is 2021 — a cavernous gap in reliable data. This, of course, assumes the long-form census is reintroduced or a new way of collecting data emerges that skirts the problems of voluntary surveys.

“We are losing a generation of data in terms of trying to understand and address labor market issues,” said Block. “It’s kind of disastrous.”

Often aboriginal communities are unrepresented in census data because of low response rates. In 2011, Statistics Canada reported that it had to suppress data from a number of reserves because the number of responses did not meet the required threshold for quality control.

Costs up, quality down 

Harvey Low, the manager of social research and analysis for Toronto, is a fast-talking, passionate numbers man. He worked with his first census as a university student in 1987 and later relied on the census data as an urban planner and social worker. For his entire career, he has been analyzing and reporting on survey numbers.

“We need to understand the profile of who we serve and the places where these people live, because we provide services by areas, by geographies and by neighborhoods,” he said. “How has that yardstick moved? If we’re investing in trying to get youth more jobs, we would like to know how that’s changed over time.”

Because the mandate of municipal governments is to provide services to residents, obtaining data about those residents is crucial. Public transit planning, emergency response services and affordable housing services all rely on data from the census 

Toronto, like many municipalities across the country, provides subsidized child care for low-income families. A memo prepared by the city provided to Al Jazeera America depicts how these subsidies are allocated. If a ward has 5 percent of the city’s low-income families, 5 percent of the subsidies are allocated to that ward.

However, research by statisticians consistently shows that lower-income families are underrepresented in census responses. When census data are used to direct subsidies, those in the areas with the most need face magnified shortages.

The city now spends more time inspecting and analyzing the quality of the data rather than the content, leading to increased expenditures. “The costs have gone up. The quality has gone down,” Low said.

In fact, the quality has eroded so much that some departments have fully jettisoned the use of NHS data. Toronto Public Health no longer uses the voluntary survey's results.

“We’re in a situation where my best source to check a survey I do today, to see if it’s representative of the city, is from 2006. That’s not representative of the city,” said Nancy Day, the supervisor of health status and epidemiology at Toronto Public Health. Lower-income residents are the least likely to respond to voluntary surveys like the NHS. According to Day, with income closely tied to health, reduced income information hampers the abilities of public health officials to determine the geographic location of the groups most in need.

In order to determine the composition of the city, Toronto Public Health now relies on tax data. This tells them nothing about the different ethnic enclaves around the city, nor does it help them track the internal migration of vulnerable and marginalized groups.

Hobbled researchers

This data deficit is cascading into other areas, with case after case of researchers and policymakers stopped in their tracks. Countless examples of hobbled researchers are emerging with increasing frequency as sources of reliable national data slowly dry up.

From 1981 to 2006, Lars Osberg, a professor of economics at Dalhousie University in Nova Scotia tracked the rise of income inequality in Canada. Because of the inconsistency in data in the 2011 NHS, the study is effectively terminated, as there is no way to consistently link the NHS to previous censuses.

On May 7, 2014, the premier of New Brunswick held a news conference to concede that, because of the inconsistent data provided by the NHS, there is uncertainty whether a province-run program tasked with reducing poverty is effective, because accurate measurement is so difficult. 

The University of Toronto’s David Hulchanski developed a methodology that many called groundbreaking for tracking urban poverty. He created maps of Toronto that visualized the clear dividing zones of income inequality. After the success of that project, he developed maps for Montreal, Vancouver, Halifax, Winnipeg and Calgary and Chicago in the U.S. Of all those cities, the only ongoing project with usable data is Chicago. 

The University of Toronto, along with other postsecondary institutions across the country, has called for the return of the census.

But it isn’t just researchers who are outraged.

Businesses are also expressing frustration with the diluted data previously used to understand employment trends and demographics. The Canadian Chamber of Commerce, representing more than 200,000 businesses across the country, is advocating bringing back the long-form census. “We will no longer be able to measure the effectiveness of our social programs and economic policies,” the Chamber wrote in a policy note. Business schools have written editorials arguing that the new census harms Canada’s ability to compete with other nations.

Census as anchor

What irritates many who work with data is how a technical issue became fodder for politics. Throughout the controversy, economists and researchers have repeatedly pointed out that a voluntary survey can never replace one that mandates a response.

In his resignation letter posted to the Statistics Canada website (and later taken down), Sheikh was forceful in his disagreement that a voluntary survey would be of practical use.

“I want to take this opportunity to comment on a technical statistical issue which has become the subject of media discussion,” he wrote. “This relates to the question of whether a voluntary survey can become a substitute for a mandatory census. It can not.”

Mandatory surveys allow statisticians to identify and sample subpopulations, such as aboriginals, immigrants and low-income earners. After demographic concentrations are identified, they can be factored into additional voluntary surveys to ensure accurate sampling. Sheikh likened the census to an anchor. All surveys conducted in the country, because they’re voluntary, must be anchored against a reliable set of data.

“Anyone who runs a survey without the census will not be able to adjust their numbers to accurately reflect the nature of the country,” he said. “It doesn’t matter who does the survey — if it’s in the private sector or the provincial government or the municipal government or by an academic. The census is the foundation for all data that we collect.”

The United States Census Bureau examined the voluntary versus mandatory question and concluded that because of excess cost and significantly decreased response rates, it made little sense to hold a voluntary census. 

Ted Hsu, a former scientist and investment banker, likes to be able to measure what he’s doing and is open to changing his mind when his measurements show he’s wrong. Now a member of Parliament for the Liberal Party, he applies the same reasoning to government.

“If we’re spending money on programs trying to help people, we should be measuring if it’s having an effect,” he said. “And if it’s not having an effect, we should cancel the program and save money.” In September 2014 he introduced Bill C-626, which aimed to restore the long-form census.

After two readings in the House of Commons, a glimmer of hope emerged on Feb. 4 as C-626 was put to a vote. The bill had the support of not only every member of the opposition in the House but also more than 475 businesses and policy-oriented groups, ranging from left-leaning think tanks to the former chief economist of Toronto Dominion Bank.

The bill was defeated, with only one Conservative Party member breaking ranks.

Advocates for the voluntary survey received another victory last week as Wayne Smith, Sheikh’s replacement as Canada’s chief statistician, claimed the data from the NHS was “better than he had expected.” Jake Enwright, the press secretary for Industry Minister James Moore, echoed these comments and added that “the National Household Survey yielded high-quality data.”

Low, the numbers man, concedes that not all the data drawn from the NHS is useless. On large scales, for example, for entire cities, conclusions can be drawn. However, the wealth of detailed information that accompanied the long-form census is lost, as is the ability to understand small pockets and neighborhoods around the country.

“We’re moving away from evidence-based decisions to … I don’t know what. I don’t know what the label for it is,” said Day.

Five years later, Sheikh remains one of the harshest critics of the NHS. “There is a truth out there. The census gets the truth to you. If there’s no census, you have no idea what’s going on in the country,” he said. Of those who wish to keep the voluntary survey in place, he wryly added, “If you know what the ‘truth’ is, you don’t want to be burdened by other evidence.”