Job-vacancy rate plunges as Tories drop Kijiji data – Evidence vs Anecdote

A reminder that bad and incomplete data can lead to bad policy decisions and arguments, as exemplified by the over-stating of labour shortages and justification for programs like Temporary Foreign Workers.

Employment and Social Development Canada recently revised its Employment Insurance, Monitoring and Assessment report to take out weak data from on-line sites like Kijiji (see earlier post How Kijiji’s data threw off Ottawa’s math on skills shortages – The Globe and Mail):

“There’s isn’t really any good data out there. Online postings are online postings. How well can you clean those up?” he [Mostafa Askari] asked, pointing out the need to avoid double counting jobs or counting jobs that have been filled but were not taken offline. He said the solution would be to give Statistics Canada more money to improve its research on job vacancies, which are based on surveys of employers.

“I think Statscan can definitely provide better data if they have the means to,” he said. “I assume they are obviously under budget constraints as well. So they have to put that as a priority but they won’t do it unless there’s pressure on them to provide that kind of information.”

Job-vacancy rate plunges as Tories drop Kijiji data – The Globe and Mail.

Konrad Yakabuski’s take on the problem with big data and lack of rigour in analysis:

Yet, if Mr. Kenney and his advisers are guilty of anything, it is of falling victim to the same social media hype that has led many data enthusiasts to spurn official statistics as oh-so yesterday. Want to know if the flu is headed your way or the housing market is set to take off? Why, go to Google Trends. Forget the official unemployment rate. Just track “lost my job” on Twitter.

The idea that the trillions of bytes of data we generate on social media are equipping policy-makers with vast new predictive powers is all the rage these days. Official statistics, the kind compiled by bureaucrats through scientifically tested surveys and representative samples, seem to bore the geeks. But they get all hot and bothered at the mere mention of the word algorithm….

This is but one example of how big data can lead to misguided policy. Mr. Kenney’s Kijiji snafu is another. You’d think this would make people cautious. But in our insatiable desire to make sense out of an increasingly complex world, we are turning evermore to big data to sort it out.

The latest trend is “data journalism” with The New York Times and several upstart media outlets hiring an army of twentysomething computer geeks to massage the numbers in order to spot trends, predict elections and provide funky, counterintuitive insights in the vein of Freakonomics.

The problem is that much of what they report is probably wrong, or at least tendentious. The Upshot, The Times feature launched April 22, has come under fire for stories that either read too much into the data or leave too much out. “First-rate analysis requires more than pretty graphs based on opaque manipulations of data unsuited to address the central substantive points,” prominent U.S. political scientist Larry Bartels wrote in response to one piece on Southern politics.

The most common sin in data journalism is making spurious correlations. Just because Google searches of the term “mortgage” have closely tracked Canadian housing sales in the past two years means nothing on its own.

Big data’s noise is drowning out the signal

A final irony, the final report of the Prime Minister’s Advisory Committee on the Public Service comes out at the same time as these news reports, affirming the need for outside information and discounting the value of more objective surveys:

New sources of information and data have shaken up the process of providing advice to government, he [David Emerson] said, and the public service is adapting to accept data from outside Statistics Canada or other traditional sources.

“I think we made some real progress in helping public servants to open up and I think political staff now have access to a lot of that same information, so there are checks and balances that I think are a little sharper-edged than they were perhaps in the past,” he said.

While I don’t disagree with opening up, we also need to learn the lessons from Kijiji jobs data, ensure better quality control and analysis, and strengthen the role of official statistics and Statistics Canada.

PS thinking more about the digital revolution: Emerson