In praise of induction – The Washington Post

Another way at looking at the difference between evidence and anecdote, and the merits and utility of each, by Daniel W. Dresser of  the Fletcher School of Law and Diplomacy at Tufts University:

One of the tensions that explains the fraught relationship between politics and the academy is that academics are big fans of deductive thinking and politicians are not.

At the risk of exaggerating the gap, academics like to think deductively — i.e., start from theory and then test whether that theory explains parts of the real world. When I was in graduate school, my professors talked a lot about the perils of thinking inductively — i.e., building a general theory from looking at a particular case. The obvious danger was to build a theory from a particular case, and then use that case as evidence of the theory’s power — the very definition of a tautology.

Politicians preternaturally think in an inductive manner. They build from experience, narrative and analogy to articulate what they think matters in the world of policy. For politicians, this makes a great deal of sense, because they trust their own experiences far more than abstract data, and because they know that narratives resonate far more with voters and citizens than abstract theories. Consider, for example, Chris Christie’s moving discussion of how to treat drug addicts. It’s a brilliant demonstration of a politician using a particular narrative to make a deeper point on policy.

In splitting the world like this, I’m simplifying things a lot. One could argue that Barack Obama’s problem as a politician is that he is too abstract and not inductive enough. Similarly, most scholarship emerges from the interplay of deductive and inductive thinking. But still, I think there is some truth in this dichotomy.

My reason for bringing this up is to point out that my own tribe of academics still looks down on the inductive method of theorizing as a flawed approach that is prone to error. And those flaws are real. But I fear that this has blinded many academics to the virtues of induction, because they exist. Indeed, twice in the past week, it’s come up in policy debates.

Source: In praise of induction – The Washington Post

The Case for Teaching Ignorance – The New York Times

Good advice to all of us, whether policy makers or not, on uncertainty and the need to understand the limits of available evidence:

Presenting ignorance as less extensive than it is, knowledge as more solid and more stable, and discovery as neater also leads students to misunderstand the interplay between answers and questions.

People tend to think of not knowing as something to be wiped out or overcome, as if ignorance were simply the absence of knowledge. But answers don’t merely resolve questions; they provoke new ones.

Michael Smithson, a social scientist at Australian National University who co-taught an online course on ignorance this summer, uses this analogy: The larger the island of knowledge grows, the longer the shoreline — where knowledge meets ignorance — extends. The more we know, the more we can ask. Questions don’t give way to answers so much as the two proliferate together. Answers breed questions. Curiosity isn’t merely a static disposition but rather a passion of the mind that is ceaselessly earned and nurtured.

Mapping the coast of the island of knowledge, to continue the metaphor, requires a grasp of the psychology of ambiguity. The ever-expanding shoreline, where questions are born of answers, is terrain characterized by vague and conflicting information. The resulting state of uncertainty, psychologists have shown, intensifies our emotions: not only exhilaration and surprise, but also confusion and frustration.

The borderland between known and unknown is also where we strive against our preconceptions to acknowledge and investigate anomalous data, a struggle Thomas S. Kuhn described in his 1962 classic, “The Structure of Scientific Revolutions.” The center of the island, by contrast, is safe and comforting, which may explain why businesses struggle to stay innovative. When things go well, companies “drop out of learning mode,” Gary P. Pisano, a professor at Harvard Business School, told me. They flee uncertainty and head for the island’s interior.

The Case for Teaching Ignorance – The New York Times.

Don Cayo: Policy decisions made without facts can only fall back on whim, bias and ideology

More on anecdotes vs. evidence, this time in the context of Vancouver house prices but a broader message by Cayo:

But populism, political pandering, intuition or ideology — call it what you will, the made-up facts that are spewed in so many debates — are always a poor basis for decision-making. Yet we see a heavy reliance on this at every level of government, not to mention with voters every time we go to the polls. And, sadly, we seem doomed to see a lot more.

Every government is prone to ignore relevant data, even when it’s available, or to spin it to favour their ideological or political priorities. And there are lots of gaps in the data they can draw on — like the gaping hole in the Vancouver housing picture that invites us to jump to any conclusion that suits our mood.

Worse, the federal government has been refining this shortcoming. It has launched what looks like a focused assault on the sources of information that could inform debate on myriad issues.

The worst step in this direction was the 2010 decision to scrap the mandatory long-form census. Despite continuing outcries of not just political opponents but also of apolitical researchers and analysts in many important fields, the government won’t budge.

This move didn’t have many immediate consequences, but its effect will multiply over time. Because the response rate to the voluntary replacement for the mandatory long form plummeted from well over 90 per cent in 2006 to less than 70 per cent in 2011, and because most non-responders are from smaller places and poorer segments of society, the resulting data is skewed. It’s no longer reliable enough to do the job it once did: informing intelligent planning and the like.

The result will be more policy debates and decisions driven by ideology, bias or whim, just like Robertson’s and Clark’s conflicting views on housing policy. And no side will be unable to authoritatively back its case or counter the other side’s without trustworthy data.

The list of growing impediments to intelligent debate goes on: Cutbacks to research budgets; the muzzling of federal scientists and other experts with information to contribute; the chill on non-profits, some with considerable expertise, because they fear loss of funding and/or being singled out for audits that seem to be targeting perceived advocates of positions not favoured by the federal government.

Don Cayo: Policy decisions made without facts can only fall back on whim, bias and ideology.

Advice for Policy Makers and Researchers

While this was written to assist government scientists and policy makers better understand each other these are both very good lists, compiled by British and Australia policy makers and researchers. They capture the dynamic well between the technical expert and the more general policy advisor roles and perspectives, and tap into themes of ideology, evidence and risk of Policy Arrogance or Innocent Bias: Resetting Citizenship and Multiculturalism.

Good reading both within the public service and with the political level, given some of the ongoing tensions regarding evidence and anecdote and how the different perspectives play out.

Top 20 things scientists need to know about policy-making

  1. Making policy is really difficult
  2. No policy will ever be perfect
  3. Policy makers can be expert too
  4. Policy makers are not a homogenous group
  5. Policy makers are people too
  6. Policy decisions are subject to extensive scrutiny
  7. Starting policies from scratch is very rarely an option
  8. There is more to policy than scientific evidence
  9. Public opinion matters
  10. Economics and law are top dogs in policy advice
  11. Policy makers do understand uncertainty
  12. Parliament and government are different
  13. Policy and politics are not the same thing
  14. The UK has a brilliant science advisory system
  15. Policy and science operate on different timescales
  16. There is no such thing as a policy cycle
  17. The art of making policy is a developing science
  18. ‘Science policy’ isn’t a thing
  19. Policy makers aren’t interested in science per se
  20. We need more research’ is the wrong answer

Top 20 things politicians need to know about science

  1. Differences and chance cause variation
  2. No measurement is exact
  3. Bias is rife
  4. Bigger is usually better for sample size
  5. Correlation does not imply causation
  6. Regression to the mean can mislead
  7. Extrapolating beyond the data is risky
  8. Beware the base-rate fallacy
  9. Controls are important
  10. Randomisation avoids bias
  11. Seek replication, not pseudoreplication
  12. Scientists are human
  13. Significance is significant
  14. Separate no effect from non-significance
  15. Effect size matters
  16. Data can be dredged or cherry picked
  17. Extreme measurements may mislead
  18. Study relevance limits generalisations
  19. Feelings influence risk perception
  20. Dependencies change the risks

 

Harper’s Greatest Hits: the science of fundraising | iPolitics

One of the stronger critiques, and a bit over the top, of the Conservative government’s rejection of science-based evidence, fitting into one of the themes in Policy Arrogance or Innocent Bias, evidence or anecdote.

Harper’s Greatest Hits: the science of fundraising | iPolitics.

Half a cheer for Jason Kenney’s revolution in immigration policy | Toronto Star

Natalie Brender in The Star on Policy Arrogance or Innocent Bias: Resetting Citizenship and Multiculturalism, focussing on the risks and limits of anecdotes for decision-making. Nice to see words like epistemological  (theory of knowledge – yes, I had to look it up too!) to capture the issues and dynamics.

In the end, I am more in the camp of anecdotes and evidence, understanding the strengths and weaknesses of each one, but using both to ensure the best possible policy outcome.  Article as follows:

Andrew Griffith, a retired senior official at Citizenship and Immigration Canada, has just published a book about the tense period beginning in 2007 that saw minister Jason Kenney bring a tidal wave of change to two federal departments. Among the many virtues of Griffith’s book, Policy Arrogance or Innocent Bias: Resetting Citizenship and Multiculturalism, is a striking commitment to epistemological modesty and self-reflection.

Throughout his case studies of various policy issues, Griffith underlines how officials working on multiculturalism and citizenship issues under Kenney were forced to confront their own latent ideologies and grapple with challenges to their expertise under a regime that broke starkly from the approach of previous governments.

From vocabulary to policy priorities to the deepest questions of what counted as sound evidence for policy-making, the Conservatives upended decades of received wisdom. For instance, Griffith reports, Kenney and his staff held in particular odium the blame-laying perspectives taken by “downtown activists” and researchers in analyzing mainstream discrimination toward cultural minorities.

An organization’s use of terms such as “oppression,” “white power” and “racialized communities” became grounds for striking it from a pool of grant applicants. This aversion was part of the minister’s larger distaste for the issue of barriers facing visible-minority Canadians, and his desire to shift focus toward discrimination within and among minority communities.

Because Griffith writes as a consummately professional public servant, he doesn’t pass explicit judgment on the policy shifts effected during the Kenney years. As he notes, it’s the job of elected officials to decide government priorities, and the job of public servants to be loyal implementers of those decisions.

On the other hand, it’s also the job of public servants to provide expert insight and advice to their ministers, who are supposed to take that advice into account in making policy decisions. It’s on this score that some of the book’s most revealing insights lie, since there was an unprecedented parting of ways between Kenney and officials on the question of what counted as sound evidence.

Multiculturalism and citizenship officials had long been used to basing their insight on social scientific research such as large-scale surveys and data collection on a range of standard topics. In Kenney, they were confronted with a minister who took his bearing from first-person anecdotes gathered from tireless meetings across Canada. (Such a minister, in the words of another official quoted by Griffith, was “like Halley’s comet, only coming by once every 76 years.”) Through the nuggets of information gained from his unmatched ear-to-the-ground contact with the nation’s increasingly suburban ethnic communities, Kenney was confident in his knowledge of their realities and concerns.

That confidence accompanied what Griffith alludes to as “the minister’s (and the government’s) general skepticism about social policy research,” and their disdain for the “downtown activists” who had forged deep ties with multiculturalism staff. Two starkly different “evidence bases,” as he puts it, were being drawn on by the political and bureaucratic levels.

Notably, Griffith does not depict the outcome as an unmitigated disaster from a policy-making perspective. Kenney was indeed gleaning real insights into experiences and concerns within different communities, which could not be captured in large national surveys or data sets. He gathered anecdotal reports on topics it had never occurred to officials to investigate systematically – for instance, on violations of citizenship integrity within certain immigrant groups in matters such as cheating on citizenship tests or so-called “birth tourism.”

Expert officials sometimes found to their surprise that the minister’s revamped multiculturalism priorities met with approval among diverse communities in the department’s focus group testing. And in Griffith’s own judgment, the anecdotal evidence that Kenney gained sometimes did produce worthwhile new directions in policy and programming (such as initiatives to address discrimination within and among ethnic groups).

For these reasons, Griffith writes, “officials had to learn to listen to — and respect — the key messages and insights coming from the minister, reflecting his anecdotes and conversations from his extensive community outreach.” It was a wrenching adjustment for many to have their expertise challenged and world views dismissed. Eventually, though, most staff took on board the insights that anecdote could offer, and worked to incorporate them into programming and policy.

There is no indication that Kenney and his staff reciprocated in the epistemological modesty department. In one exceptional instance, Griffith reports, officials found studies that managed to persuade them that racism and discrimination indeed pose real barriers to the success of certain ethnic groups in Canada. But other than that, the learning and broadening of world views seems to have been entirely one-sided.

And in the bigger picture, even anecdotes reflecting a partial reality give precious little for policymakers to go on. Stories of fraud whispered in the minister’s ear don’t tell policy makers how widespread the incidence of citizenship-test cheating or birth tourism is. They don’t tell policymakers what the relative dollar costs of taking action or keeping the status quo will be; nor do they predict what side effects might come from dramatically changing current policy.

Only careful data collection and analysis can do that. And that’s precisely what the Kenney regime (and the Harper government) couldn’t be bothered with in their haste to get tough on “abusers of Canadians’ generosity.”

Writing as a loyal civil servant, Griffith doesn’t say it explicitly, but the lessons of his book are clear. Anecdote is a lousy basis for policymaking, and modesty and self-reflection are not virtues to be expected only on one side of the relationship between the public service and politicians. As Chris Alexander takes over these files as minister of immigration, he could get a fine start by bearing those truths in mind.

Half a cheer for Jason Kenney’s revolution in immigration policy | Toronto Star.