A Field Guide to Polling: Election 2020 Edition

Useful advice in general, not just in terms of US political polling:

How can you tell a ‘good’ poll from a ‘bad’ one?

The longevity of phone polls has allowed scholars time to study them and establish basic standards and best practices. For this reason, it’s a fairly straightforward task to sort more rigorous phone polls from the rest. In general, rigorous surveys are those that are paid for and fielded by a neutral source; have selected a probability-based, random sample of the public (or the population of interest, such as registered voters); dial cellphones in addition to landlines; make multiple attempts to reach people; use live interviewers; and make public both the questionnaire and a detailed methodology.

On the other hand, creating a quality checklist for online opt-in polls remains a challenge and a work in progress.7 Some considerations are the same as for phone polls, as they too should be funded and conducted by a neutral source and transparent with their questionnaires and methodologies. Beyond these basics, evaluating the quality of an opt-in election poll should consider the following questions: Does the sample include all kinds of Americans? Does it include them in roughly the right proportion compared to their share of the population? If not, how are researchers working on the back end to address these issues?8

The Center is now several years into a sustained effort to evaluate these surveys, and several key findings have emerged. Online opt-in polls are not monolithic – some vendors produce more accurate data than others. What separates the better vendors is that they adjust their surveys to be representative on a large set of variables that includes both demographics (e.g., race, age, sex and, per the lessons of 2016, education) and political variables (e.g. party affiliation, voter registration status). Less accurate vendors tend to either not weight their data at all or adjust for just a few demographics. When evaluating these polls, look for evidence that the pollster has thought carefully about these kinds of problems and taken steps to correct them.

When it comes to opt-in online polls, enormous sample sizes aren’t necessarily a sign of quality.

Perhaps surprisingly, Center research found that having a sample that looked demographically representative of the country (through use of quotas or weighting) did not predict accuracy. In other words, just getting the survey to look representative with respect to age, sex, etc. does not mean that the survey estimates for other outcomes are accurate. At this stage in their development, opt-in polls require a very thoughtful, hands-on effort for those fielding the survey, requiring them to consider what they are trying to measure and how the characteristics of the sample may interact with those concepts. As a result, consumers of these polls should also be particularly attentive to these issues.

One last “false flag” to ignore: When it comes to opt-in online polls, enormous sample sizes aren’t necessarily a sign of quality. Given that opt-in surveys are so cheap to field, it’s not hard to drive up a sample size to provide the illusion of precision. But Center research suggests that an 8,000 person opt-in survey is not necessarily more accurate than a 2,000 person survey.

The bottom line for now is that, at least in our own explorations, “even the most effective adjustment procedures were unable to remove most of the bias”9 from opt-in polls. That said, the level of precision these opt-in panels can provide may be adequate for some research purposes.

Source: pewrsr.ch/34cknjD

ICYMI: Pollsters starting to see uptick in government work

Back to “committing sociology:”

The Trudeau government is reinvesting in public opinion research after it was virtually abandoned in the final years of the last Conservative government, though spending remains far below historical averages, according to veteran pollster Frank Graves.

“They’ve committed to doing more and more work…but it’s certainly nowhere near the levels it was historically both with the early stages of the Conservative government, certainly the Liberal government before that, and the Mulroney government before that,” he told The Hill Times.

Mr. Graves, founder and president of Ekos Research, said in an interview that the federal government has contracted more public opinion research work from his company since the election last fall. He linked this to the Liberals’ push to what they see as a return to evidence-based decision-making.

It pales in comparison, however, to what was seen in even the early stages of the Conservative government, he said.

The Harper government spent $31.2 million polling Canadians in the 2006-07 fiscal year before cutting back to $4.9-million in 2013-14, The Hill Times reported.

This stretch of scarce funding represented a “very unusual period,” Mr. Graves said, with the government conducting “virtually no research” of any significance during this span.

A 2003 auditor general’s report clocks in federal spending on public opinion research in 2002-03, under a previous Liberal government, at $23.7 million and $26.2 million the year before.

Mr. Graves partly attributed lagging “rust” in the bureaucratic channels in preventing the Liberals from revving up polling efforts back to previous levels.

“[It’s] going to take awhile for the bureaucracy to catch up and for the resource envelope [to expand] to do this in levels which would be more commensurate with the need and demand and express priority provided to this approach,” Mr. Graves said, noting that civil servants would also need to catch up with technological advancements in the field.

When reached, the Treasury Board Secretariat said it did not have up-to-date figures on spending on public opinion research or consultations specifically.

….Pollsters optimistic after lean decade

Stephen Kiar, CEO and founder of Ottawa-based public opinion and market research firm Phoenix SPI, said his company has also started to see an increase in public opinion research work in the last month or so.

After the election, the Liberals proceeded “cautiously and deliberately,” as new ministers learned their departments, relevant issues, mandates, and staff, among other considerations, he said.

As a result, Phoenix didn’t see any increase in work before the government’s fiscal year ended on March 31, though things picked up afterwards, Mr. Kiar said, as departments began putting together their research plans for the coming year, and seeking the necessary approvals.

“It appears that many departments have finished that planning process and are starting to engage research firms like ours for their projects,” he explained.

Mr. Kiar said it’s too early to compare spending to the previous Conservative government, which he argued “savaged” the public opinion research budget, while dramatically increasing the media monitoring budget.

Under the Trudeau Liberals, mandate letters to cabinet ministers noted a need for Canadians to see the government’s “willingness to listen” and for the government’s work to “be informed by performance measurement, evidence, and feedback from Canadians.”

Mr. Graves framed the period under the Conservatives that saw a “real paucity” of public opinion research as an “anomaly,” and partly blamed the scarcity of polling on what he saw as the government’s indifferent, sometimes “hostile” approach to empirical research.

Critics accused the Harper government of gutting funding for research and muzzling federal scientists. The Conservatives axed the mandatory long-form census in 2011, drawing strong criticism from a wide range of groups worried about the consequences the decision would have on the reliability on the vital data gleaned from the sweeping survey of Canadians.

The Liberals reinstated the long-form census as mandatory shortly after assuming office last November.

Kara Mitchelmore, CEO of the Marketing Research and Intelligence Association, an industry advocacy group, says numbers on polling activity for 2016 won’t be known until the end of the year, though she cited the re-establishment of the mandatory long-form census as leading to an increase in work.

“I can say anecdotally that with the re-instalment of the long form census, which MRIA strongly supports, there is an obvious noticeable increase in data collection roles,” she said in an emailed statement, noting that this will “trickle down” into more analyst roles, which is “great news” for the industry.

Mr. Graves said he expects funding for polling to eventually be restored to previous heights, though predicted it would only reach a quarter of the historic average this year.

That’s still “a lot better” than what we saw in the late stages of the previous government, he noted.

Source: The Hill Times