Under the Accountability Act, federal departments must submit final reports and executive summaries of commissioned public opinion research to Library and Archives Canada (LAC) within six months of the completion of field work. LAC then posts them to a public website.
In the past six months, public opinion reports on 32 different subjects have been posted to the website. Eleven involved focus group research only and 20 included surveys. In four of those 20 surveys, the government conducted the analysis of the results.
“They never used to do that,” says Darrell Bricker, CEO of the polling company Ipsos Global Public Affairs. “They always used to ask us to do analysis of our surveys. We would do it stem to stern, but that has evolved.”
As well, the sheer volume of public opinion research commissioned by government departments has fallen sharply in recent years. In 2010-11, the government contracted 136 projects worth $7.9 million to private-sector survey firms — two-thirds fewer than in 2007-08, when it paid $24.7 million for 446 research projects.
In an email, a government spokesperson said the shift to internal analysis is “one way departments are doing their part to help reduce overall costs for taxpayers.
“This practice not only saves analysis and reporting costs, but also permits tracking and comparison of results over time,” the email said. “Choosing to conduct analysis in-house or externally will vary depending upon the project, budget and capacity.”
But transparency suffers when surveys are analyzed internally. Analysis done by private pollsters is included in reports posted to LAC’s website, along with an easy-to-read presentation of the detailed findings.
That is missing entirely from the reports done in-house by government. Instead, the reports offer only a description of the methodology used and the survey questionnaire.
Tables that show raw survey data are also supposed to appear. Indeed, the government readily acknowledges it has a legal obligation to post them. But even those are sometimes missing. For example, a Nov. 16, 2011, report on a survey of Canadians’ perceptions of the state of the economy, commissioned by the Finance Department, not only offers no summary or analysis of the findings, it also lacks the data tables required by law.
Another report, dated March 2, 2012 and commissioned by Health Canada, examined attitudes toward the health care system. Conducted by the Strategic Counsel, the research involved a survey of 2,500 Canadians, as well as focus groups.
While the report presents the findings from the focus groups, it is silent on the survey results because, the report says, “a detailed analysis of the results from the survey was not requested as part of this contract.” The raw data tables initially were missing as well, and were only added to the website after the Citizen asked Health Canada to release them.
When data tables do appear in reports analyzed in-house, they are presented in a user-unfriendly format that hinders comprehension.
Despite that, Bricker doesn’t regard the shift to internal analysis of government-commissioned surveys as anything “particularly nefarious. It’s just that, over time, they’ve developed a certain number of skills inside, and they’ve decided to take advantage of it.”
But Chris Dornan, director of Carleton University’s Arthur Kroeger College of Public Affairs, said divorcing the architecture and design of a survey from its analysis does not seem methodologically sound. “There’s something weird about that.”
Making raw data from commissioned surveys public while withholding the analysis may satisfy the letter of the Accountability Act, Dornan said, “but it could be interpreted as an attempt to skirt its intent. By not posting the analysis, they’re not showing what contributed to the government’s thinking on an issue.”
Some say the shift to in-house analysis also reduces the amount of independent advice the government receives.
“On many issues, bureaucrats actually have a point of view about what they think should happen,” said Chris Waddell, director of Carleton’s School of Journalism. “They may try to interpret the results of any poll to support whatever point of view they’re trying to persuade the politicians to implement.”
By contrast, outside polling firms have no vested interest in the way they interpret survey results, he said.
Donna Nixon, an Ottawa-based pollster with the Strategic Counsel, said in-house survey analysis cuts both ways. “In some respects, they really have such a strong understanding of the public policy environment that they can put some of the information they get into a very realistic context,” she said.
Waddell argued that collecting answers to survey questions “is kind of a commodity these days. You can hire a company to do that for you. So the question is, why are you hiring the polling company if you’re not hiring them for their analysis? If it’s not analysis, what exactly is it that they’re bringing to the table? How to write questions? Well, that’s not really tough.”
Original Article
Source: canada.com
Author: Don Butler
No comments:
Post a Comment