Here’s the score:
The reliably partisan Jason Cherniak takes issue with a poll released by Ipsos Reid suggesting that Stephane Dion is in third place among polled Canadians asked who they think the best candidate to be prime minister is (behind Harper and Layton). He questions the methodology of the poll in this post.
Yes, Jason has a partisan interest here. He’s a staunch Liberal, and the former co-chair of Stephane Dion’s leadership blog campaign. Further, this is Ipsos-Reid we’re talking about: a big polling company with a lengthy history. I seriously doubt that they’d make stuff up, but jumping to the National Post article which breaks this news, something jumps out at me which makes me believe that Jason has a point.
The online survey — conducted among a random group of 1,000 people from Feb. 15 to Feb. 19 — reported 46 per cent said Harper would make the best prime minister.
Now, I may only be an urban planner by education, but I’ve taken my share of statistics courses, and I have to ask, just how can an online survey be random?
The holy grail among statisticians is to find a truly representative sample — a gathering of individuals whose opinions by percentage perfectly match that of the population. Because, when you run these surveys, you want to know what the population thinks, but short of holding an election and asking what everybody thinks, it’s far cheaper to pick individuals at random.
Which means, you have to eliminate all factors, be it geography, affluence, religious belief, ethnicity, which might bias the sample. If I stood on the corner of Bloor and Yonge, and asked the first ten people who passed me what they thought of something, my poll would not be a representative sample of what Canadians think. You need to grab the names and addresses of a significant number of Canadians, selected randomly from across the nation. Ideally you should go to their door and ask them your survey, but as that usually outstrips the resources of most polling companies, polling companies used to rely on the phone.
Which is why, back in 1948, the media in the United States were absolutely certain that Republican presidential candidate Thomas Dewey was a shoe-in to defeat Democratic incumbent Harry Truman — to the point of the Chicago Daily Tribune printing on its front page “DEWEY DEFEATS TRUMAN” before the poll results were in, and having to embarrassingly retract its statement the next day. You see, all the opinion polls showed Dewey having a substantial lead, but the opinion polls of the day were being conducted by telephone. At the time, the telephone, while popular, had not achieved the widespread acceptance that it has today. Poor families, those more likely to vote for Truman, were more likely not to have a phone and were thus more likely not to be contacted by the opinion pollsters.
This is why alarm bells sounded as soon as I heard the phrase “the online survey”. Now, I did have to step down from my initial impression, which was of some web poll run on the Globe and Mail website where anybody who visits can just click a button and vote (multiple times unless the poll is secured to accept only one vote per IP address). However, even if the “online survey” was an e-mail sent out to various participants, inviting them to participate, it fails the sniff-test of being a representative sample in one key way. Jason touches on it:
They only represent the views of a subset of Canadians who are active online, choose to join public opinion surveys and, perhaps, hope to win a free iPod. This is completely different than a random sample of people who are lucky to be called by a company like Ipsos once in their lives.
That’s harsh, but until I see the methodology of the online survey, it’s a fair criticism. Now, someone purporting to be a representative of Ipsos Reid did comment on the methodology, noting that the sample of 1000 participants was carefully adjusted so that demographically it represented as closely as possible the Canadian makeup, but my concern is that online surveys tend to get a biased sample by asking only those who are active online. There is no randomness here, you are only getting those with an axe to grind, or who have an interest in participating in surveys. And we know this because if Ipsos really sent survey requests to random Canadians on the net, they would be severely attacked for sending out spam.
And let’s not forget that, while the majority of Canadians are online, a significant minority are not, and that potentially biases the sample enough that we should take this poll with at least the same amount of salt we should be giving all opinion polls.
That’s the nature of opinion polls in general. Most statisticians will tell you that you have to take all surveys with a grain of salt, since there is no way you can get a 100% representative sample; that’s why you get disclaimers saying that responses are accurate within X%, 19 times out of 20. However, there are methods which make your sample less representative than other samples. Only polling those Canadians wealthy enough to access the Internet, and only polling Canadians with enough time on their hands or interest to participate in online surveys, adds a bias in my mind. And getting sensitive about such criticism requires us to add more salt.
I wouldn’t be posting about this, had not Jason’s post ruffled a few feathers, as seen here:
For the post directly below, where I question the value of online polls in general, I just received a threat from Ipsos Camelford Graham. I reproduce it in full:
Sent: February 21, 2007 3:18 PM
(snip!) - jb
Jason, if you do not want to hear from a lawyer, I would strongly suggest that you retract what you wrote about Ipsos on your little blog. We take attacks on our reputation VERY seriously and I doubt that you can afford to find out just how seriously.
Ipsos Camelford Graham
As Jason is a lawyer, I suspect he has some idea of how solid the ground is on which he stands. Certainly, the response does seem rather out of proportion to Jason’s initial criticism, however partisan his reasons for giving that criticism might be.
The research associate has since apologized for the e-mail, and for that reason I have blacked out his name. But perhaps this incident shows that we are taking our opinion polls too seriously. We’re taking nothing more than snapshots here, whereas it’s the federal election that actually gauges what most Canadians think at any given time. As the old truism goes, the only poll that matters is the one on election day.