NEWS Ltd did not look down and shuffle its feet a bit. There was no sense of embarrassment or even bemusement.
This fortnight’s regular Newspoll published on Tuesday revealed an astonishing increase in Labor’s vote and a dramatic decline in the Coalition’s. The two-party preferred vote swung four per cent to Labor from 52-48 to 56-44 from the Newspoll a fortnight before.
Was this hailed by News Ltd and The Australian as evidence of brilliant handling by the Government of refugees and climate? No. “Rudd loses on Viking standoff” was the headline. The “Viking” being the Oceanic Viking which until then had housed 78 Sri Lankan refugees for three weeks.
How can it be that with such a surge in support for Labor over the very two weeks that the refugee incident had hotted up that The Australian says Labor was losing support over the refugee question?
Well, the simple answer is that journalists are not very good at statistics and even worse at logic. Honest, yes. Knowledgeable, thoughtful and diligent about statistics and logic, no.
The critical thing here was the poll the previous fortnight which showed a huge fall in Labor and rise in Coalition support. It showed astonishingly the Coalition and Labor primary vote neck and neck at 41 per cent each – seven percentage points different for each side from the fortnight before that – from 14 percentage points apart to nothing apart. Gasp.
It showed the two-party-preferred vote (after the distribution of minor party and independent preferences) going from a landslide 59-41 in Labor’s favour to a mere 52-48 split in Labor’s favour. The Coalition was suddenly competitive, according to the new poll.
At the time, The Australian gushed that Rudd was on the nose because of the refugee question. Clearly the Howard battlers, racists and red-necks had taken one look at the Oceanic Viking and decided Coalition was the way to vote.
Other more circumspect commentators suggested that this was a “rogue” poll.
Two weeks later, The Australian could hardly say the earlier poll was wrong and certainly would not say that the conclusions it drew from it were wrong. Sure, the poll a fortnight later showed Labor still down a bit from two fortnights previously, so The Australian could cling to that to argue that in the whole month of the saga of the Oceanic Viking and the arrival of other boats, Labor’s support had fallen.
The more circumspect commentators patted themselves on the back at their conclusion that the poll a fortnight ago was a “rogue” poll.
There are some lessons here for readers and journalists. In fact, it was not a “rogue” poll. It was within the realm of ordinary expectation.
Newspoll interviews about 1000 people. That is a pretty good sample. If selected randomly it means that you can be 95 per cent confident that the result will be within three percentage points of the real picture. It means that if you took the same poll 100 times, in 95 cases it would be within three percent of the first poll.
In practical terms that is important for Newspoll and The Australian which poll every fortnight, or 25 times a year, or 100 times every four years.
It means that five of those 100 polls every four years will be more than three percent out – more than one a year.
So we should expect at least one dud every year. Being three per cent out in a two party race is a six percent difference. In an election, a swing of that dimension almost invariably means the difference between government and opposition.
Just a three percent swing would end the Rudd government, just as it ended the Howard government.
If you want to increase the confidence level and decrease the margin of deviance from the original poll the sample size you need goes up. If you wanted to be 99 per cent confident that you were within three percent or 95 percent confident that you were within one percent you might have to poll 10 times the usual 600. These samples are beyond the reasonable financing of most news organisations.
I am not saying polling is mad, unscientific and silly. To the contrary, much can be gained from polling. It can be a quite inexpensive way of finding out what the public thinks with a great degree of accuracy.
When a pollster with a 600 sample across Australia gets a 60-40 split on an issue like abortion, the republic, emissions trading, capital punishment or whatever, you can be utterly confident that that is where majority opinion lies.
The trouble with political polling is that virtually every election in that past 60 years has had a narrower that 55-45 split, usually much narrower, and usually less than the magic three percent.
Journalists should understand that if you poll every fortnight you should expect at least one dud every year. If they understood that, they would be less likely to draw dramatic inferences from poll changes.
Also, they often fall into the logical trap of “after this therefore because of this”. Even if the poll is an accurate reflection of opinion, the opinion change might be because of the major issue of the day – say, refugees, but it might also be because of lots of other reasons.
Journalists fall into a compounding error. A poll showing a radical swing might be dud or it might be true. Either way, journalists see it as radical, different, unusual and therefore newsworthy. Journalists are interested in change. Things that stay the same are not newsworthy.
A poll recording a big slump or rise in Labor support (even if not reflecting reality) is very newsworthy. The bigger the change the more newsworthy it is, but the bigger the change the more likely it is to be a polling error. In short, the newsworthiness of a poll rises with the likelihood of its inaccuracy. Let’s call it Hull’s Law.
A scientist would say, “We’d better check this out.” But a journalist would say, “We’d better put this on Page One.”
The bigger the polling blunder the greater prominence it will get and the more journalists will fall for the “after this therefore because of this” fallacy.
The past three weeks prove the point. A fortnight ago the “slump” in Labor support shown in the poll was not greeted with any caution about polling error. Rather The Australian leapt in with a headline about Labor’s handling of refugees causing the slump.
Well, the refugees were still being handled in the subsequent fortnight when the next poll showed a “surge” in Labor support. Did we get a headline saying, “Labor support surges in refugee crisis”? Not from The Australian we didn’t.
CRISPIN HULL
This was first published in The Canberra Times on 21 November 2009.
I enjoyed this article.
However, may I suggest that there is a discrepancy between the statement that journalists are honest with the contrary implication in the last two paragraphs?