Getting harder for pollsters to sample

by Crispin Hull on May 19, 2012

THINGS are getting more difficult for pollsters. Polling techniques and interpretation may be getting more sophisticated, but the critical base of all polling – the random sample – is getting harder to come by.

Most politicians act with public bravado when asked about the latest poll at any time. They usually say there is only one poll that counts. They know they cannot act with glee when their side gets a boost because it would require them to acknowledge bad polls as well.

But privately and in the party rooms, polls matter. A leader is safe if the polls indicate victory awaits. A string of bad polls puts a leader in jeopardy.

Purists might say we take too much notice of changes in the fortnightly polls. The latest Newspoll is a case in point. Labor’s vote went up by 3 percentage points to 30 per cent. But if you read the fine print, that is hardly statistically significant.

With a sample size of 1141 you can be 95 per cent confident that the result achieved (either for Labor or for the Coalition) is within 3 percentage points either side of what the population as a whole really thinks.

So you can be 95 percent confident that Labor’s vote is between 27 percent and 33 percent.

Given a rise this time in Labor’s vote of just 3 percentage points, purists would say it is not STATISTICALLY significant. But more seasoned observers would still say it is significant because they would add into their poll analysis all sorts of other things that statistical purists would throw their hands up in despair at.

These are factors like a Budget being brought down that week that gave lots of people money in hand which might explain the change.

You might think that the 3 per cent rule makes the Greens standing in the polls hard to pick. At 12 per cent, you can be 95 per cent sure only that between 9 and 15 per cent of the population would vote for them. But as it happens the more extreme the percentage (either high or low) the more confident you can be that a result is a true one.

All of this, of course, depends upon the true randomness of the sample. Pollsters have been constantly tweaking this.

Telephone numbers are generated randomly. People are rung at different times in the day so some work-time cohorts are not missed. Once rung the selection of the householder to answer the questions is randomised (the person whose birthday is next) so the poll is not skewed towards chronic phone answerers. And so on.

But despite all this care, the random sample is becoming more elusive. Newspoll, like most other pollsters, polls by landline telephone.

The trouble is, landlines are being used by fewer young people.

The Australian Communications and Media Authority’s latest figures show that the number of landline services dropped during 2010-11 from 10.59 million to 10.54 million.

More significantly the number of Australia without a landline service rose from 2.3 million to 2.7 million in that year. At that rate you would expect the number to be about three million now – more than a fifth of the total.

The number of landline services has only fallen slightly. This has disguised the fact that the number of people with access to landlines is falling dramatically. The mismatch is because older people are keeping their landlines after grown children leave and spouses die, so there are fewer people per landline.

Meanwhile, the departing grown children are not getting their own landlines. They are often renters. They can get all the phone, texting and internet services they need on the mobile network.

The ACMA says mobile services grew by 13 per cent during 2010–11 to reach 29 million. That includes mobile internet services, but the handset services amount to more than one per person. Many people have work and private mobile phones.

A lot of them are forsaking their landlines.

So the randomness and accuracy of the polling sample must wane as the number of landline users declines.

The pollsters are clearly missing a lot of young people who prefer mobiles, especially renters.

My guess is that this would result in under-estimation of the Green vote, and possibly the Labor vote.

That said, in recent times, the effect has been neutralised at election time because the campaign tends to hurt Green support as they get squeezed in the media as journalists and voters concentrate on the main players. Also, big donors give to the major parties who concentrate their spending of it at election time.

Some pollsters, notably Morgan, prefer face-to-face political polling. But the logistics of getting a truly random sample that way in this wide brown land must be extremely difficult.

You can bet the pollsters will be working on ways to overcome the landline bias. One way is weight answers according to age-bracket census data. So if people aged 18 to 25 form, say, 8 per cent of the population and only 4 per cent of the “random” sample, you double the weight given to the poll answers from that age group.

Tricky stuff.

Pollsters desperately want to get it right. Their livelihoods literally depend on it. Newspoll tells its customers it wants to provide “cost effective research you can trust”. “Our accuracy has been proven time after time through our regular voting intention election surveys,” it says. “We use the same commitment to quality to support all of our clients.”

Pollsters like to continue polling right up to election day because their poll result is then more likely to be closer to the validating actual election result.

This far out from an election, Labor MPs must be asking will the latest gains be kept and improved upon. At present the 30 per cent primary vote and preferences would give it a 45 percent two-party preferred vote. If uniform that would result in the loss of 20 of its 72 seats – more than one in four Labor MPs.

Not all of those would consider their case hopeless. Not all would consider their chances any better if they changed leader now. But history shows that backbenchers get techier as election day nears if polls are bad and if they show the party’s position would improve with a change in leadership. Remember that in 1983 Labor threw out Bill Hayden as leader on the day the election was called.

Winning is everything. This is why polling is so important and why pollsters have got to face up to the declining use of the landline.
This article first appeared in The Canberra Times on Saturday 19 May 2012.

{ 0 comments… add one now }

Leave a Comment

You can use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>