The Perils Of Poll Driven Policy

Post by: luna_lava

Back when a computer took up two floors of the Engineering Faculty (UNSW) one of my favourite books was “The Use and Abuse of Statistics” by W.J. Reichmann (… I know a bit nerdy). Since that time, not a lot has changed with the central theory of Statistics and the concept that when you “sample a population” your sample must be representative of the larger group.

Most of the polls published in the mainstream media (MSM) are based on the assumption that they are sampling the population of Australian voters to draw inferences on their intentions and to gauge a reaction to a particular issue

One of the traps for the guys who design these polls is the belief that:

“As the size of a sample is increased it approaches more closely to the population itself ….there is safety in numbers”

In 1936 an American magazine forecast that the Republicans would sweep all before them in the Presidential elections. A huge sample of 10 million people was selected and, of these over 2 million replied to the questionnaire. The size of the sample, coupled with the fact that the same sample had previously been used successfully in four earlier elections, was taken as a sure guide to the outcome of the 1936 election. The sample returns indicated that Franklin D Roosevelt would be defeated whereas, in fact, he was elected with one of the largest majorities ever recorded.

The apparent reason for failure to make a correct prediction was the fact that the sample chosen was not representative of the total population of American voters. The magazine’s inquiries had been addressed to its own readers and also to telephone subscribers as listed in the telephone directories, so that the sample represented only the population of people who either owned phones or where readers of the magazine.

In 1936, choosing a sample from those who could afford a telephone was in effect selecting people who preferred one party (Republicans).

Fast forward to today and the use of phone polls. This type of poll is relatively cheap and quick however they are suspect because fixed line users are over represented. Many voters from the younger demographic use mobile phones which are very difficult to sample (you can look up my fixed line but even I don’t know my mobile number).

Pollsters can factor in corrections for some of this stuff but it costs more for what maybe considered marginal improvement.

I suspect that the marketing guys have taken control of newspapers polls (they are a good little earner) and statisticians are a little too close to “scientists” to be trusted with important stuff.

Greg Jericho in his brilliant blog has shown how editorial agendas distort how the polls are published and promoted depending on the political message. Pollsters know what the right wing media wants and they are in the business of selling a product with the “correct message”.

Grog’s Gamut

So do I think the polls are accurate – no .

Do I trust Newspoll given their News Ltd association. Well send me $100 dollars and I will tell you how to avoid being conned.

35 comments on “The Perils Of Poll Driven Policy

  1. So when Julia said “there will be no carbon tax”, was this poll driven?

    When she changed her mind, was this driven by principle or expedience?

  2. Do you have a P.O. box l_l 😀

    what you say makes sense, and it’s no surprise that Newspoll comes up with the aswer that ltd news wants…..
    It becomes a self-fulfilling prophecy.

  3. Excellent analogy luna. In 1936 a phone poll skewed the poll results towards the Republicans (the wealthy). In 2011 a phone call skews the poll result away from the under 35yrs demographic due to under-representation in those having landlines.

  4. Tom of Melbourne, the Prime Ministe did state in her Climate Change speech just before the election that she favoured an emissions trading scheme.

    The speech is still on her web page.–julia-gillard,–moving-forward-together-on/

    I also offer this word of temperance: there is not a switch to flick or one single behaviour to change.

    There is hard work to do that will be even harder without consensus.

    But with hard work and with consensus we can achieve great momentum. We can achieve genuine progress.

    Of course, taking action to reduce pollution was part of the Labor Party’s platform in 2007. Many Australians supported it. And we developed a policy, through the leadership of Kevin Rudd, the tireless work of Penny Wong and many other Ministers.

    Our approach to developing an emissions trading scheme to suit the Australian economy – the Carbon Pollution Reduction Scheme (CPRS) – got a long way during the last two years.

    Ours was an ambitious, economy-wide scheme, which sought to put a limit on pollution – a limit that reflects the true cost of that pollution for our economy and our environment.

    The CPRS would require firms that need to use carbon in their work to acquire permits and allow those permits to be traded so that there is an incentive to reduce carbon pollution and so the permits can be used in the most efficient and most productive ways.

    Revenue from the scheme – from the sale of permits – was planned to be used for assistance to help communities, families and firms to make the transition to a low carbon economy.

    So the principle is that the biggest polluters should pay for the pollution they create, that overall pollution should be limited and reduced, and that people should get assistance with the costs of adjusting to this new way of doing things.

    We never see the question which prompted the answer about the carbon tax, so it’s quite possible that she was referring to the fact that the ordinary punters won’t be taxed on carbon, given that she and Greg Combet have repeatedly said just that.

  5. Not sure what the motive of this post is. I am sure the pollsters are well aware of the pitfalls of telephone polls.

    It should be noted that the ABS uses a telephone poll to determine the unemployment rate. So if phone polls are wrong our unemployment rate is wrong.

  6. l_l

    Very good comments and things I have totally agreed with. The other area of polls is the following.
    1) The structure of the question.
    2) The tone and attitude of the pollster.
    3) The mood of the recipient.
    4) The accent on the certain syllables in the questions.
    5) The ommission of real questions with alternative answers.
    6) The number of answer options and the wrording of the answer options.

    These can all markedly alter a poll or questionnaire. Not just for politics but for big business as well. Manipulation of results and questions to achieve the desired outcome is a skill well utilised by Big Businesses.

  7. Neil, the motive of the post is of course to debate a topic. It’s up to the reader to discern points of interest. I think the stand out for me is: Greg Jericho in his brilliant blog has shown how editorial agendas distort how the polls are published and promoted depending on the political message.

    It’s worth a thought or several, that the polls are skewed to lead the debate rather than which most are lead to believe..the other way around.

  8. Neil of Sydney

    I agree the unemployment rate could be wrong at any given time. Any poll is subject to manipulation and overcorrection on either side.

    A poll of Economists in 2007 predicted Oil at $200 a barrel and Interest rates of 10% by 2010. As the world pelted along.

    Instead we ended up with the GFC, Oil at $50 at one stage and interest rates that dropped to 3.75%.

    How wrong were the economists and how wrong was that poll.

  9. “I think the stand out for me is: Greg Jericho in his brilliant blog has shown how editorial agendas distort how the polls are published and promoted depending on the political message.”

    I agree.

    However I find your sudden conversion to “honesty in the media” to be a little late.

    This sort of thing went on for 11.5 years from 1996-2007.

    In general the media do not like Conservatives. They make then out to be dinosaurs while ALP supporters are more “progressive” like themselves.

    But I do think polls are generally accurate. They got the 2007 election correct.

    And they use a telephone poll to determine the unemployment rate which was news to me when I found out.

  10. 2007 was 4 years ago. I would suggest that since 2007 that far fewer people in the 35yrs and under demographic have landlines. Clearly all polls and all methods of data-gathering will have to move into the 2nd decade of the 21st Century.

    Oh dear that reminds me of Duck-Dodgers..Min waves to Migs..

  11. Only partly correct Neil:

    From the Explanatory Notes of the latest issue of Labour force Australia from the ABS


    3 The Labour Force Survey is based on a multi-stage area sample of private dwellings (currently approximately 29,000 houses, flats, etc.) and a list sample of non-private dwellings (hotels, motels, etc.), and covers approximately 0.33% of the civilian population of Australia aged 15 years and over.

    4 Information is obtained from the occupants of selected dwellings by specially trained interviewers using computer-assisted interviewing.

    5 Households selected for the Labour Force Survey are interviewed each month for eight months, with one-eighth of the sample being replaced each month. The first interview is conducted face-to-face. Subsequent interviews are conducted by telephone (if acceptable to the respondent).

  12. Thanks Bacchus

    One thing i do find strange. I have never been picked and no-one I have met has ever been picked or perhaps they did not tell me.

    I must admit it does seem a strange way to determine the unemployment rate. I think most people would have thought that the unemployment rate would be determined by the number of people receiving the dole or something like that.

  13. No, the problem with relying on Centrelink for unemployment numbers is the large number of people who are ineligible for the spouse of just about anyone who has a job – they may be looking for work, but cannot register with Centrelink because their husband or wife earns too much, or a dependent teenager still living at home, etc. etc…

  14. ‘This sort of thing went on for 11.5 years from 1996-2007.’

    Yea, we get it nil, you didn’t like the way the Sydney Morning Herald reported rorting in a report that addressed, yea, rorting


  15. I have never been picked and no-one I have met has ever been picked

    Neither have I Neil, but look at the numbers… covers approximately 0.33% of the civilian population of Australia aged 15 years and over That’s not very many each month – about 49,000 out of almost 15million.

  16. It is urgent that Mr. Abbott suspends standing orders???

    Yes it is urgent for Mr. Abbott, he only has three days left. She is not listening to anyone anymore. Does that mean she is not listening to him.

    Did you know that PM Gillard is putting the climate change through by stealth.

    If I hear the word “she” again,I will throw something at the TV. Cannot do that now, as I am at my young daughters house. Ms. Bishop says “she” in a such lovely manner.

    They are screaming and uptight, I had to turn my hearing aid down.

    They do sound as time is running out.

    Neither s saying why they cannot get a plebiscite through. The fact is that no one on the cross bench will support of them.

  17. The Opposition are weak. The deputy leader of the Opposition calls the PM every name in the book. Mr. Albanese for once returns the abuse. The Opposition could not take it and called the member to be no longer heard.

    Things have turned nasty. Both sides have pulled the gloves off. At least the governments information seems fresh.

  18. “One thing i do find strange. I have never been picked and no-one I have met has ever been picked or perhaps they did not tell me.”

    Same here. I have only been polled about banks, phone companies and what TV programmes I watch.

  19. Push polling at its worst. Today’s word cloud at the other News site the ABC online.

    How convenient for Abbott who wants a plebiscite to have the ABC run a push poll. Then the Coalition gets to try for yet another suspension of Question Time with Julie Bishop surprise ,surprise being able to use the results of the word cloud.

  20. I once watched a media event about the presentation of poll results where someone in the audience came up with the line:
    “I have never been picked and no-one I have met has ever been picked”
    The presenter went on to explain that being picked to take part in a poll was about as rare as being struck by lightning.
    The questioner then went on to say that they had been struck by lightning!

  21. I’ve been picked 2 times for political polling Shane (and once for a “climate change” survey that I reckon was a dead set push-poll) – that’s in over 35 years as an enrolled voter – just never for the ABS Labour Force survey Neil & I were talking about…

    Once again, that’s not really surprising – there are about 14 million registered voters in Australia & these companies poll around 1000 people once or twice a month- You’re better at figures than me Shane – work out the odds of getting polled 😉

  22. Steve re “ take part in a poll was about as rare as being struck by lightning.” And probably not near as exciting…

  23. Pingback:

  24. Albanese is quite rightly hauling Abbott across the coals for his stunt. What was that scenario again?

    Abbott calls for a plebiscite – Abbott admits that should a plebiscite go ahead and the answer isn’t to his liking that he won’t abide by the result..isn’t that kinda of a waste of umpteen million $s – Abbott who has obviously done zilch work on the small matter of whether he can get his plebiscite passed in parliament is told to take a running leap by none other than Senator Steve Fielding – Abbott phones Fielding to try to get him to change his mind….

    Please please!!! This man cannot become Australia’s Prime Minister!!!!

  25. Obviously the MSM have realised just how stupid and ignorant the idea is Min, they appear to have moved on. Forget the fact that he is still trying to move ahead with it.

    Of course, if the Government had proposed something so stupid, we wouldn’t hear the end of it.

  26. G;day folks – nothing to add, just figured I’d swing by and stick my head in. Rumours of my death have been greatly exaggerated, I’ve just quit Facebook : ) I’ll be back periodically when I’ve got something useful to add.

  27. Damo

    Good for you, I quit Facebook 6 months ago. Never had and never will have Twitter.

  28. Shane, I don’t ‘twit’ either…I would most definitely go into data-overload.
    Facebook is good as a communication tool, but it is a matter of being selective, in fact I can see a big change happening where it changes from social networking to becoming a valid method viz citizen journalists.

  29. …and just a little promotion – I posted at GT –

    People are sick of the mainstream parties, and obviously crap ideologies as represented by the Greens aren’t the answer. Certainly Bob Katter isn’t.
    How much better would the system be if it was the Australian Democrats holding the balance of power in both houses!
    You know where they stand, and can vote for a Democrat candidate with reasonable confidence.
    The Democrats are trying to rebuild at the moment, they are a far better option than the Greens and the independents.
    People disillusioned with the mainstream parties should think about supporting them.

  30. Back again – funny you should mention Twitter as I now pop in there a couple times a day since ditching Facebook.

    Twitter has actually replaced FB for me quite nicely. Whereas I found FB to be a great medium if you wanted to spend most of your days arguing that black isnt white with a seemingly endless supply of argumentative trolls etc, Twitter is more of a one-way medium and much more useful in letting me keep up to date with what I’m actually interested in without having to get into continual pointless debates.

  31. Horses for courses Damo 😉 Clearly with FB one has to be discerning and concentrate on the people who matter and not worry about the people who don’t.

Leave a Comment

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s