Political polling is an integral part of the market research industry, and uses many of the same tools that the broader industry relies on. A key difference though, is that this part of the industry is highly visible to the broader community – and there is an incontestable, real-world event where the accuracy of research can be tested. This means that the confidence of the public in polling has a direct impact on the commercial viability of market research as an industry.
The inconsistency between the polling and the results of the 2019 federal election was a traumatic time for everyone, but provided an opportunity for pollsters to reflect and revitalise. Two major changes have happened since. Firstly, polling companies have been forced to change their methods and the way they weight the data. Secondly, many market research companies involved in public polling have taken the opportunity to commit to new standards of transparency and disclosure by signing up the Australian Polling Council’s Code of Practice.
While some in the academic world imagine that research is mechanical and based on purely random sampling, the reality in the commercial world is that samples need to be constructed through sample frames, and ‘treated’ through weighting during the analysis phase. Everyone in the industry does this, whether they are doing political polling or not, and so they should – samples have never been truly random and have always been prone to skews as some participants are more likely to agree to take part than others, or to answer the phone, or to be at home when an interviewer calls.
The widespread adoption of panels of pre-recruited online participants has made the choices made for sampling and analysis even more critical – a sample drawn in a strictly random fashion from such a panel is likely to differ in systematic ways from the Australian population as a whole. While the panel companies make efforts of course to ensure their panels are as representative as possible, not all participants respond to invitations at an equal rate. Quotas, and post-hoc weighting are therefore essential.
As an industry, we need to ensure that our quotas and weights are sophisticated enough to represent the population faithfully. While quotas and weights by age, gender and location are commonplace, it may be desirable to include other variables such as education, income or ethnicity rather than merely by age and gender. While there are trade-offs involved here because of the impact on weighting efficiency, this has implications for the entire industry – if we need to weight political polls by such variables to deliver a representative sample, then it may be also appropriate to fix these skews in other market research that draws sample from the same sources.
As an industry, there has been a reluctance to talk about the technical aspects of our sampling and analysis, both with our clients and the public. We’ve hidden behind terms like ‘secret sauce’ and the ad hominem assumption that ‘quality’ researchers can be trusted to work their magic. In some cases, we’ve actually implied this wasn’t happening at all, by using concepts like a theoretical margin of error that assumes a truly random sample.
This is unfortunate, because at the heart of it, market research is based on science – and good science is scrutable and replicable. We have a duty to be transparent with our clients and the public and explain what is being done, what variables are being used to weight the data, and what the effect of this weighting is having. The reality is that such disclosure has been inconsistent in the past, both in terms of political polling and the broader market research industry.
The Australian Polling Council was established by nine research agencies that conduct public research that has political implications. Not all of them prefer to be described as pollsters or even publish polling that includes voting intention, but all recognise that published research that has implications for governments and policy makers has a special responsibility to demonstrate that it is professionally conducted, and the way to achieve this is by being more transparent.
In June 2021 we launched the Australian Polling Council Code of Practice. This was the result of six months of study, drafting and debate. Key inputs to the Code were similar documents created by the industry in Britain, New Zealand and the United States, as well as the AMSRO Inquiry report and consultations with TRS and ADIA. The Code requires those companies that sign up to it to provide detailed information about their research and how it is conducted.
A key element is a methodology statement that allows clients and other stakeholders evaluate the quality of the polling and the veracity of the results. Some of the things it requires include the following: the full question text for questions which are published, details of the weighting method that was used, the weighting efficiency or the effective sample size of the research, the margin of error associated with the effective sample size for the research, the type of online panel that was used or how many completed interviews came from a mobile phone rather than a landline (where applicable).
The Code of Practice also requires its members to adhere to all clauses of The Research Society’s Code of Professional Behaviour.
The Code recognises that political polls have a special role in society and therefore needed to have a specialist set of rules in place for them. Nonetheless, while the Code of Practice requires compliance only for where the publicly released polling is related to political issues, or expectations and performance of public policy or governments, the truth is that this sort of disclosure is desirable across the entire market research industry.
We encourage clients, journalists and publishers to embrace the Code and to insist that all research with implications for politics or government they put in the public domain adheres to its standards of transparency. We unashamedly say that these standards for this sort of research should be universal and not limited to members of the Australian Polling Council. Such disclosure has become the norm in the UK, where journalists decline to publish polling that is opaque and doesn’t follow the established standards.
Australians deserve the same.
Author: Dr Campbell White – Chair, Australian Polling Council