What is the current state of play in government and social research when delivering the research trinity and what is likely in the future? Darren Pennay reports.
A popular joke in the survey research profession says that three things are valued above all: quality, speed and affordability. The punch line is that you can have any two that you want. The reality facing government and social researchers however is quite different; clients often require that we deliver on all three. This sets up an inherent tension, one that inevitably involves trade-offs and judgements about delivering ‘fit-for-purpose’ research solutions.
If quality is the overriding dimension then probability-based samples are still the prerequisite and will remain so into the future. There are preponderance of studies in the survey research literature which document, beyond contention, that when compared to survey estimates generated from non-probability samples, even the low response rate probability surveys that we have to live with nowadays are less biased, less variable and generally benefit from the application of standard weighting techniques.
Above all else, probability-based survey estimates can be generalised to the population of interest – and it’s possible to calculate sampling errors and confidence intervals. Survey estimates generated from non-probability samples simply do not have these highly desirable design-based statistical properties.
I see this reliance on probability-samples continuing for surveys when quality is the overriding criterion.
Speed and affordability
The increase in online research in Australia tells us all we need to know with regard to speed and affordability. Industry data (ESOMAR Global Market Research Industry Reports, 2012, 2014, 2016 and 2017) shows that between 2011 and 2016 the proportion of total industry revenue attributable to online research increased from 29 per cent to 44 per cent. Over the same period, CATI’s share of research industry revenue declined from 22 per cent to six per cent. Of course, most of the revenue generated by online research is from non-probability online panels, which have been adopted more fervently in Australia than many other places in the world. However, for government and social research, speed and affordability can mean diminished quality (i.e. accuracy).
Delivering on the research trinity
Research purists who hold on to the primacy of probability sampling at the exclusion of all else will end up on the wrong side of history. One of the emerging trends, both in Australia and overseas, is the increasing use of probability and non-probability samples in combination. This applies for both online research and CATI. Combining probability and non-probability samples to deliver quality, speed and affordability was the theme of the 2018 AMSRS conference paper I co-authored with Stephen Prendergast (managing director, Prescience Research) and Dina Neiger (chief statistician, The Social Research Centre). In this paper, we provide an introduction to using the statistical properties of probability surveys to reduce the biases found in surveys administered on non-probability online panels. This is a growing area of work internationally and one that I suspect will continue to expand as clients (including academics) become increasingly wary of the pitfalls of relying overly, or solely, on survey data generated from non-probability samples.
The other trend I am starting to see in terms of probability-based surveying of the general community, made possible via access to the Geocoded National Address File (G-NAF), is the increasing use of address-based sampling approaches in combination with the ‘push to web’ methodologies as espoused by Don Dillman and others.
What does the future look like?
By way of summary, the emergent methodological trends in government and social research as I see them are as follows:
• The decline in demand for CATI has almost reached its nadir and, in future, CATI will need to become part of single-frame, mixed mode and lower cost data collection solutions
• The increase in demand for surveys to be administered on probability-based online panels will continue
• Mixed methods, mixed modes, mixed data (e.g. survey data, admin data, social media data, auxiliary data, paradata, etc.) and mixed sample approaches will continue to increase in popularity.
This last point is particularly salient, as herein lay both the opportunity and the risk. The opportunity is that this more complex research environment enables us to continue to provide high quality data to inform decision-making and help shape our understanding of the world we live in. The risk is that unsophisticated or naïve approaches to this more complex environment (e.g. just bunging together data and hoping for the best) will quickly erode trust and damage our profession.
Although we can move to lower cost data collection platforms and better utilise existing data sources, we also need to realise that pre-existing data are rarely fit-for-purpose. This means that some of the savings realised by gaining access to cheaper data need to be invested in sophisticated statistics and methods to ensure the validity and reliability of the resultant data. If we don’t pay due regard to the complexities of the new reality, it will just be a race to the bottom!
Author: Darren Pennay, Founder & Executive Director Research, methods & strategy, The Social Research Centre Pty Ltd
The views expressed in this article are those of the author alone and not necessarily those of The Social Research Centre or its clients.
For more information
The 2016 AAPOR Taskforce report, titled ‘Evaluating Survey Quality in Today’s Complex Environment’, is available for free from the AAPOR website at https://www.aapor.org/Education-Resources/Reports.aspx. This report sets out ‘a series of 17 questions that users of survey data should ask to help them make judgements about the validity of a survey’s results regardless of the method used.’
This article also appears in the May-July 2019 edition of AMSRS publication, Research News – State of play. Check out the rest of the articles in this edition.
Photo by Paul Smith on Unsplash