Dianne Gardiner explores the dangers of the ‘some research is better than no research’ mentality.
The need for speed is real. This is not new news – but dealing with the implications of the continuous push for shorter research timelines remains a burning issue for client-side insights managers who need to deliver speed and accuracy. This was reaffirmed when Bastion Latitude recently undertook a series of interviews with 16 clients, including insights managers and marketing managers, to understand the state of the industry from their perspective.
A key theme to emerge from these interviews was, like it or not, organisational agility and Agile software design and development methods are being embraced by more businesses as iterative ‘test and learn’ environments gain traction. Insights teams face being pulled into ‘sprint’ teams (which can be problematic to maintaining their insights role) or risk being marginalised and left out of projects all together. Their key stakeholders – designers, developers and Agile team members – often don’t see getting customer feedback as a specialised skill. What this boils down to is a mentality that ‘some research is better than no research’. This lack of rigour and prioritisation of speed ultimately results in ‘guerrilla’ or ‘rogue’ research.
This is having an impact on research teams in the following
• Timelines are being compressed into days, rather than weeks or months
• Budgets for development work may be tight or non-existent
• The depth of research is reduced, as team members ask how much information is necessary to make the informed decision, using whatever information they have access to and/or using rogue research approaches to get a ‘quick read’ without input from the research team.
Client-side researchers are worried about rogue research, not because they are wedded to traditional market research or research based on a solid scientific foundation but because it can lead to bad decisions being made; decisions with costly implications. Rogue research also undermines the value of all research and leads to a cycle of bad decisions and attitudes e.g. ‘we conducted research last time, and it led to a poor decision, so let’s not invest in research anymore’.
Rogue research is more dangerous than no research at all because it gives people the confidence to make bad decisions. This is best encapsulated in a quote from the film, The Big Short: ‘It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so.’
If we accept that rogue research is dangerous, then we can turn our attention to identifying its characteristics and developing solutions to avoid it.
Characteristics of rogue research
Our study with client-side researchers identified three characteristics of rogue research:
- Dodgy sampling: this is often due to an over reliance on convenience sampling i.e. just speaking to anyone that’s available. There is an obvious risk in poorly selected respondents on the quality of outputs (i.e. asking six people on the street or, even worse, in the office). Who cares what the guy on level 20 said, if they are not in your target market!
- Flawed questioning: the most obvious examples provided here related to the priming of respondents with leading questions or failing to build context around the subject matter of interest (e.g. asking about the user interface before establishing a usage occasion or need for the service altogether). Client-side researchers also pointed to the ability of skilled moderators to get under the surface level reactions/reasoning and to understand the importance of what is often not said.
- Shallow analysis: this comes down to the ability of the researcher to conduct root cause analysis, identify themes and go beyond describing the results to provide true insight. One client demonstrated the difference to his internal team by sitting in on an in-home interview with a sprint team member and comparing notes at the end of the session. His eight pages showed a lot more understanding and depth than the few scribbled notes of his lesser-trained colleague. And, as any good qualitative research will tell you, it’s not just what they say that matters!
Solutions to avoid rogue research
To reign in rogue research, insights teams need to adapt their processes and work to accommodate Agile business processes through adopting some or all of the following:
• Develop just-in-time solutions with research partners who can execute projects in two to three days and at lower costs. This is about having ‘off the shelf’ solutions that are ready to go for certain situational needs that are known to occur frequently (e.g. concept sessions/prototype refinement etc.). Insight communities or customer panels can provide the infrastructure.
• Explore automation of certain steps in the research project. This
is especially relevant for repeatable projects that can follow consistent principles in terms of the survey instrument and the analysis frameworks.
• Implement knowledge management systems for historical research so that stakeholders can easily find relevant findings from past studies that relevant to the task at hand.
• Seek research partners who are flexible and can pivot as parameters change at the eleventh hour.
• In-source: bring the research in-house and help execute surveys or interviews to ensure quality is maintained. This can also be done via training and capacity building.
• Use syndicated or existing research to offer research input upfront to help shape opportunities and provide a ‘sense check’ for ideas/concepts.
• Fund foundational studies where possible – insights teams are looking to fund broader foundational studies that provide benefits across business units and help prioritise opportunities for innovation.
While obviously not all research is done for Agile teams, organisational agility is not going away. More and more, data and insights specialists are required at ever increasing speed to assist in business decision-making. Therefore, I am not interested is arguing about speed versus accuracy but would rather ask, how can we deliver speed and accuracy? Because in market research, sometimes something is more dangerous than nothing!
Author: Dianne Gardiner, CEO, Bastion Latitude
This article also appears in the February-April 2019 edition of AMSRS publication, Research News – Speed vs Accuracy