Artificial intelligence, once something of science fiction – is now science fact.
Matthew Jorgenson explains how bots can be used for qualitative moderation.
Some say the artificial intelligence (AI) revolution will change the world for the better.
Others say it will be our destruction; we’ll be taken over and killed off by super intelligent machines. Others say it’s all a load of ‘hype’ and nothing much will come of it.
According to the 2016 Greenbook Research Industry Trend (GRIT) Report, only 23 per cent of research professionals believe AI is a game-changer. Yet most weren’t sure what to make of it.
I believe we not only need to learn to live with it, but that we should smartly adopt it and adapt how we work.
The holy trinity: faster, cheaper, better
AI technology brings us closer to delivering the holy trinity of market research: offering fast, cost effective solutions that deliver quality outputs. We thus see a (near) future where we can truly start to capitalise on the benefits of AI, allowing us to do more with less.
Working with Heineken, we took on this exact challenge in late 2017. Heineken have recognised that the traditional approaches to innovation are too slow for a fast-paced future. They needed to find a way to do things faster with shrinking budgets. Our approach was to explore how we could leverage new technologies in online qualitative research to reshape the innovation journey.
We wondered whether we could engage consumers in shorter, chat-based engagements, and then how we could further streamline that process by handing some of the heavy lifting of moderation over to bots.
Traditional forum versus chat-based discussion
Our first experiment was to explore how consumers responded to a chat-based qual discussion. We ran a parallel test with a chat-based group (in WhatsApp) alongside a traditional forum conversation. Our satisfaction survey at the end of both groups confirmed our hypothesis: chat can help consumers feel more bonded with one another (and their moderator) and to engage in more conversational-style discussions. We found that chat- based conversations brought more in-the-moment contextual
information and specifics as people didn’t have to rely on recalling past experiences as often.
The chat approach worked best for simple exercises, as consumers who entered the conversation half way were often reluctant to go back and catch up on what they had missed. As a result, we learned that conversation needed to follow a simple and linear approach
that would make sense no matter when a participant steps in. For anything that didn’t fit this bill, we found that a more traditional online approach made more sense.
The benefits of chat-based research were clear; it could yield some useful outputs.
But what if we automated that chat? Could we reduce the role of the moderator even further – freeing up their time to focus on the thinking?
Putting bots in charge
We decided to put this to the test by using three types of bots with different moderation roles. We programmed these bots using Facebook Messenger:
- Interviewer Bot – this bot was programmed to mimic a qual moderator, asking the questions from a discussion guide; no probing, just asking the questions.
- Ethnographic Bot – this was related to a specific challenge where we wanted people to report on an activity. For example: on a day, every time you are drinking, take a picture and tell us about where you are, what and why you are drinking at that time.
- Probing Bot – this bot was our smartest bot as it could It could pick up on an answer, and say ‘oh can you tell me more?’ or ‘I don’t think I quite understand, what do you mean by X or Y?’
Our consumers were briefed that they would be working with bots, and that these bots were still ‘young’ and needed ‘training’.
In addition, a moderator kept an eye on things to make sure the conversation didn’t go completely off track and could assist participants if required.
We weren’t sure how consumers would react to the bots, but we were pleasantly surprised; consumers really enjoyed interacting with the bots. We feared they might try and actively confuse the bots – however we found that consumers started empathising with the bots. We think this was the case because the bots were not quite perfect, and because we’d been upfront with participants that the bots were still in their early stages of development.
The bots were smart, but they were not so smart that consumers would resist the bot or be ‘afraid of it’. AI Researcher Justine Cassell noted that previous cases showed, as soon as AI starts to feel almost human, people start to feel uncomfortable with it. Bots that try to be
too human (tell jokes, use slang or engage in banter) risk pushing the boundary too far. Our ‘imperfect’ bots did not have this problem.
We ran a parallel test of bots versus humans in this experiment to have a benchmark of success. When we compared our results from the bots with a parallel human-moderated study, the overall research finding with the bots was largely the same, but we had saved approximately 80 per cent of the time that would have otherwise been spent on moderation and analysis.
This was because the bots did not only take over a lot of the moderation load (following up participants, probing, greeting people etc.), but they also produced outputs in a neat, tidy, and consistent fashion making it much easier for the researcher to make sense of the outcomes and findings.
We saw this as a first step to unlocking what could very well be a powerful technology for our industry. Human moderators and researchers still very much have a role in this process, but bots offer a chance to take over a lot of the heavy lifting.
Our experience shows that bots can indeed help us to streamline research; the opportunity is to continue to further explore these approaches to help us all get to a place where we can really start to do more for clients with less.
DIRECTION FIRST INSITES CONSULTING
This article first appeared in the October-November 2018 edition – Research without Researchers: AI & New technologies – of the Australian Market and Social Research Society’s publication Research News.