1、Lets Chat!:Why Conversational AI is the Future for SurveysBy Phil Sutcliffewith contributions from Josh Seltzer and Kathy ChengPhoto by charlesdeluvio on Unsplash2Table of ContentsWhy Conversational AI Leads to Better Insight01Analysis and Human Understanding enabled by AI03Demystifying the AI behin

2、d chatbots02Engaging Survey Questions045 principles for Conversational AI to maximise insight05What does the future hold for Conversational AI and Market Research?0601Why Conversational AI Leads to Better InsightPhoto by Ghen Mar Cuao on UnsplashConversational AI has the power to reimagine online re

3、search for better insights.In this e-book we explain why anyone working in market research,marketing,UX,CX,innovation or creative development should be excited about Conversational AI.We use examples based on inca,the Conversational AI platform from Nexxt Intelligence.4Why Conversational AI Leads to

4、 Better Insight01To start,lets give a brief explanation of Conversational AIConversational AI leverages the latest advances in machine learning and natural language processing to create compelling conversational experiences.It does this through a chatbot interface to ask questions and converse with

5、participants.Participants can be asked a range of closed,quantitative,and open,qualitative,questions.NLP is used to respond to participants open ended responses in a way that feels natural.The experience for participants is more akin to having a WhatsApp conversation with someone than completing a t

6、raditional online research survey.Its as much about listening to people as asking them questions.The AI is also used to analyse the open ended data,theming it for quick analysis.Fundamentally there are two reasons why Conversational AI leads to better insight:1.Better data quality2.Qualitative insig

7、ht at Quantitative scale5Why Conversational AI Leads to Better Insight01First,lets focus on better data quality.For the last few years the market research industry has often been guilty of behaving like ostriches when it comes to data quality.Most market researchers are aware that some of the data t

8、heyre getting back from online surveys isnt great but many of them bury their heads in the sand and try to ignore it.To be fair,the panel companies have been making great efforts to address the data quality issue through various quality checks,for example to weed out bots or respondents who speed th

9、rough surveys or give a flat line response to questions.However,experienced researchers who have taken the time to go through data line by line have told us that they are still deleting up to 20%of respondents even after the panel companies checks.And,of course,more often than not researchers dont h

10、ave either the time or experience to make these respondent level checks,so there is the danger that less than optimal data ends up being used,with all the risks that entails for poor insight and bad decisions.Whats not happening often enough is addressing the problem at source.That is,creating a bet

11、ter online survey experience for participants so that they give more considered,better answers to questions.This is where good Conversational AI works much better than a standard online survey.6Why Conversational AI Leads to Better Insight01Conversational AI uses a chatbot to engage participants in

12、a conversation.Unlike online surveys which tend to be one directional,question and answer sessions;chatbots are conversational and conversations are two directional,a dialogue where participants are more reflective and considered in their response and which can be more emotional,more fun and lead to

13、 unexpected insights.These same principles are applied to a good Conversational AI survey.This means its important when considering Conversational AI that you dont just use a chatbot designed for customer experience interactions that has been re-purposed for research.Chatbots used in CX are designed

14、 to give answers,whereas for a chatbot to be effective for market research it needs to be designed to ask questions.A conversational format also mimics the way in which people spend most of their time interacting with others online,especially on mobile,for example through WhatsApp conversations.Good

15、 Conversational AI,such as inca,is built on qualitative principles-in qualitative research making participants feel comfortable,engaging them in a dialogue and ensuring they feel listened to are key to generating good insight.7Why Conversational AI Leads to Better Insight01Second,lets talk about the

16、 advantages of qualitative data at quantitative scale.Quantitative data is great at providing the“what”insight.For example:What%of people are aware of different brandsWhat xyz need statements people have when buying a categoryWhat xyz brand perception statements people associate with a brandWhat%of

17、people say they will buy a new product conceptWhat%of people prefer ad A to ad B etc.However,quant is less good at providing the why,which is where qualitative research comes into its own to understand peoples beliefs,motivations,attitudes and perceptions to a much greater level of depth.To date,qua

18、ntitative and qualitative have been separate disciplines and separate projects;extending project timelines and costs when a client needs to know both the what and the why.Getting to the what and the why in the same project has been tantalizingly out of reach,the holy grail of market research!8Why Co

19、nversational AI Leads to Better Insight01Conversational AI gets us closer to that holy grail.Whilst were not claiming that Conversational AI can get to the same level of deep insight as extensive,well conducted qual research,what it can do is provide much more why insight than standard online quant

20、research.Conversational AI does this through use of open ended questions and follow up probing to generate rich verbatim data.Doing this effectively leads to much deeper insight,as illustrated by this client quote following a recent inca project:However,its not as simple as asking a lot of open ende

21、d questions.The questions need to be well structured to engage people and encourage good response and,importantly,smart probing is needed which is where the AI comes in(more on this later).There is also the issue of how to analyze a large volume of unstructured,open ended data at speed another featu

22、re of the AI which Ill return to later in this e-book.“Ive read it(the report)twice and cant believe the wealth of insight youve uncovered!this is really great stuff!”9Why Conversational AI Leads to Better Insight01One of the unique things about incais the Conversational AI not only asks lots of ope

23、n ended questions with smart probing but there is also the option to use qualitative projective techniques at scale.These can get to even deeper levels of insight.For example the guided fantasy projective takes participants on a journey to a brand planet(e.g.planet Apple)and they are asked what thei

24、r feelings are on the journey,how they feel when they get to the planet and then,perhaps,how they feel as theyAnother example of a projective technique we commonly use is Treeman.This projective is particularly useful to get people to express frustrations,fears,hopes,aspirations etc.and,as such,is r

25、eally insightful for early stage innovation research.However,it also has a number of other useful applications,for example to get a deeper understanding of how people feel about their job,as shown in the example below.leave planet Apple to journey to planet Samsung.Hopefully you can see how this typ

26、e of projective can lead to more insightful answers than just asking a participant“how do you feel about Apple”.02Demystifying the AI behind chatbotsPhoto by Victor Larracuente on UnsplashLast chapter we mentioned that one of the ways a good Conversational AI approach generates depth of insight is t

27、hrough the use of appropriate probing.Good probing leads to more detailed verbatim response from participants,resulting in better insight.However,not all Conversational AI approaches probing in the same way and the degree of intelligence in the AI differs.In this chapter,we illustrate how incaapproa

28、ches probing.11Demystifying the AI behind chatbots02There are two ways in which the incaAI works for probing:Note that these two types of probes are in addition to user defined probing which doesnt actually use any AI but can nevertheless be useful think of this as the type of probing you might use

29、when writing a qualitative discussion guide.For example,we might program inca to ask a question such as,“if you were the CEO of(company X),what one thing would you do to improve this product for customers like yourself?”.This could then be followed up with a user defined probe such as,“What makes yo

30、u give that such a high priority?”1.Smart Probing 2.Targeted Probing 12Demystifying the AI behind chatbots021.Smart Probing Smart probing is where the AI reads what the participant says and delivers an appropriate smart probe in response.Depending on the nature and depth of the participants initial

31、response the type of smart probe will be different.In this first example,the participant has given a one word answer,as is often the case with online survey verbatim questions,inca smart probe asks the participant to be more specific in order to elicit more feedback.In the second example,the partici

32、pant provides a more interesting answer with a little more detail,yet still a brief response.Here inca picks up on a theme or word from the verbatim to ask“why”.1213Demystifying the AI behind chatbots02In this third example of smart probing,the participant provides a good,insightful answer.However,i

33、t is still worth probing to see if they have anything else to tell us and inca does this,per the conversation.Finally,if a participant enters gibberish,another fairly common issue with online survey verbatims,inca pulls them up on it and seeks further information.1314Demystifying the AI behind chatb

34、ots022.Targeted Probing Targeted probing is part pre-defined probe,part smart probe.The way it works is we set a pre-defined instruction for the AI to use a smart probe if the participant mentions a particular term.In these examples,we we were particularly interested in probing further if the partic

35、ipant mentioned either AWS or technology when giving feedback on an an AWS(Amazon Web Services)ad we tested,so we set those terms as targeted probes.Note that the second example illustrates the ability of the AI to do“fuzzy”search,that is to pick up the targeted word without the exact spelling(in th

36、is case tech instead of technology).1415Demystifying the AI behind chatbots02Hopefully this chapter illustrates how good Conversational AI,such as inca,uses probing to elicit detailed,verbatim response from participants.Clearly this has the potential to generate a lot of rich insight,much more so th

37、an the open ended response from traditional online surveys.However,a large volume of open ended,unstructured data brings with it the challenge of how to analyse it all,particularly when project timelines are tight.But lets save that for the next chapter.03Analysis and Human Understanding enabled by

38、AIPhoto by Fabio on UnsplashLast chapter,we discussed how we use AI to probe in Conversational AI chats and generate large volumes of rich,open ended feedback.Once weve gathered all that unstructured data,we have the challenge of how to analyse it all,particularly when project timelines are tight.So

39、 in this chapter our CTO and AI expert,Josh Seltzer,explains the second way in which AI works in Conversational AI-that is to cluster and theme verbatim feedback.17Analysis and Human Understanding enabled by AI03Natural language processing(NLP)is the key ingredient of incasConversational AI capabili

40、ties.You can think of it as using machines to try to understand human language-which,as well see,in this context can be used for automatically clustering texts into meaningful themes(or,in market research terminology,codeframes).A nice explanation of NLP is given by IBM:NLP combines computational li

41、nguistics-rule-based modeling of human language-with statistical,machine learning,and deep learning models.Together,these technologies enable computers to process human language in the form of text or voice data and to understand its full meaning,complete with the speaker or writers intent and senti

42、ment.”Moving beyond this basic definition of NLP,however,the way in which it is applied differs depending on the use case.“1803Many applications of NLP make use of a large training set of labelled text or voice data,and use machine learning to then classify new(unseen)text or voice data into those p

43、redefined labels.This works well in certain contexts,for example for call centre conversations,where common problems and questions predominate across many different calls.With market research verbatim data,however,it is not as straightforward.Market research surveys cover a wide variety of topics ac

44、ross different projects,each with a substantially smaller sample size than what might be seen in other contexts(think hundreds instead of thousands,or even millions).Different approaches used in the market research context have different ways of dealing with these constraints;many try to focus on co

45、mmon topics that generalize across many surveys,and amass as much training data as possible corresponding to those topics.inca,however,makes use of unsupervised clustering techniques,fine-tuned on troves of open-ended market research data,which looks for semantic relationships between all the verbat

46、ims within a survey and groups them accordingly into themes.From there,a representative sentence is chosen from each theme,which describes that theme in the participants own words.Analysis and Human Understanding enabled by AI1903Each verbatim is chunked,so that if a participant mentions multiple id

47、eas,each one might be assigned to a different theme.The end result is that inca produces bottom-up themes which dont need to conform to a preconceived codeframe,so that unique and context-specific ideas wont be discarded in favour of the few topics which reoccur across many surveys.Or,in other words

48、,incas thematic clustering has been trained on tons of market research surveys,but it isnt limited to themes that it has seen before,and can therefore generate themes for whatever people are saying in your survey!Theres one other aspect of incasthematic clustering that makes it particularly smart.Al

49、though it still takes into account keywords and other common features such as sentiment,we all know that language is incredibly complicated,and that there are a lot of different ways of expressing the same idea.Analysis and Human Understanding enabled by AIFor that reason,inca looks beyond what is c

50、alled lexical similarity(e.g.sentences that contain the same words),and instead represents sentences based on semantic similarity(where the underlying meaning of the participants utterances are represented).As a concrete example,even though the phrases oh I cant afford that”and“its way too expensive

51、 dont use any of the same words,the idea is the same,and so with semantic similarity they can be grouped together into the same theme.2003This process happens in real time,so that all of incas open ended data is provided in themes on the dashboard as soon as fieldwork is finished.An example is shown

52、 below where the AI classified the verbatims for a question about“how to improve the ad”into 11 themes.The verbatims for each theme are shown on the right hand side,in this case the verbatims associated to the second(highlighted)theme are shown.The AI selects what it feels is a representative verbat

53、im to use as the title for each theme.Analysis and Human Understanding enabled by AI202103As you can see from the example,the clustering into themes is good but not perfect.Therefore,with incawe include a feature called QuickTag.This allows the researcher to easily and quickly edit the themes.For ex

54、ample,the researcher may want to title one of the themes differently e.g.theme 3 would be clearer if it was titled“more clearly explain what the program is”.The researcher can simply over-write the theme title to make this change.Or the researcher may feel that the 5th verbatim shown above,i.e.“Some

55、 people might lose interest if they dont truly know what the program is”would fit better in the 3rd theme.To make this change,the researcher simply drags and drops the verbatim into the different theme.Generally we find that the NLP is very good at identifying the key themes.This provides a great an

56、alysis tool for the researcher who can quickly understand the key patterns in the data and dive into the verbatims for each theme to find good examples that help tell the story from the data.Often this process is enough for the researcher to be able to make the most of the rich verbatim data from in

57、ca to identify key insights and tell a compelling story.However,if the researcher wants to build the best possible codeframe,we typically find that the AI gets us 80-90%of the way to a really good codeframe.The researcher then only needs to spend a little time to use Quick Tag to finalise the codefr

58、ame.Analysis and Human Understanding enabled by AI22Analysis and Human Understanding enabled by AI03Hopefully this chapter and the last one have explained how AI is used in Conversational AI to gather rich,insightful verbatim data and to theme the verbatim to enable the researcher to quickly identif

59、y the key storyline(s)for their analysis.In the next time well turn our focus to more typical quant survey questions and illustrate how Conversational AI can bring these questions to life for participants in a fun and engaging manner.04Engaging Survey Questions in Conversational AIPhoto by Keren Fed

60、ida on UnsplashThe last two chapters have focused on how Conversational AI is used to generate and theme a lot of rich verbatim data.Conversational AI,however,really comes into its own as a Qual x Quant approach.In our opinion,the best case of Conversational AI at the moment is to significantly impr

61、ove the online survey experience for participants and deliver better insight as a result.Therefore,it is important that with Conversational AI we focus not only on verbatims but also on engaging participants through improved versions of typical online quantitative questions.In this chapter,well show

62、 some examples of how Inca does this through its Conversational AI.24Engaging Survey Questions in Conversational AI04There are 3 ways in which we can improve the survey experience with Conversational AI:1.More visually engaging&intuitive questions2.Gamification 3.Combining quant and qual questions25

63、Engaging Survey Questions in Conversational AI041.More visually engaging and intuitive questionsAt the most basic level,we make scale questions visual for participants and deliver an appropriate experience for however they are taking the survey,mobile desktop or tablet.This can be seen in the exampl

64、es below.26Engaging Survey Questions in Conversational AI04Beyond these basic examples,anyone writing a questionnaire is able to easily import appropriate images for the question they are asking.This can be seen in the examples on the right,firstly by using emojis rather than words to understand how

65、 people feel about the topic.Secondly by using photos of famous monuments to engage in a question about countries people would like to visit.The overall experience is one that is more aligned with how people interact with social media or pretty much anything online these days,leading to a more engag

66、ing experience for participants and better data.27Engaging Survey Questions in Conversational AI042.GamificationWeve previously discussed how incauses qualitative projective techniques to elicit deeper,richer verbatim.However,projectives can also be used to gamify the quantitative research experienc

67、e.One example that we use with inca is the hot air balloon exercise.Here we ask people to imagine they are in a hot air balloon and they must throw items over the side to make it fly.This can be a fun way to understand how people would rank a list of items,for example brands or potential features fo

68、r a new product.We ask participants to get rid of their least favourite item first and then throw other items overboard until we are left with their favourite item and have identified their ranking from worst to best.2728Engaging Survey Questions in Conversational AI043.Combining quant and qual ques

69、tionsGiven that Conversational AI at its best,such as inca,is Qual x Quant,many of the questions we ask combine quantified data with open ended feedback.For example,typically for any research that involves showing a stimulus such as a concept,an ad or a video,we ask people to click what they like,di

70、slike it or find confusing,providing a quantified read on the most and least motivating elements of the stimulus.Having done that,participants are asked why,enabling us to link these motivating,demotivating or confusing aspects of the stimulus with the verbatim that helps us understand why participa

71、nts evaluated the stimulus that way.In the first example,with the video ad on the left we can see the peaks where most people like or dislike something about the ad.Here the verbatims are shown which explain why that part of the ad was particularly liked or disliked by participants.2829Engaging Surv

72、ey Questions in Conversational AI04Overall,the use of imaginative,engaging quantitative questions together with lots of open ended questions with smart probes leads to a much better experience for participants and richer,more insightful data for researchers.055 principles for Conversational AI to Ma

73、ximise InsightPhoto by Johannes Plenio on UnsplashSo far,weve shared facts about Conversational AI,how it works and why it leads to better insight.Weve discussed why qual at quant scale leads to better insight,how projective techniques and AI smart probing are used,how NLP is used to quickly theme l

74、arge volumes of open ended data and how quant questions are improved to drive participant engagement.In this penultimate chapter,were sharing 5 broad principles that should be built into Conversational AI design to maximize insight.31051.Build Rapport If we ask the right questions,using the right la

75、nguage and the right tones for the right audience,then we know they will reciprocate.We know that the extra effort put into scripting pays off when we receive vivid consumer language in return.2.Demonstrate Active ListeningChatbots create the impression that someone else is listening.Respondents say

76、 it motivates participation if they feel they are heard.Appropriate acknowledgment and segues throughout the conversations are effective ways to show good listening.3.Ask Smart Follow-up QuestionsRelated to active listening,we can turn acknowledgements into probes.Asking intelligent questions is not

77、 just for the sake of the richness of research findings but also an effective engagement measure.Probing for deeper insights shows the chatbot understands and cares.It also conveys the importance that the participants should think harder,too.5 principles for Conversational AI to Maximise Insight3205

78、4.Interface Matters!We need to create an interface to support both open-ended and close-ended questions to optimize the chat experience and research outcomes,but,of course,the close-ended questions need to be designed to fit the chat.Theres an ocean of opportunities to create smart,engaging“survey”q

79、uestions.5.Come Back to the Basics.We often need to remind ourselves why we need Conversational AI in the first place.To us,Conversational AI allows one-on-one attention,regardless of the number of people we are chatting with.One-on-one attention is whats lacking in traditional survey research and i

80、s fundamental to engagement and data quality.5 principles for Conversational AI to Maximise Insight06What does the future hold for Conversational AI and Market Research?Photo by Rachael Ren on UnsplashIn this book,weve provided an introduction to Conversational AI;what it is,its advantages and how i

81、t leverages AI to probe to get rich feedback from people in their own words and theme this feedback at speed.Weve done this through the lens of exploring a few of the ways that inca already uses Conversational AI to enhance the process of understanding people.In this last chapter,our CTO,Josh Seltze

82、r,discusses what the future might hold for Conversational AI and its application to market research.3406As we mentioned,the magic sauce of Conversational AI goes by the technical term of natural language processing(NLP),along with techniques from related fields such as information retrieval and stat

83、istics.In the last few years,the field of NLP has seen some major advancements,which some experts argue constitute a paradigm shift in what is possible with language technology.The technology is new and has a lot of hurdles to face before seeing widespread incorporation across industries,but already

84、 tech startups and enterprise R&D are dumping venture capital and ramping up to understand and leverage the competitive edge to be gained as these technologies mature.Eventually,one could imagine a world of virtual agents regularly conversing with humans,with at least some ability to understand(and

85、even influence)people while emulating the voice of a real human being.While this might sound a bit scaryespecially given the growing prominence of political propaganda and false information on the webeven with the latest advances we are still a far ways away from having language models that can hold

86、 deep,nuanced conversations on their own.To keep things a bit more grounded,for this blog Id like to focus on some concrete examples of how Conversational AI can transform market research even within the next decade.What does the future hold for Conversational AI and Market Research?3506Quantitative

87、 ResearchAlthough some platforms have already started to move towards an increased focus on text analytics and other NLP-powered features such as sentiment analysis,the industry has seen very little use of Conversational AI capabilities to date.While it is probably safe to say that humans will alway

88、s lead the research design process,in terms of deciding the research objective and relevant demographics,the traditional paradigm of devising and analyzing fixed questionnaire structures will be challenged by Conversational AI.As we have seen in the previous weeks,a conversational agent can not only

89、 increase engagement but also elicit richer information from respondents.Conversational AI will be built into the role of a virtual moderator,providing an extra layer of interaction which brings surveys closer to in-depth(or semi-scripted)interviews.Virtual moderators will be trained to detect and r

90、espond to fundamental consumer motivations and opinions such as emotions,brand awareness or perception,impressions and judgments of products and services,and expected behaviour.Virtual moderators will use conversational cues to pick up on these dimensions,but will also proactively probe consumers to

91、 explore and to clarify their perspective.Nuance,rationales,and unanticipated opinions will become commonplace expectations of quantitative research,ultimately blurring the lines between quantitative and qualitative.What does the future hold for Conversational AI and Market Research?3606Qualitative

92、ResearchWith COVID-19 having forced qualitative research online,digital qualitative research has grown and this trend is expected to continue.Although qualitative research will never lose its quintessential human dimension,the digital nature of online interviews and focus groups provide ample opport

93、unities for Conversational AI to improve researchers workflows.Moderation Assistant AI will be increasingly used to automate the tedious and painful parts of digital qualitative research,for example prompting participants to provide more information or to ask common follow-up questions.Tagging/Analy

94、sis/Reports basic features like sentiment and keywords will be replaced by more sophisticated thematic and semantic analyses of conversations,allowing researchers to derive and aggregate insights across many different conversations.Multimodal machine learning this is a hot area of research which wil

95、l extend Conversational AI beyond the text-or-voice modalities,allowing truly integrated analyses of human facial expressions,vocal cadence and intonation,and semantic meaning of utterances to produce a holistic understanding of consumers which can be operated at-scale.What does the future hold for

96、Conversational AI and Market Research?3706Social MediaConversations on social media constitute a huge portion of the data that large language models are trained on.Social media monitoring is already a key tool for understanding consumer opinions(and dissent)for marketers.As NLP advances we can expec

97、t sentiment models applied to social media data to become more accurate and nuanced,resulting in a much better understanding of the opinions people voice online and what it means for brands,for example by gaining a more accurate read on the emotions behind the language people use.What does the futur

98、e hold for Conversational AI and Market Research?3806Fraud/Fraud Detection&Simulated ConversationsLarge language models such as GPT-3 make it possible to generate texts that are convincingly human(at least,until held up to a fairly high level of scrutiny).The downside is that the technology is ready

99、 to be abused by fraudulent respondents and,more worryingly,at scale by bot farms.Though at this point it may still be prohibitively expensive for large language models to be abused in this manner,in the very near future it will be possible for fake respondents to make use of NLP to input plausible

100、sounding open-ended responses.To counteract the above,researchers are hard at work developing methods for detecting when text has been generated by a language model.In the interim we may see more use of methods such as asking respondents a nearly-identical pair of questions at the start and end of a

101、 survey,to ensure that their answers are consistent(logical inconsistency and limited long-term memory are one of the major constraints of current language models).What does the future hold for Conversational AI and Market Research?3906What does the future hold for Conversational AI and Market Resea

102、rch?Its possible that companies may emerge that offer respondent-less market research via a proliferation of low-cost services for simulated“panels”,maybe called“aggregate personas”,which use language models trained on specific demographics of participants.We should be wary of such services:the illu

103、sion of real-sounding conversations specific to your research objectives could be created with the click of a button,but its important to understand that these are statistically-plausible inventions based on existing data(typically scraped from the internet).In other words,language models can genera

104、lize to previous conversations that they have seen,but if your research objective is unique or nuanced then you need genuine human understanding and judgment.4006So what does this mean for market researchers?1.Statistical and analytical methods will need to be extended to incorporate insights derive

105、d from open-ended conversations,in order to make sense of huge amounts of conversational data.2.Blended methodologies which interweave quantitative and qualitative insights will become the norm.3.The industry of conversational design,which has already grown to support customer support agents and oth

106、er uses of Conversational AI,will bleed over into market research.Research will increasingly involve designing dynamic conversations around targets of interest.The impact will be deeper,richer understanding of people and,consequently,a stronger base of insight for business to make better decisions.T

107、his can only serve to strengthen the importance of research and insight to organisations.What does the future hold for Conversational AI and Market Research?Thanks!Our aim with this e-book has been to show you why we believe Conversational AI is the future for surveys.Please get in touch if youd like to experience this future for yourselves.philnexxt.inkathynexxt.inPhoto by charlesdeluvio on Unsplash