《Canvs:开放式问题分析入门指南(英文版)(31页).pdf》由会员分享,可在线阅读,更多相关《Canvs:开放式问题分析入门指南(英文版)(31页).pdf(31页珍藏版)》请在三个皮匠报告上搜索。
1、Core InsightsThe Essential Guide to Analyzing Open-Ended QuestionsPresented byDeeper Consumer Insights&Analytical Confidence1Introduction There are few,scalable substitutes for the nuance and detail that open-ended questions(a.k.a.“open ends”)deliver as part of research studies and customer feedback
2、.They help present a more complete picture of consumer attitudes,preferences and feelings,all without the bias of a priori answer options.In some cases,theyre critical to the research mission itself,as in unaided brand awareness and recall,or analysis of consumer reviews and comments.Insights from o
3、pen ends help:Complete the picture.Open-ended questions allow respondents to provide feedback that might not be possible using the close-ended questions alone,or give rationale or detail around a particular response,like a rating.Discover unexpected findings.Open ends generate surprising or unexpect
4、ed fjndings that werent considered in the programming of the study;Reveal emotional reaction.Open-ended text is a source of emotional reaction by consumers,providing the opportunity to understand not just what a customer is saying,but how they feel about it.Add color and narrative.Perhaps foremost,o
5、pen ends provide a source of consumer narrative,supported by the adage:“fact tell,but stories sell.”Deeper Consumer Insights&Analytical Confidence2Unfortunately many research and CX teams dont fully-utilize open ends as a source of statistically signifjcant insights,often relegating their use to a f
6、ew verbatim quotes to spice up their report,confjrming fjndings derived from the closed-ended,quantitative questions alone.The biggest risk of this approach of course is missing critical,“core”insights that are locked in the unstructured text data of open ends.For example,a leading food company woul
7、d have missed how consumers really feel about their retail shopping experience without analysis of open ends.A restaurant chain would have missed how their guests are responding to new menu items without open ends.A leading moving and storage provider would have missed why customers gave the rating
8、they did.And a media&entertainment brand would have missed how viewers feel about their new program without open ends.Deeper Consumer Insights&Analytical Confidence3To put it simply,we believe that open-ended text is under-utilized as a source of core consumer and market insights.With that mission i
9、n mind,weve developed this eBook as a guide to consider both the types of insights that can be generated from open ends and their application across common research and CX use cases.We believe that with the right approach,the unstructured,unfjltered text data of open ends can help insights professio
10、nals and the organizations they serve better understand their customers,audiences and markets.In working with insights teams analyzing many thousands of surveys and millions of open ends,we have a unique perspective on the value that open ends deliver.Well look at how open ends can be analyzed gener
11、ally and then how this analysis can be applied across six insights and customer experience use cases.Our mission at Canvs AI is to help researchers,insights and customer experience professionals unlock the full value of open ends as a means to better understand consumer attitudes,preferences and fee
12、lings across a range of research types.Deeper Consumer Insights&Analytical Confidence4Section 1:Framework for Analyzing Open-Ended Text.6Thematic analysis.7Emotion analysis.8Correlation analysis.9Section 2:Open Ends in Research Context.10Consumer Behavior.11Brand Studies.12Customer Satisfaction&Expe
13、rience Surveys(aka NPS surveys).17Product Feedback.20Ad&Creative Testing.23Product Reviews,Social Media and Non-Survey Verbatims.26Section 3:Summary.28About Canvs AI.30Table of ContentsDeeper Consumer Insights&Analytical Confidence5Analyzing Open-Ended TextAt fjrst glance,analyzing text might seem a
14、s simple as answering:“What did the respondent(or customer)say?”And that is exactly what we want to understand,but its often far from simple!Humans are amazingly expressive(even in surveys),developing seemingly infjnite and varied ways of saying things and imbuing those expressions with nuance and e
15、motion along the way.Three(3)vectors of analysis can help organize our thinking around the informational value of open ends:Thematic,Emotion and Correlation analysis.Deeper Consumer Insights&Analytical Confidence6Thematic analysisThematic analysis is the result of parsing the ideas or things that ar
16、e expressed in the text and presenting those ideas in a structured way.Questions that generally ask the respondent for“what”are typically analyzed for themes.For example,“What was the main idea?”or“What brands come to mind?”But questions like“Describe your perfect burger,”might generate topics like“
17、juicy”“medium-rare”and“bacon”among many others(and in various combinations).While a topical analysis may suffjce for some questions,such as brand recall,thematic analysis is frequently organized hierarchically,grouping the responses into categories.Our“perfect burger”question generates responses tha
18、t can be categorized into themes like”ingredients”,“fmavor profjle”and“texture.”Deeper Consumer Insights&Analytical Confidence7Emotion analysisEmotion analysis is an extrapolation of how the respondent“feels”based on how they have expressed themselves.Questions that ask the respondent for“how”or“why
19、”are often ideal for emotion analysis,such as“How do you feel about your current economic situation?”Naturally,some subjects and/or open-ended questions are more likely than others to elicit an emotional response.Even our“Describe your perfect burger”question is likely to generate signifjcant emotio
20、nal responses(given our general love of burgers)!While emotion analysis is often expressed in terms of overall positive or negative sentiment,more granular emotion classifjcation will reveal more nuance in the level of emotional intensity or identifjcation of feelings that dont fjt the positive/nega
21、tive paradigm,such as“nostalgia.”Deeper Consumer Insights&Analytical Confidence8Correlation analysisCorrelation analysis actually isnt analysis of the open-ended text itself,but rather the ability to identify connections between information from the open ends and closed ends,such as respondent demog
22、raphics.We include this here because typical approaches to incorporating open-ended responses into the research process often dont lend themselves to discovery of these connections.Going back to our“perfect burger”question,you may identify a correlation between mentions of the topics“cheddar”and“bac
23、on”and the emotion of“love,”indicating a heightened emotional engagement with certain fmavor profjles vs.others.Or,you may identify that women are more likely to identify texture in their description of the perfect burger vs.men.With this analysis framework in mind,we hope its clear that open-ended
24、text represents a rich and valuable source of(relatively)unfjltered,unbiased consumer feedback!Deeper Consumer Insights&Analytical Confidence9Open Ends in Research ContextLets take a look at how open-ended questions and responses are used across various research types,including samples of typical op
25、en-ended questions and more importantly the“questions”that open ends answer within the context of each research type.Specifically well look at the use of open ends in:Consumer behavior research Brand and awareness studies Product research Customer experience/NPS surveysAd andcreative testingProduct
26、reviews andother“non-survey”analysisDeeper Consumer Insights&Analytical Confidence10Consumer Behavior Consumer behavior research seeks to understand consumer attitudes and preferences for a particular topic,product or service category.Generally this research is designed to inform business decisions
27、or strategies,but could also be deployed to inform journalism and public policy.Examples might include examining consumer attitudes towards business and leisure travel,opinions about curb-side pickup in the grocery category or how infmation is impacting back-to-school shopping.What questions can the
28、 open end answer?The“questions”open ends can answer in consumer behavior studies are driven signifjcantly by the objective and subject of the overall study.When consumer behavior studies capture consumer trends and preferences,the use of open ends can reduce the bias of a priori answer choices.With
29、this in mind,open ends are used to create a more complete picture of consumer habits and better understand sentiment around certain subject areas that might be valuable in shaping go-to-market and creative strategies.Sample Consumer Behavior Open-Ended Questions What do you consider when choosing wh
30、at hotel to book for a vacation?Think of your favorite restaurant.How would you describe it?Describe your perfect sandwich.What are your thoughts or feelings on holiday travel this year?Whats the most important issue facing the country?Deeper Consumer Insights&Analytical Confidence11Application of A
31、nalysisThematic AnalysisThematic analysis in consumer behavior studies allows researchers to identify the top ideas expressed about the subject,as well as to group and quantify respondents that express similar themes,whether those themes are emotional,topical,or broad categories.Emotion AnalysisUtil
32、izing nuanced emotion measurement(vs.general sentiment)can help identify the level of emotional intensity around a subject.Some questions,such as“How do you feel about infmation?”are designed specifjcally to unlock emotion measurement.Correlation AnalysisCorrelation analysis is often used to identif
33、y meaningful difgerences among sub-segments of the sample (eg.demographic cross-tab analysis).Deeper Consumer Insights&Analytical Confidence12Key takeawaysOpen-ended questions are key to discovering(or confjrming)changes in consumer attitudes,and understanding the“why”when a consumers actions change
34、.Consumer behavior studies often inform core product development,brand,and market strategies,and open-ended questions help both brand managers and creatives understand the mind(and heart)of the consumer.Case in PointA consumer technology brand wanted to understand consumer understanding of the metav
35、erse.Given the newness of the subject area,the use of open ends was critical in surfacing both a structured analysis of current perceptions,but also key emotional information.Deeper Consumer Insights&Analytical Confidence13Brand StudiesBrand and awareness surveys are utilized to understand both the“
36、mindshare”that a brand commands petitors within a category,as well as the perceived attributes and consumer attitudes associated with the brand.Brand and awareness studies may also utilize either pre/post surveys(survey of a sample to test the impact or efgectiveness of a message,ad or campaign)or t
37、racking studies(an ongoing survey that typically generates monthly waves of data to compare over time).What questions can the open end answer?Brand studies typically include either or both unaided awareness and aided association questions.Unaided awareness questions provide a trackable metric,unders
38、tanding what percentage of respondents are aware of a brand,product,or campaign top of mind,without assistance.Association questions ask for a respondents thoughts or feelings about a given,prompted brand or product.Sample Brand Study Open Ends“What brands of toothpaste do you know?“What movies or t
39、v shows have you heard about recently?”“What smart-home-devices are top of mind?”.When you think of BRAND X,what comes to mind fjrst?What thoughts or feelings come to mind when you think of BRAND X?What three words would you use to describe BRAND X?Deeper Consumer Insights&Analytical Confidence14App
40、lication of AnalysisThematic AnalysisNaturally,in Unaided questions thematic analysis is used fjrst and foremost to identify the specifjc brands mentioned.In association questions,thematic analysis will identify the attributes driving emotional response.Researchers can track KPIs of awareness overal
41、l using unaided awareness questions,or breakthrough of certain messages based on a campaign(association of family with a bank brand,for example).Emotion AnalysisWhile Unaided Awareness questions are,by design,non-emotional,Brand Association questions can elicit extreme and nuanced emotions.Researche
42、rs can use this approach to understand what attributes and themes drive“annoying”versus“hate”,or“enjoy”versus“love.”Additionally,overall emotional response can be measured to establish a benchmark that can be tracked over time.Correlation AnalysisBeyond demographic and brand-specifjc cross-tabs,corr
43、elation analysis can be used to identify the emotional resonance of common brand associations,or understand how a brand is resonating across demographic segments.Deeper Consumer Insights&Analytical Confidence15Key takeawaysOpen ends are not only a critical tool for measuring unaided awareness,recall
44、 and attributes,they can reveal powerful information about how consumers emotionally relate to brands and brand experiences.Case in PointA consumer fjnancial services provider runs a monthly brand tracking study to understand the impact of their campaigns on awareness and perception.They include the
45、 questions:“When you think of your personal banking,what brands come to mind?”and“What are your thoughts and feelings about the brand?”The fjrst,unaided awareness question is straightforward,allowing the brand to measure the percentage of consumers who included their company in the list.The open-end
46、ed nature of the question allows the company to reduce the infmuence of reinforcing brand mentions in the survey itself.The brand perception question allows respondents to indicate their views of the brand in an unfjltered way.The fjrm analyzes the open-ended responses to categorize them into key ca
47、tegories that are tracked over time,such as“trustworthy,”“community,”and“service.”Deeper Consumer Insights&Analytical Confidence16Customer Satisfaction&Experience Surveys(aka NPS surveys)Perhaps no survey has become as ubiquitous as the NPS survey.First developed in 2003 by Fred Reicheld and Bain&Co
48、mpany,and popularized in the book“The Ultimate Question,”the Net Promoter Score(NPS)survey asks simply,“How likely are you to recommend this product or service to a friend or colleague?”Then,an important open-ended follow up:“Why?”While“NPS”may be the brand leader,there are certainly non-NPS custome
49、r satisfaction/experience surveys in wide use.And while the NPS survey design is notable for its brevity and simplicity(it was developed in part as a reaction to lengthy customer satisfaction surveys),all customer satisfaction surveys are likely to include follow up questions generally intended to p
50、rovide the organization with more specifjc information regarding the customer experience(overall experience,satisfaction with the product,satisfaction with the service,satisfaction with the facilities,etc.).What questions can the open end answer?Open-ended questions in customer satisfaction surveys
51、generally seek to provide detail regarding the customers experience with the product or service.One can think of the open end as providing the“Why”to the customers rating.Sample Customer Satisfaction/NPS Open EndsGENERAL What is the primary reason for your score?Why did you provide that rating?What
52、is one thing we could do to make you happier?What most infmuenced your recommendation?RATING SPECIFIC What was missing or disappointing in your experience?What could we do to improve the product or service?How do you benefjt from using our product/service?How are customer experience open ends analyz
53、ed?Deeper Consumer Insights&Analytical Confidence17Thematic AnalysisThematic analysis is used to identify and categorize specifjc aspects of customers experience that naturally arise in the comments.These can be both positives(essentially,what do people like most)and negatives(what are people unhapp
54、y or complaining about).Emotion AnalysisEmotional analysis can provide color to your NPS scores.Identifying what topics&themes drive emotions,and can provide actionable feedback for teams hoping to improve their NPS scores.Correlation AnalysisPerhaps the most compelling use of correlation analysis i
55、n customer satisfaction research is the use of open ends to identify potential root causes of customer satisfaction/dissatisfaction.Essentially“what topics are more likely to be mentioned by promoters/detractors”in the NPS lexicon.Additionally,correlating emotional engagement with specifjc elements
56、of the experience can also be useful.Application of AnalysisDeeper Consumer Insights&Analytical Confidence18Key TakeawaysIn our consumer-centric culture,customer experience/satisfaction surveys,including NPS,have become ubiquitous.The open-ended questions in these surveys provide vital,and valuable
57、insights into why customers gave your product or service a certain rating or score,and creates an actionable feedback loop between you and your customers.Efgectively utilizing these responses in this way can also help reduce the need for detailed,follow-up closed-ended questions because drivers to p
58、ositive and negative reviews will surface in the open ends.As a result,you may be able to reduce survey length and thereby boost response and complete rates.Case in PointA quick-serve restaurant uses customer surveys to understand why customers make the dining decisions they do.The open ends allow t
59、he company to identify and monitor customer satisfaction drivers over time,as well as capture the emotional reaction to the restaurant experience and new menu ofgerings.Deeper Consumer Insights&Analytical Confidence19Product FeedbackProduct feedback is a broad category of surveys in which users are
60、asked to describe their experience,attitudes,preferences with a product that they have interacted with in some way.This can be a physical product like a granola bar or a TV remote or a car,or it can be a digital product like an application,and there are opportunities to connect with users via open e
61、nded text at almost every stage of the product lifecycle-from discovery to concept testing to post-launch.Product Feedback datasets might be used to go“beyond liking”-its great to know what your users like and dislike,but its even more valuable to uncover the nuanced emotions underneath the positive
62、s and negatives(beautiful,funny,weird,curious,nostalgic,sad).Additionally,open-ends can provide tangible and quick feedback that can help product leaders make decisions.What questions can the open end answer?There are two common types of open-ended product feedback questions.First,questions that pro
63、mpt a respondent to answer what they liked,disliked,found confusing,or remembered-can be used to identify topics or attributes.Second,questions that ask broader“why”or“how do you feel”questions help us understand the nuanced emotions consumers express when interacting with our products,and make stra
64、tegic decisions to improve our products,add value,and grow.Sample Product Research Open Ends What was your favorite part about.?What did you like least about.?What are your thoughts and feelings about.?How would you describe the beverage?What do you think is missing?Deeper Consumer Insights&Analytic
65、al Confidence20Application of AnalysisThematic AnalysisThematic analysis in product research is used to identify the attributes perceived and valued by the respondent.Feedback captured as part of sensory lab setting may be looking for the impact of certain product choices or formulations within the
66、open-ended responses.Attributes are often summarized into thematic categories,such as“taste”and“texture”in food research,often referred to as codes(the attributes)and nets(the broader themes).Emotion AnalysisEmotion analysis in product research can unlock powerful consumer feelings around the catego
67、ry and product.These can include general positive and negative sentiment,as well as more nuanced(and less binary)feelings,like“sentimental.”Correlation AnalysisBeyond looking at demographic cross-tabs,correlation analysis can be used in product research to connect attributes with emotional engagemen
68、gement.For example,respondents whos responses indicate“Love”may be more likely than others to identify a specifjc product attribute(eg.rich,Corinthian leather).Deeper Consumer Insights&Analytical Confidence21Key TakeawaysIn todays rapidly changing markets,brands are increasingly taking an agile appr
69、oach to product development and require accelerated feedback loops.Using open-ended questions efgectively can approximate the qualitative data derived from focus group and in-depth interviews,while delivering statistically signifjcant fjndings.Case in PointA“modern food company”with a large presence
70、 in the yogurt category partnered with one of their quick-serve restaurant partners on menu innovation research.Respondents were asked to describe their“dream smoothie”using an open-ended question.The open end allowed for respondents to be creative and for the food brand to generate unbiased feedbac
71、k.The result was identifjcation of two major smoothie fmavor themes:fruit-forward and functional additives(i.e.protein,etc.).They were also able wto identify emotional signatures around certain fmavor combinations.Deeper Consumer Insights&Analytical Confidence22Ad&Creative TestingAd and creative tes
72、ting is used to identify the impact and resonance of particulate ads or creative variations.Researchers generally test for aspects of purchase interest/intent,brand and message recall,and emotional resonance and engagement.Research indicates that emotional engagement drives brand loyalty,so this las
73、t point is of particular importance.What questions can the open end answer?Regardless of the stage of development,open-ended questions can provide actionable feedback that can be shared with agency or in-house content or communication teams.Is the message getting across?Are the jokes landing?Anythin
74、g rubbing people the wrong way?Researchers may also use open-ends to determine which ad to select,or to benchmark ad success.Sample Ad&Creative Test Open Ends What are your initial thoughts and feelings after watching the ad?What is the main message of the ad?What is the name of the company?What wou
75、ld you expect,based on the trailer you watched?Deeper Consumer Insights&Analytical Confidence23Application of AnalysisThematic AnalysisIn ad and creative testing,thematic analysis is used to measure message efgectiveness and recall without pre-selected options that might bias the response.Additional
76、ly,specifjc creative elements can be identifjed(“what did you like best about”).Emotion AnalysisAnalysis of open ends in creative testing is likely to start with sentiment.How did the viewer feel about the ad or show?In many cases,researchers measure overall emotional reaction(%of comments with emot
77、ion)and specifjc emotional reactions(love,hates,funny,etc.).Over time,benchmarks can be developed to measure emotional reaction against norms.Correlation AnalysisOne of the most compelling uses of correlation analysis is the ability to identify the topical drivers of emotional reaction(i.e.correlati
78、on of topics and emotions).Additionally,demographic cross-tabs are critical in a fjeld where cultural relevance is a primary success factor.Deeper Consumer Insights&Analytical Confidence24Key TakeawaysEfgective use of open ends in ad and creative testing allows researchers to measure and compare ad
79、performance against emotional reaction benchmarks,as well as to identify the topics that are associated with specifjc emotional reactions.Open ends can also help researchers assess message efgectiveness without the bias of pre-selected choices.Using open-ended questions efgectively can approximate t
80、he qualitative data derived from focus group and in-depth interviews,while delivering statistically signifjcant fjndings.Case in PointWith an understanding that emotional resonance drives brand performance,a top-fjve brand consulting fjrm asked respondents how they felt about specifjc creatives.By a
81、nalyzing the emotional reaction within responses to this open-ended question,the fjrm was able to identify creatives with the most emotional impact(while delivering on message recall objectives).Deeper Consumer Insights&Analytical Confidence25Product Reviews,Social Media&Non-Survey VerbatimsSurveys,
82、of course,arent the only source of open-ended text feedback from consumers that can be valuable for insights teams;online product reviews,product feedback forms and social media comments are among the most prevalent.They are also the most“free form”in that there is no survey context for the comments
83、,presenting both an opportunity to capture“in the wild”customer feedback and a challenge in terms of analysis.Among those challenges is the potential for sample bias towards the most vocal segments of the user base.What questions can the open end answer?Given the open nature of non-survey verbatim c
84、omments,the uses may vary.They can be used to understand overall experience,identify potential product defects or new product ideas.In the context of owned social media channels,analysis of comments on social posts can provide early reads on content efgectiveness.Deeper Consumer Insights&Analytical
85、Confidence26Application of AnalysisThematic AnalysisThematic analysis can be used to identify specifjc topics and broader themes emerging from non-survey sources of open-ended text.Given the wide range of potential comments,researchers are more likely to focus on the top themes vs.exhaustive,long-ta
86、il analysis.Emotion AnalysisNon-survey open ends can be mined as a source of unfjltered consumer sentiment.As with ad testing,overall emotional engagement can be benchmarked to identify meaningfully elevated(or depressed)emotional reactions.Correlation AnalysisWithout access to weighted demographics
87、 for cross-tab analysis,correlation analysis will typically focus within the text itself in terms of how consumers feel and whats driving those feelings.Deeper Consumer Insights&Analytical Confidence27Key TakeawaysThere are seemingly unlimited volumes of consumer open-ended text that can be mined fo
88、r valuable business insights.These sources provide an unvarnished view into the consumer mind,but may be biased towards the loudest voices.Case in PointA major consumer goods company wanted an unfjltered view of how consumers were reacting to an updated detergent product formulation.By analyzing the
89、 review comments on the products Amazon listing,they were able to quickly gain an understanding of overall reaction as well as the product attributes driving customers comments.Deeper Consumer Insights&Analytical Confidence28SummaryWe hope this eBook may help further open you up to the potential ben
90、efjts of open-ended questions and analysis,and provide inspiration and guidance on how to further utilize them in your insights and CX work.But what about the challenge of analyzing open-ended questions at scale?This is often what holds organizations back from fully realizing the insights value hidd
91、en in open-ended text.Open ends require signifjcant time and resources to decipher and categorize,and that process is prone to human bias,if not error.As a result,many insights organizations analyze only a fraction of the open ends they receive,or may just cherry-pick for verbatims that seem to supp
92、ort the qualitative themes of the research,or worse yet,may use open ends only as a data quality tool.However,new advances in artifjcial intelligence(AI)and machine learning(ML),and cloud computing have dramatically opened the aperture of use cases for advanced text analytics.As a result,researchers
93、 have new tools to bring automation to the challenge of analyzing open-ended responses,allowing qualitative analysis to be done at nearly quantitative scale.This also expands the potential use cases for open-ended questions.Ask yourself:“what if we could completely and accurately code our open ends
94、in seconds?”Would you be tempted to include a few more open-ended questions in your next survey?At the very least,wed bet you wouldnt let those open ends go to waste.Deeper Consumer Insights&Analytical Confidence29About Canvs AIOk,maybe were biased,but we love open ends Canvs AI is the easiest,faste
95、st and most accurate insights platform for analyzing unstructured text data.That data can come from virtually any source,including surveys,reviews,chats and social media comments.Our artifjcial intelligence(AI)engine for natural language processing(NLP)was trained on the“unnatural”language of social
96、 media and represents an ontology of over a trillion terms.This advanced analytical technology has been“productized”in the form of an insights platform built for market research and consumer insights professionals called Canvs MRX.Canvs is used by some of the worlds most admired brands,research agen
97、cies,and media&entertainment companies to accelerate time-to-insights,reduce cost,and deepen understanding of consumers.Wed love to show you the power of Canvs fjrst hand.Please get in touch at salescavns.ai or canvs.ai/demo.Deeper Consumer Insights&Analytical Confidence30canvs.aiCopyright 2022 Mashwork,Inc.,All rights reserved.Canvs is a product of Mashwork.Canvs,Mashwork,and mashw*rk are registered trademarks and/or service marks registered in the U.S.Patent and Trademark Office.Get Started