《广告实验是如何“欺骗”你的 —— 如何发掘真正的实验效力.pdf》由会员分享,可在线阅读,更多相关《广告实验是如何“欺骗”你的 —— 如何发掘真正的实验效力.pdf(28页珍藏版)》请在三个皮匠报告上搜索。
1、2023 DataFunSummit广告实验是如何“欺骗”你的-如何发掘真正的实验效应段玮韬领英实验带头人领英目录IntroductionThe ChallengeExisting MethodsBudget SplitContents01 IntroductionThe worlds largest professional network143M+United States14M+Canada30M+Brazil11M+Mexico6M+Columbia4M+Chile6M+Argentina90M+Europe6M+South Africa1M+Kenya1M+Nigeria2M+Egy
2、pt2M+Saudi Arabia 1M+Israel6M+Turkey1M+Morocco45M+India36M+China1M+Hong Kong1M+Republic of Korea1M+Japan9M+Australia1M+New Zealand5M+Philippines9M+Indonesia3M+Malaysia2M+SingaporeStart a PostDigest Updates from My ConnectionsLand Your Next PlayKeep in Touch with Industry Trend60KSchools900MMembers26
3、MCompanies50KSkills15MOpen Jobs190BUpdates viewedLinkedIn Marketing Solution LinkedIn Marketing Solution helps advertisers reach the desired audienceSponsored Post AdAd02 Experimental Challenge&Misleading Results Strong Experiment CultureWe experiment on UI changes,relevance algorithms,backend chang
4、es,and even bug fixes.Advanced Experiment InfrastructureWe have start-of-the-art in-house platform to meet the growing need of experimentation30000+Metrics Computed1500+Daily Active Experiments40+TBMetric and Experiment Assignment Data ProcessedData is in Our DNA Marketplace experimentationSupply av
5、ailable ads slotsDemand advertiser platformMatching bidding,pacing and optimizationWe experiment on LinkedIn FeedLinkedIn Audience NetworkMemberCampaignAdvertiserExperimentation in Online PlatformsTREATMENTCONTROL$1/user$0.995/user+$0.005/user+0.5%liftExperimentation in Online PlatformsTREATMENTCONT
6、ROL$1/user$0.995/user+$0.005/user+0.5%liftWhen experiment ramped to 100%,we often observe a much smaller lift in overall revenue.Is A/B result cheating?Cannibalization Bias in Marketplace ExperimentationTREATMENTCONTROLReal revenue impact is no more than 0%Measured revenue impact in an experiment is
7、 100%In practice,the bias can be 230%of the treatment effect1 click/user2 clicks/user+1 click/user+100%lift in revenueAssume one ad campaign with 100%budget utilizedTreatment doubles ads-click Indicator of the cannibalization bias We require each ramp to go through an“iterative experiments”with incr
8、easing allocation before launching Each experiment is a collection of Bernoulli trials,with user-specified success/treatment probabilityIf no cannibalization bias,the estimated effect in each iteration should be similarLarge Shifts in Treatment Effect Indicates Cannibalization03 Typical Experimental
9、 Designs in MarketplacesExperimental Design PrinciplesTreatment effect estimates need to be accurate as they are baked into financial forecastsUnbiasednessLow Minimum Detectable Effect(MDE)PowerExperimenters are not expected to perform data analyses to determine RobustnessSolution(1)Model-based Adju
10、stmentsThe modeling errors are orders of magnitude larger than the target MDE Solution(2)Campaign RandomizationLow power;biased;overlapping targetingCONTROLTREATMENTSolution(3)Alternating-Day Randomization Unbiased under daily budgeting;Biased under lifetime budgeting;Low power timeDay 1Day 2CONTROL
11、TREATMENT.04 LinkedIn Budget Split Experimental DesignBudget SplitCONTROL Budget&MembersTREATMENTBudget&MembersGoogle/ads campaign 1Walmart/ads campaign 2End-users/consumersUnbiased under Mild AssumptionsA few mild assumptions1)Limited interference:The results two marketplaces do not interference wi
12、th each other2)Stable systemAlthough budget and targeted audience is halved,the campaign behaves similarly to the original.Budget SplitUnder mild assumptions,Unbiased;High power;30 x experiment velocity CONTROL Budget&MembersTREATMENTBudget&MembersGoogle/ads campaign 1Walmart/ads campaign 2End-users
13、/consumersVariance of the Budget-split EstimatorEmpirical Performance-Cannibalization BiasEmpirical Performance-PowerBudget Split Experiments at LinkedInWe extended budget split experiments in other marketplaces such as the hiring marketplaceBudget split is the universally trusted way to estimate treatment effect in marketplace experimentsUnique challenges in marketplaces call for innovative experiment designs and analyses2023 DataFunSummit感谢您的观看 THANKS 段玮韬领英实验带头人领英