1、How big data can help small data?Fei ShaDepartment of Computer ScienceUniversity of Southern CaliforniaLos Angeles,CAWhere does this“small”data come from?Examples from yesterdays talksL3 Edge case study“常常会遇到偶发的大雾，大雪，”Quan Finance“左尾”“右尾”没有足够多的数据Education planning is complex decision-makingTime scal

2、es Short-term Which school?Extracurricular activities:sports,arts,etc?Mid-term Which university?Which speciality?Long-term What kind of career path?What else makes this hard?Individual differences Genetically Environmentally Harsh constraints One-shot game Costly to recover from mistakes Our goal:pe

3、rsonalized learning and education Bill&Melinda Gates Foundation on Personalized LearningCNN on Personalized Learning,2016Can machine learning/AI achieve that?Sadly,not 100%yet Individualized models need individual-specific data The amount of data is fundamentally limited,hence being Small.Most moder

4、n learning algorithms require Big Data about the individual.Can you build a model of me?Can machine learning/AI achieve that?Sadly,not 100%yet Individualized models need individual-specific data The amount of data is fundamentally limited,hence being Small.Most modern learning algorithms require Big

5、 Data about the individual.Can you build a model of me?I will need more data from you.Can machine learning/AI achieve that?Sadly,not 100%yet Individualized models need individual-specific data The amount of data is fundamentally limited,hence being Small.Most modern learning algorithms require Big D

6、ata about the individual.Can you build a model of me?I will need more data from you.Or I can give you the model of those3 vignettes of how Big Data can help Small Data3 Learning settings Multi-task learning Domain adaptation Zero-shot learning Primary application focus Computer vision3 vignettes of

7、how Big Data can help Small Data3 Learning settings Multi-task learning Domain adaptation Zero-shot learning Primary application focus Computer vision3 vignettes of how Big Data can help Small Data3 Learning settings Multi-task learning Domain adaptation Zero-shot learning Primary application focus

8、Computer vision3 vignettes of how Big Data can help Small Data3 Learning settings Multi-task learning Domain adaptation Zero-shot learning Primary application focus Computer vision3 vignettes of how Big Data can help Small Data3 Learning settings Multi-task learning Domain adaptation Zero-shot learn

9、ing Primary application focus Computer visionVignette 1 Multi-task Learning(MTL)“众人拾柴高”Vignette 1Problem setting M tasks,each with its own data Need to find solutions for all of them Traditional framework for supervised learning Solve each task independently Task 1 D1Task 3 D3Task 2 D2Task 4 D4w1w2w

10、3w4argminwm(Dm;wm)+?mR(wm)Multi-task learning(MTL)Main idea Learn jointly multiple related tasks Force knowledge sharing Combine small data into big data Benefits Improve generalization performance Require less amount of data Works in both deep and shallow learning modelsTask 1 D1Task 3 D3Task 2 D2T

11、ask 4 D4w1w2w3w4N.B.Caruana,97.Bakker and Heskes,03.Evgeniou,et al.04.Ando and Zhang.05.Yu,et al.,05.Lee,et al.,07.Argyriou,et al.08,Daum,09.argminw1,w2,wMMXm=1(Dm;wm)+?R(w1,w2,wM)What MTL is really aboutExploiting task relatedness Encode prior knowledge by selecting the regularizer Constrain the hy

12、pothesis space for all tasks Choices of regularizer All parameters are similar to each other Parameters should have similar sparsity patterns.Task 1 D1Task 3 D3Task 2 D2Task 4 D4w1w2w3w4argminw1,w2,wMMXm=1(Dm;wm)+?R(w1,w2,wM)Examples of incorporating external knowledgepolar beardalmatianwhitespotsym

13、1ya1yaAymMpolar beardalmatianwhitespotsObject classAttributes lifiym1ya1yaAymMclass classifierclassifier?11?12?D(M+A)u1u2uDSharedfeaturesu3Input visual featurex1x2x3xDobject categories and attributes,CVPR,2011leopard:cat=wolf:dog leopard:tiger=horse:zebra Visual feature space Analogies Semantic Embe

14、dding Space?Regularization Analogy-preserving embedding,ICML,2013N.B.Sharing ontologies(NIPS 2011,2012)Challenge:discover knowledge automaticallyNot all tasks are beneficial Challenge:discover knowledge automaticallyNot all tasks are beneficial Challenge:discover knowledge automaticallyNot all tasks

15、 are beneficial“碰瓷”宝？Challenge:discover knowledge automaticallyNot all tasks are beneficial“碰瓷”宝？不make sense啊？！Challenge:discover knowledge automaticallyNot all tasks are beneficial How to discover groups of related subtasks?“Learning with whom to share”(ICML,2011)“Resisting the temptation to share”

16、(CVPR,2014)Why this is useful?Learning in noisy task data Learning from a set of irrelevant tasks Ex:comp bio,noisy labels Task 1 D1Task 3 D3Task 2 D2Task 4 D4w1w2w3w4Group 1Group 2Vignette 2 Domain adaptation“以不变应万变“MotivationClassification task:given a face image,determine man or woman?Standard re

17、cipeCollect a lot of labeled images training datamanwomantrainingInfer a classification boundary feature spacex1x2measure test performanceClassify on test image x1x2measure test performanceClassify on test image Success!x1x2Why this works?Shared statistical properties,useful for classification Why t

18、his works?Shared statistical properties,useful for classification training datatest dataunseen datatell-tale feature:length of hairWhen it fails?Mismatch between training and testing training datatest dataunseen dataWhen it fails?Mismatch between training and testing training datatest dataunseen dat

19、a“length of hair”no longer effective!Limitation of learning algorithms Unrealistic,oversimplifying assumptions Learning environment is stationary Training,testing and future data are sampled in i.i.d from the same distribution Works well in academic/well-controlled settings.Limitation of learning al

20、gorithms In real-life,Learning environment changes.Training,testing and future data are sampled from different distributions.We suffer from poor cross-distribution generalization,where accuracy for disparate domains drops significantly.A common theme across many fieldsComputer vision Object recognit

21、ion:train&test on different datasets Vehicle pedestrian avoidance systems:train&test in different vehicular/city environments Natural language processing Syntactic parsing:train on business articles but applied to medical journals Speech recognition:train on native speakers but applied to accented v

22、oicesObject recognition for computer visionChallenges Many exogenous factors affect visual appearances:pose,illumination,cameras quality,etc.Collecting data under all possible combinations of those factors is expensive.Labeling those data is even more costly.Caltech-256 Amazon DSLR WebcamExample ima

23、ges from 4 domains in our empirical studiesEffect of using bigger datasets for adaptation Just being Big is not enoughAccuracy5060708090100AmazonWebcamImageNetAdapted AmazonAdapted ImageNetAnonymous source,2014targetSmall sourcelarger sourceDomain-invariant features Theoretical motivation Exploit in

24、trinsic structures Learn kernels How to adapt?discriminative clustering(ICML 2012)landmarks(ICML 13,NIPS 13)linear subspaces(CVPR 2012)G(d,D)Source domainTarget domainGrassmannmanifold ofsubspacesGeodesic flow captures domain-invariant representation(for visual recognition)Shared representation Exis

25、tence of a(latent)feature space The marginals of source and targets are the same(or similar)in this space Exist a single classifier works well on both domains Domain-invariant featuresBen-David et al 06,Blitzer et al06,Daume III07,Pan et al,09,Sugiyama et al,12Th Sh+A(PS,PT)+infh2HTh+ShMeasure how m

26、arginal distributions are similarhow well a single classifier can doGeodesic flow kernel(GFK)Domain-invariant features Parameterized as linear kernel mapping of original features Constructed to minimize discrepancy between two domains Model domains with subspaces Compute discrepancy as differences b

27、etween subspacesG(d,D)Source domainTarget domainGrassmannmanifold ofsubspacesGeodesic flow captures domain-invariant representation(for visual recognition)Gong,Shi,Sha and Grauman,CVPR 2012Adaptation improve accuracy!012.52537.550C-AA-WW-CD-AC-DA-CNo adaptationSGF(Gopalan et al,ICCV 2011)Geodesic Fl

28、ow kernel(ours)Comparing diff.structures015304560C-AA-WW-CD-AC-DA-CGeodesic flow kernel(GFK)ClusteringLandmarkAccuracy(%)Vignette 3 Zero-shot learning“无中生有”Motivation:visual object recognition in the wildClassical machine learning framework Multiway classification Labeling space is determined a prio

29、r A large number of annotated training samples for every class Challenges for recognition in the wild Labeling space grows arbitrarily large with emergence of new classes Collecting data for new classes is not always cost-effective Some classes do not have enough labeled or zero labeled imagesVisual

30、 recognition(Caltech-101)“dog”“cat”“flower”“bear”“bench”“bird”Visual recognition(ImageNet)Visual recognition(even more categories?)producesInteresting statisticsNumber of species(total:1,589,361)Birds:9956 Fish:30,000 Mammals:5,416 Reptiles:8,240 Insects:950,000 Corals:2,175 Plants:297,326 Mushrooms

31、:16,000 10,000 new animal species are discovered every year(source:wikipedia)“Skywalker”gibbonCommon issue:long-tail phenomenaObjects in SUNIn case you are curiouszigguratSimilarly,in ImageNetZero-shot learningTwo types of classes Seen:with a lot of labeled examples Unseen:without any examples SeenU

32、nseenCatHorseDogZebra?FiguresfromDerekHoiems slidescf.Lampert et al,2009Is it even possible?What is it?BearWhat is it?ZebraWhat is it?bambooQuizWhat is it:bear-like,with black and white stripe and often with bamboo?Ask googleYou will get thisYou will get thisGeneral strategyClass labels discrete num

33、bers Need to assign semantic meanings to class labels Need to define relationships among class labels Key assumptions There is a common semantic space shared by both types of classes Configuration of the embeddings enable“transfer”.seen classuneen classSemanticEmbeddings Attributes(Farhadi etal.09,L

34、ampert etal.09,Parikh&Grauman 11,)Wordvectors(Mikolov etal.13,Socher etal.13,Frome etal.13,)SemanticEmbeddings Attributes(Farhadi etal.09,Lampert etal.09,Parikh&Grauman 11,)Wordvectors(Mikolov etal.13,Socher etal.13,Frome etal.13,)Zero-shotLearningSeenObjectsUnseenObjectFiguresfromDerekHoiems slides

35、HasStripesHasEarsHasEyesHasFourLegsHasManeHasTailBrownMuscularHasSnoutHasStripes(likecat)HasMane(likehorse)HasSnout(likedog)Howtoeffectivelyconstructamodelforzebra?Problem statementTraining Seen classes and their semantic embeddings Annotated training samples Goal Unseen classes and their semantic e

36、mbeddings Classifier:S=1,2,SD=(xn,yn)Nn=1U=S+1,S+UAS=a1,a2,aSAU=aS+1,aS+2,aS+Uf:x!y 2 USynthesized classifiers for zero-shot learningSemantic space Model space penguin cat dog a1 a2 a3 b1 b2 w1 w2 w3 v1 v2 v3 b3:class exemplar Semantic representations a House Wren a Cardinal a Cedar Waxwing v Cardin

37、al Visual features (ac)?vc ()?PCA Semantic embedding space (au)for NN classification or to improve existing ZSL approaches a Gadwall a Mallard CVPR2016,ECCV 2016,ICCV 2017Main ideaDictionary learning Introduce phantom classes as bases Learn bases semantic embeddings as well as models for bases Graph

38、s structures encode“relatedness”Define how classes are related in the semantic embedding space Define how classes are related in the model spaceSemantic space Model space penguin cat dog a1 a2 a3 b1 b2 w1 w2 w3 v1 v2 v3 b3 0BBBBBBB0.10.2?1.0.?0.31CCCCCCCA0BBBBBBB1.1?.21.0.0.41CCCCCCCA0BBBBBBB0.500.5

39、.?0.41CCCCCCCAEvaluation:setupDatasets Dataset#of seen#of unseenTotal#nameclassesclassesof imagesAwA401030,475CUB1505011,788SUN645/64672/7114,340ImageNet1,00020,84214,197,122Main results:compare to other methodsClassification accuracy MethodsAwACUBSUNImageNetDAP 2241.4-22.2-IAP 2242.2-18.0-BN 4343.4

40、-ALE 137.418.0-SJE 266.750.1-ESZSL 3649.3-ConSE30-1.4SSE-ReLU 47?76.330.4-46?80.542.1-Ourso-vs-o69.753.462.81.4Ourscs68.451.652.9-Oursstruct72.954.762.71.5Main results:compare to other methodsClassification accuracy MethodsAwACUBSUNImageNetDAP 2241.4-22.2-IAP 2242.2-18.0-BN 4343.4-ALE 137.418.0-SJE

41、266.750.1-ESZSL 3649.3-ConSE30-1.4SSE-ReLU 47?76.330.4-46?80.542.1-Ourso-vs-o69.753.462.81.4Ourscs68.451.652.9-Oursstruct72.954.762.71.5“I challenge you on ImageNet.you are actually beating us!”Main results:compare to other methodsClassification accuracy MethodsAwACUBSUNImageNetDAP 2241.4-22.2-IAP 2

42、242.2-18.0-BN 4343.4-ALE 137.418.0-SJE 266.750.1-ESZSL 3649.3-ConSE30-1.4SSE-ReLU 47?76.330.4-46?80.542.1-Ourso-vs-o69.753.462.81.4Ourscs68.451.652.9-Oursstruct72.954.762.71.5“I challenge you on ImageNet.you are actually beating us!”1.8Summary:small dataFrequently occurring in many application scena

43、rios Rich contextual variables will always dwarf data.We will never have enough data for rare probability events.For full autonomy,we always have to extrapolate instead of interpolating.Challenges Machine learning does not have a whole lot of approaches for it So far,we are just using big data to bias the inductive bias for the small data Opportunities More personalized and precise actionable knowledge for decision-making Pivotal to mechanism and causality discovery AcknowledgementBoqing GongZhuoliang KangKristen GraumanYuan ShiDinesh JayaramanWeilun ChaoSung Ju HwangSoravit Changpinyo