1、Data Analytics at Texas A&M LabDecomposition Based Explainability forDeep Neural NetworksMengnan Du Department of Computer Science&EngineeringTexas A&M UniversityEmail:dumengnantamu.eduhttps:/Data Analytics at Texas A&M Lab1Playing GoMedical DiagnosisScene UnderstandingVoice RecognitionData Analytic

2、s at Texas A&M Lab2What have been learned inside the models?Data Analytics at Texas A&M Lab3Explainability of DNNs enable us to explain the behaviorexplain the behavior of ablack-box DNN model in understandable termsunderstandable terms to humansMultilayer Perceptron(MLP)Multilayer Perceptron(MLP)Co

3、nvolutional Neural Networks(CNN)Convolutional Neural Networks(CNN)11Recurrent Neural Networks(RNN)Recurrent Neural Networks(RNN)catData Analytics at Texas A&M Lab4TraditionalTraditionalDeep LearningDeep LearningExplainableExplainableDeep LearningDeep Learning1 Fan Yang,Mengnan Du,Xia Hu.Evaluating e

4、xplanation without ground truth in interpretable machine learning.arXiv,2019.1Data Analytics at Texas A&M LabResearcher/developerEnd-users5DNNExplanationResearchersRefine ExplanationEnd-usersTrust Explanations are beneficial both to end-users and researchers For end-users:increase trust and transpar

5、ency For researchers/developers:diagnose why the model might fail and help them improve the modelData Analytics at Texas A&M Lab6Gradient based method Calculate gradient or variants of gradient using backpropogation Computational efficient“Eagle”Gradient base methods One backpropagation pass Data An

6、alytics at Texas A&M Lab7Perturbation based method Perturb the input,and feed perturbed input to model Observe the models prediction difference“Eagle”Perturbation base methods Perturb input,multiple backpropagation pass Data Analytics at Texas A&M Lab8“Understandable terms to humans”?(a)Prediction C

7、ardoon flower(b)Unreadable Explanation(c)Understandable ExplanationModel predictionExplanation heatmapsData Analytics at Texas A&M Lab91 Ribeiro,Marco Tulio,Sameer Singh,and Carlos Guestrin,“Why Should I Trust You?Explaining the Predictions of Any Classifier”.KDD,2016.Approximating the local behavio

8、r around input instances Local behavior is not linear Low faithfulness score to the original DNN modelPost-hoc Explanation“Faithfully explain the behavior of a black-box DNN”?1Data Analytics at Texas A&M LabUnderstandable interpretationFaithful interpretationShould be hierarchicalShould investigate

9、Internal neuronsCan we utilize decomposition based methods to derive interpretations?10Data Analytics at Texas A&M Lab11Recurrent Neural NetworkGraph Neural NetworkData Analytics at Texas A&M Lab12Graph Neural NetworkRecurrent Neural NetworkData Analytics at Texas A&M Lab13GRULSTMBidirectional GRUTh

10、e recurrent architecture Not feed forward structure Input processed in a recurrent way The RNN unit comes in different formatsData Analytics at Texas A&M LabKey factors-A pre-trained RNN and an input text-The prediction of RNNExplanation for RNNs-Contribution score for each feature in input-Deeper c

11、olor in the heatmap means higher contribution 14Explanation heatmap111 Hermann,Karl Moritz,et al.Teaching machines to read and comprehend.Advances in neural information processing systems,20151Data Analytics at Texas A&M Lab151111EvidenceUpdatingEvidenceForgettingInsights from the information flowin

12、g process of RNN Evidence updating from time step t-1 to time step t Evidence forgetting from tto final time step T Contribution score of xttowards prediction y could be decomposedInformation flowing process within a RNN modelData Analytics at Texas A&M Lab Symbol partial evidence is brought to the

13、time step Symbol =():the evidence that RNN obtains at time step Some follow this rule exactly,e.g.,GRU.Some approximately,e.g.,LSTM161111 Abstracted RNN updating rule:1Data Analytics at Texas A&M Lab171111Abstracted RNN updating rule:RNN prediction decomposition:Two essential elements:Hidden state v

14、ector updating vector RNN logit value:Data Analytics at Texas A&M Lab181111EvidenceUpdatingEvidenceForgettingContribution score for:Updating from 1 to Forgetting from +1 to Contribution of a word?Key ideaKey idea A RNN prediction is decomposed into additive contribution of each word in the input tex

15、tData Analytics at Texas A&M Lab191q1EvidenceUpdatingEvidenceForgettingContribution of a phrase x xA?Contribution score for:Updating from 1 to Forgetting from +1 to Key ideaKey idea-A RNN prediction is decomposed into additive contribution of each word in the input textData Analytics at Texas A&M La

16、b20Hidden state vector updating rule for GRUUpdating ruleGRU contribution score for a phrase,=,:Only need to replace with GRU updating gate vector Data Analytics at Texas A&M LabOur method accurately reflect the prediction score of different architectures21 Visualizations Under Different RNN Archite

17、cturesGRULSTMBiGRUThefightscenesarefunbutitgrows tediousThefightscenesarefunbutitgrows tediousThefightscenesarefunbutitgrows tedious GRU positive prediction(51.6%confidence)LSTM positive prediction(96.2%confidence)BiGRU negative prediction(62.7%confidence)Green:positive contribution,red:negative con

18、tribution Data Analytics at Texas A&M Lab22 Hierarchical Attribution:LSTM negative prediction with 99.46%confidence WordPhraseClauseThe story maybenewbutthe movie doesntserveuplotsoflaughs,The story maybenewbutthe movie doesntserveuplotsoflaughs,The story maybenewbutthe movie doesntserveuplotsoflaug

19、hs,Green:positive contribution,red:negative contribution In general,the first part of the text has negative contribution The second part of the text has positive contribution This hierarchical attention represents the contributions at different levels of granularityData Analytics at Texas A&M Lab23E

20、valuation MetricFaithfulness-The higher accuracy drop is observed with the deletionof a generated interpretation phrase,the more faithful the explanationalgorithm isProposed method has higher fidelity to the RNN classification modelsModelsModelsGRUGRULSTMLSTMBiGRUBiGRUGradient0.2720.2430.068Integrat

21、ed Gradient0.2550.2530.113Lime0.2090.1880.092ProposedProposed methodmethod0.3110.3110.3180.3180.1960.196Data Analytics at Texas A&M Lab24“Schweiger is talented and terriblyterribly charismatic,qualities essential to both movie stars and social anarchists”.99.97%negative sentiment prediction by LSTM

22、99.97%negative sentiment prediction by LSTM.Polysemous word“terribly”LSTM captures the meaning relevant to“terrible”,while ignore other meanings,such as“extremely”This LSTM may fail to model polysemous wordsData Analytics at Texas A&M Lab25“Schweiger is talented and terriblyterribly charismatic,qual

23、ities essential to both movie stars and social anarchists”.11Negative sentiment,99.97%confidenceData Analytics at Texas A&M Lab26“Schweiger is talented and terriblyterribly charismatic,qualities essential to both movie stars and social anarchists”.extremelyextremely11Data Analytics at Texas A&M Lab2

24、7Positive sentiment,81.29%confidence“Schweiger is talented and extremelyextremely charismatic,qualities essential to both movie stars and social anarchists”.11Data Analytics at Texas A&M Lab2811“Schweiger is talented and terriblyterribly charismatic,qualities essential to both movie stars and social

25、 anarchists”.veryveryData Analytics at Texas A&M Lab29Positive sentiment,99.53%confidence11“Schweiger is talented and veryvery charismatic,qualities essential to both movie stars and social anarchists”.Data Analytics at Texas A&M Lab30“Occasionally melodramatic,it s also extremelyextremely effective

26、.”“Occasionally melodramatic,it s also terriblyterribly effective.”Positive,99.53%Positive,99.53%Negative,99.0%Negative,99.0%“ExtremelyExtremely well acted by the four primary actors,this is a seriously intended movie that is not easily forgotten.”“TerriblyTerribly well acted by the four primary act

27、ors,this is a seriously intended movie that is not easily forgotten.”Positive,99.98%Positive,99.98%Negative,87.7%Negative,87.7%Data Analytics at Texas A&M Lab31Recurrent Neural NetworkExplainability ApplicationsGraph Neural NetworkData Analytics at Texas A&M Lab32Explanation is crucial for GNNMuch o

28、f the data is stored in the form of graphWide application in social networking,advertising recommendation,drug generationData Analytics at Texas A&M Lab33Data Analytics at Texas A&M Lab341.Approximation based methods do not guarantee the fidelity2.Perturbation based approaches may trigger the advers

29、arial natureData Analytics at Texas A&M Lab35Data Analytics at Texas A&M Lab36A and D are two target nodesB,C,E,F are background nodesWe want to study contributions of A and D=,Information about A and DInformation about other 4 nodesData Analytics at Texas A&M Lab37A and D are two target nodesB,C,E,

30、F are background nodesWe want to study contributions of A and DInformation about A and DInformation about other 4 nodesData Analytics at Texas A&M Lab38Suppose last layer is decomposable,then we decompose next layerData Analytics at Texas A&M Lab39A and D are two target nodesB,C,E,F are background n

31、odesWe want to study contributions of A and DSuppose A and D are the important subgraphOtherwise,both parts have contributions=.f(x)Data Analytics at Texas A&M Lab40Graph Convolution Layer:Graph Convolution Layer:Our Decomposition(graph convolution is a linear operation):Our Decomposition(graph conv

32、olution is a linear operation):Data Analytics at Texas A&M Lab41Fully Connected Layer:Fully Connected Layer:Our Decomposition:Our Decomposition:Data Analytics at Texas A&M Lab42ReLUReLU Activation:Activation:Step 1:update the target term firstStep 1:update the target term firstStep 2:update the back

33、ground term by subtracting this from total activationStep 2:update the background term by subtracting this from total activationData Analytics at Texas A&M Lab43A and D are two target nodesB,C,E,F are background nodesWe want to study contributions of A and D1.The total numbers of subgraphs are too l

34、arge2.How can we select the most important sub-graph?Data Analytics at Texas A&M Lab441.Starting from M subgraphs:1.Starting from M subgraphs:2.Calculating importance of neighbors:2.Calculating importance of neighbors:3.Calculating relevant importance of a neighbor to other neighbors:3.Calculating r

35、elevant importance of a neighbor to other neighbors:4.Selected nodes are merged into subgraph:4.Selected nodes are merged into subgraph:Data Analytics at Texas A&M Lab45The first row shows an incorrect prediction,second row shows the correct one.Red is negative,blue is positiveData Analytics at Texa

36、s A&M Lab46Recurrent Neural NetworkGraph Neural NetworkData Analytics at Texas A&M Lab47Explainable Deep LearningAdversarial VulnerabilityDomain ShiftShortcut LearningDiscriminationAnd BiasDeepfake DefensePoisoning AttackNetwork CompressionData Analytics at Texas A&M Lab48Closely relevant to XAIClosely relevant to XAIExplore other areas of Responsible AIExplore other areas of Responsible AIData Analytics at Texas A&M LabDecomposition Based Explainability forDeep Neural NetworksMengnan Du Department of Computer Science&EngineeringTexas A&M UniversityEmail:dumengnantamu.eduhttps:/