1、GraphTranslator:Aligning Graph Model to Large Language Model for Open-ended TasksCheng YBeijing University of Posts and TelecommunicationsOutline Background:Graph Founda1on Model Our Recent A9empts Future Direc1onsOutline Background:Graph Founda1on Model Our Recent A9empts Future Direc1onsFounda,on

2、Models“A foundation model is any model that is trained on broad data and can be adapted to a wide range of downstream tasks.”11 R.Bommasani,D.A.Hudson,E.Adeli,R.Altman,S.Arora,S.von Arx,M.S.Bernstein,J.Bohg,A.Bosselut,E.Brun-skill,et al.,“On the opportunities and risks of foundation models,”arXiv pr

3、eprint arXiv:2108.07258,2021LanguageUSMGPT4Foundation models have become a reality in language,vision,and speech.VisionSpeechCharacteris,cs of Founda,on Models Emergence suggests that as a founda1on model scales up,it may spontaneously manifest novel capabili1es.2 Homogeniza1on alludes to the models

4、 versa1lity,enabling its deployment across diverse applica1ons.Machine Transla-onQ&AText Genera-onInforma-on Extrac-onHomogeniza-onFounda-on ModelEmergence2 Wei J,Tay Y,Bommasani R,et al.Emergent abilities of large language modelsJ.arXiv preprint arXiv:2206.07682,2022.Large Language Models(LLMs)With

5、 billions of parameters,LLMs have shown abilities towards artificial general intelligence(AGI),e.g.,understanding,reasoning,planning,etc.3 Zhao W X,Zhou K,Li J,et al.A survey of large language modelsJ.arXiv preprint arXiv:2303.18223,2023.On the other handGraph(network)is a common language for descri

6、bing relational data.Citation NetworkSocial NetworkMolecule GraphUser-item GraphInternetDrug Interaction GraphA History of Graph Theory&Learning17361950s1990s2000sGraph TheoryEulers seven bridgesGraph AlgorithmDijkstras shortest pathGraph ModelsRandom graph,Stochas-c block model,Scale-free network20

7、10s2020sGraph EmbeddingLaplacian Eigenmap,DeepWalkGraph Neural NetworkGCN,GATA History of Graph Theory&Learning17361950s1990s2000sGraph TheoryEulers seven bridgesGraph AlgorithmDijkstras shortest pathGraph ModelsRandom graph,Stochas-c block model,Scale-free network2010s2020sGraph EmbeddingLaplacian

8、Eigenmap,DeepWalkGraph Neural NetworkGCN,GATGraph(Machine)LearningWhatll be the Next Paradigm of Graph Learning?Jiawei Liu,Cheng Yang,Zhiyuan Lu,Junze Chen,Yibo Li,Mengmei Zhang,Ting Bai,Yuan Fang,Lichao Sun,Philip S.Yu,Chuan Shi.Towards Graph Foundation Models:A Survey and Beyond.arXiv 2023A survey

9、 discussing the potential future of graph learningGraph Founda,on Model(GFM)A GFM is envisioned as a modelpre-trainedonextensivegraphdata,primed for adaptation acrossdiverse downstream graph tasks.Jiawei Liu,Cheng Yang,Zhiyuan Lu,Junze Chen,Yibo Li,Mengmei Zhang,Ting Bai,Yuan Fang,Lichao Sun,Philip

10、S.Yu,Chuan Shi.Towards Graph Foundation Models:A Survey and Beyond.arXiv 2023Two expected characteristics：Emergence refers to novel capabili2es shownexclusively in large-scale graph models.Homogeniza2ondenotestheadaptabilityacross different types of graph tasks.Rela,onship with Language Founda,on Mo

11、delSimilarities&DifferencesExis,ng Work towards GFMsNo clear solu1on of how to build a GFM yet LBut there are some explora1ons towards it JGNN-based ModelsIdea:Improve exis1ng graph learning through innova1on in GNN BackbonePre-trainingAdapta2onThese works typicallydraw inspira2on from the model arc

12、hitectures or training paradigms used in NLP do not explicitly model text data in their pipeline GNN-based ModelsBackbone ArchitectureMessage PassingGraph TransformerGNN-based ModelsPre-trainingGenera2ve methods:graph reconstruc2on,property predic2onContras2ve methods:same-scale contras2ve learning,

13、cross-scale contras2ve learningLiu Y,Jin M,Pan S,et al.Graph self-supervised learning:A surveyJ.IEEE Transactions on Knowledge and Data Engineering,2022,35(6):5879-5900.graph reconstructionproperty predictionGNN-based ModelsPre-trainingGenera2ve methods:graph reconstruc2on,property predic2onContras2

14、ve methods:same-scale contras2ve learning,cross-scale contras2ve learningLiu Y,Jin M,Pan S,et al.Graph self-supervised learning:A surveyJ.IEEE Transactions on Knowledge and Data Engineering,2022,35(6):5879-5900.same-scale contrastive learningcross-scale contrastive learningGNN-based ModelsAdapta1onF

15、ine-tuning:keep input graph intact,modify model parameters accordinglyPrompt-tuning:keep pre-trained model intact,modify input graph insteadLLM-based ModelsIdea:Exploring the feasibility of using LLMs as GFMs by serializing graphsGraph-to-token:describe graph structure with token sequenceGraph-to-te

16、xt:describe graph informa2on with natural language LLM-based ModelsBrief Summary of Exis1ng WorkBackbone:BERT,T5,LLaMa,GPTsPre-training:Language Model(LM),Masked Language Model(MLM)Adapta2on:Manual Prompt Tuning,Automa2c Prompt TuningGNN+LLM-based ModelsIdea:Harness the strengths of both language un

17、derstanding from LLMs and structural analysis from GNNs GNN-centric:u2lize LLMs to extract features from raw data and predict with GNNs Symmetric:align the embeddings of GNNs and LLMs to make bePer predic2onsLLM-centric:u2lize GNNs as tools to enhance the performance of LLM GNN+LLM-based ModelsBrief

18、 Summary of Exis1ng WorkBackbone:GNN-centric,Symmetric,LLM-centricPre-training:LM,MLM,Graph-Text Contras2ve Learning(GTCL)Adapta2on:(Parameter-Efficient)Fine-tuning,Tuning-free Promp2ng,Prompt TuningOutline Background:Graph Founda1on Model Our Recent A9empts Future Direc1onsOur Recent AOemptsMA-GCL:

19、Model Augmenta1on Tricks for Graph Contras1ve Learning(MA-GCL,AAAI 2023)A Data-centric Framework to Endow Graph Neural Networks withOut-Of-Distribu1on Detec1on Ability (AAGOD,KDD 2023)GraphTranslator:Aligning Graph Model to Large Language Model for Open-ended Tasks(GraphTranslator,WWW 2024)Xumeng Go

20、ng,Cheng Yang,Chuan Shi.MA-GCL:Model Augmenta-on Tricks for Graph Contras-ve Learning.AAAI 2023 Yuxin Guo,Cheng Yang,Yuluo Chen,Jixi Liu,Chuan Shi,Junping Du.A Data-centric Framework to Endow Graph NeuralNetworks with Out-Of-Distribu-on Detec-on Ability.KDD 2023Mengmei Zhang,Mingwei Sun,Peng Wang,Sh

21、en Fan,Yanhu Mo,Xiaoxiao Xu,Hong Liu,Cheng Yang,Chuan Shi.GraphTranslator:Aligning Graph Model to Large Language Model for Open-ended Tasks.WWW 2024Mo,va,on of MA-GCLMo1va1on Contras2ve learning captures invariant informa2on among different augmenta2on views.Good augmenta2ons should introduce as muc

22、h perturba2on as possible without changing the core seman2cs.However,in graph contras2ve learning(GCL),we have few prior knowledge on how to generate such good augmenta2ons.Can we generate be+er augmenta.ons than typical random dropping-based methods?MAGCLCore idea We interpret a GNN as a sequence o

23、f propaga2on operator g and transforma2on operator h:propaga2on operator g is typically the non-parametric graph filter.transforma2on operator h is typically a weight matrix with a non-linear func2on.Intui2on:different architectures(i.e.,operator sequences)wont affect the core seman2cs.Thus we pertu

24、rb the neural architecture of graph encoder as model augmenta2ons.MAGCLWe propose three strategies to introduce perturba1ons:Asymmetric strategy Use the same number of operator h with shared parameters for different views Use different numbers of operator g for different views Random strategy Random

25、ly vary the number of propaga2on operator g in every training epoch Shuffling strategy Randomly shuffle the permuta2on of propaga2on and transforma2on operatorsExperimentsWe conducted extensive experiments on node/graph classifica1on/clustering.ExperimentsAll three strategies contribute to the final

26、 performance.Mo,va,on of AAGODMo1va1on A reliable GNN should not only perform well on know samples(ID)but also identify graphs it has not been exposed to before(OOD).Existing works proposes to train a neural network specialized for the OOD detection task.Can we build a graph prompt that can solve OO

27、D detec.on given a well-trained GNN?(1)Traditional works(2)Our proposed frameworkAAGODRLS encourages high scores for amplified ID graphs and expects low scores when only seeing the amplifiers.LAG adap-vely generates graph-specific amplifiers by conver-ng node representa-ons into edge weights.Transla

28、torWe modify edge weights as prompts to highlightthe latent pacern of IDgraphs,and thus enlargethe score gap betweenOOD and ID graphs.OODIDScoreDensityDensityScoreExperimentsWe conducted experiments on five dataset pairs over four GNNs to verify performance.ExperimentsCase study:We visualize the lea

29、rned graph prompts(i.e.,amplifiers)for interpretability analysis.Mo,va,on of GraphTranslatorMo1va1on LLMs showcase impressive emergent abilities for open-ended tasks based on instructions,but struggle with processing complex graph data.Graph models(GMs)are often designed for encoding graph data into

30、 embeddings,while LLMs fail to directly process these embeddings.Can we build a model that can bridge the gap between GM and LLM for open-ended tasks?+Describe the shared interests of user and users friends.“”This user priori6zes appearance and quality of life.Their friends share interests in.Open-e

31、nded Task defined by instruc4onsResponseLLMGMLLMOpen-ended TasksTranslatorResponse35GraphTranslator FrameworkFrozen Graph Model Learn node representations for text-attributed graphs to handle predefined tasks.Frozen Large Language ModelServe as a unified interface for pre-training and open-ended tas

32、ks,responding to human instructions.Producer Module Construct alignment data for supervisory model training,i.e.,(node representation,descriptive text)pairs.Translator Module Convert node representations into tokens,enabling LLM comprehension.GraphTranslator36Producer ModuleTo capture node informati

33、on within representations,textual descriptions for each node representation encompass node feature,neighbor information,and their commonalitiesUsing“Chain of Thought”(CoT)toguide LLM to progressively generatehigh-quality descriptionGraphTranslator37Translator(Training&Inference)projectionTraining St

34、age 1:Align GM-TextTraining Stage 2:Align GM-LLMInference TranslatorGraphTranslatorGraphTranslatorp Stage 1:We obtain text embeddings with translator,then we train the translator through contrasJve learningp Stage 2:We use a linear layer to project the output of Translator module into the same dimen

35、sion with the word embedding of LLM39Training(Stage 1)!=!,#$%&!=!,#$%CLSNode RepresentationCLSEncoderDecoderDECGenerate!Contras2ve Objec2veNode representaJon Text representaJonHigh-level alignmentMatching Objec2veNode representaJon Text representaJonFine-grained alignmentGenera2on Objec2veNode repre

36、sentaJon TextReplace the CLS token with a new DEC token as the first text token to signal the decoding taskText RepresentationGraphTranslator40projectionUse a linear layer to project to token representation space of LLMConnect the projected representation with the human instruction and feed into LLM

37、Fine-tune Translator by aligning the response text of LLM with the actual descriptive textTraining(Stage 2)GraphTranslatorGraphTranslatorWe employ LLM to construct high-quality descrip-on text with Chain-of-Thought(COT).ProducerTranslator aims to align GM and LLM by conver-ng the learned node embedd

38、ing into token representa-ons.TranslatorWe propose a novel framework to align graph models(GMs)to LLM,named GraphTranslator.ExperimentsWe conducted experiments on the Taobao and ArXiv datasets in zero-shot scenario.ExperimentsWe conducted QA experiment in Taobao dataset.GraphTranslator captures the

39、preferences of users and their friends more accurate.Outline Background:Graph Founda1on Model Our Recent A9empts Future Direc1onsFuture Direc,ons1.Data Quan1ty and QualityStructure/Feature/Label Augmenta2onSerializa2on for LLM-based Methods2.Backbone and Learning ParadigmBeyond the Transformer?More

40、Advanced Pretext Tasks 3.Applica1on/Evalua1onDrug discovery,Urban Compu2ngHuman/AI FeedbackSafety/Privacy IssuesOpen-source Graph Learning PlaRormsYaoqi Liu,Cheng Yang,Tianyu Zhao,Hui Han,Siyuan Zhang,Jing Wu,Guangyu Zhou,Hai Huang,Hui Wang,Chuan Shi.GammaGL:A Mul-Backend Library for Graph Neural Networks.SIGIR 2023Han H,Zhao T,Yang C,et al.OpenHGNN:An Open Source Toolkit for Heterogeneous Graph Neural Network.CIKM 2022GammaGL:A GNN library supporting multiple deep learning backendsOpenHGNN:The first heterogeneous graph neural network libraryThanksQ&A