1、DataFunSummit#2023mPLUG-多模态对话大模型技术与应用解析徐海洋阿里巴巴-达摩院-算法专家01多模态大模型技术发展历程02多模态对话大模型mPLUG技术与应用解析03ModelScope实战分享04mPLUG项目主页目录 CONTENTDataFunSummit#202301多模态大模型技术发展历程多模态预训练背景-下游任务多模态预训练-发展历程多模态预训练发展历程：18,19年基于检测特征的两阶段方法；20，21年端到端方法，22年-23年大一统+Scaling up的方法；最近几个月，多模态对话大模型；VQA Leaderboard:多模态最重要的榜单，现在已达到86.

2、06，mPLUGCVPR2021 VQA Challenge排名第一，并以81.26分的成绩首次超越人类结果；Vision-Language Pre-training:Basics,Recent Advances,and Future Trends.多模态预训练-发展历程多模态对话大模型-GPT4视觉内容细粒度理解与推理多模态对话大模型-GPT4视觉内容富文本图片表格理解与推理多模态对话大模型-GPT4视觉内容富文本图片表格理解与推理多模态对话大模型-System CollaborationVisual ChatGPTMM-REACTHuggingGPT通过各种视觉的模型将视觉信息转换为文本信

3、息，之后通过ChatGPT进行信息的整理与回复多模态对话大模型-End2EndMiniGPT-4LLAVAKosmos-1类GPT4，通过一个模型同时拥有多模态与文本的能力DataFunSummit#202302多模态对话大模型mPLUG技术与应用解析模块化多模态模型mPLUG，大一统模型mPLUG-2mPLUG:Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections.EMNLP2022.mPLUG-2:A Modularized Multi-modal Foundation Model

4、Across Text,Image and Video.ICML 2023.mPLUG系列多模态预训练工作，借鉴人脑模块化思想，针对不同模态input，不同模态output，不同模态特有属性，不同功能（Understanding，Generation）设计不同的模块，层次模块化预训练，这样可以轻量化，可拆拔的灵活应用到各种Zero/Few-Shot/Continue Pretrain/下游Finetuning/多模态表征等层次化应用场景。模块化多模态对话大模型mPLUG-Owl-ArenamPLUG-Owl:Modularization Empowers Large Language Mode

5、ls with MultimodalityArena:上海人工智能实验室OpenGVLab组织的人工标注评测多模态LLM版单，mPLUG-Owl排名第一！模块化多模态对话大模型mPLUG-Owl-应用场景mPLUG-Owl:Modularization Empowers Large Language Models with Multimodality模块化多模态对话大模型mPLUG-Owl-应用场景mPLUG-Owl:Modularization Empowers Large Language Models with Multimodality模块化多模态对话大模型mPLUG-OwlmPLUG

6、-Owl:Modularization Empowers Large Language Models with Multimodality模块化多模态对话大模型mPLUG-Owl-训练方法mPLUG-Owl:Modularization Empowers Large Language Models with Multimodality 第一阶段：对齐视觉和语义，训练视觉基础模块和视觉摘要模块 第二阶段：通过单模态和多模态相互协同，通过LoRA指令微调模型，固定视觉基础模块、视觉摘要模块和原始LLM的参数，只在LLM引入少量参数的adapter结构用于指令微调模块化多模态对话大模型mPLUG-O

7、wl-性能评测mPLUG-Owl:Modularization Empowers Large Language Models with MultimodalitymPLUG-Owl构建了一个多模态指令评测集评测集OwlEvalOwlEval，以下是不同模型在OwlEval上的占比表现（A最好，D最差），评分指标：ABCD打分制A：听懂指令，且回答满意B：听懂指令，但是回答部分会存在一些错误C：听懂指令，但是回答错误或者用户不满意D：听不懂指令或者无效的回答模块化多模态对话大模型mPLUG-Owl-知识问答mPLUG-Owl:Modularization Empowers Large Langu

8、age Models with Multimodality模块化多模态对话大模型mPLUG-Owl-多轮对话mPLUG-Owl:Modularization Empowers Large Language Models with Multimodality模块化多模态对话大模型mPLUG-Owl-笑话理解mPLUG-Owl:Modularization Empowers Large Language Models with Multimodality模块化多模态对话大模型mPLUG-Owl-涌现能力mPLUG-Owl:Modularization Empowers Large Language

9、 Models with Multimodality在训练中，mPLUG-Owl并未见过多图和OCR的数据，以下为多图理解的能力：模块化多模态对话大模型mPLUG-Owl-涌现能力mPLUG-Owl:Modularization Empowers Large Language Models with Multimodality简单的DocumentAI能力（OCR）模块化多模态对话大模型mPLUG-Owl-视频理解mPLUG-Owl:Modularization Empowers Large Language Models with Multimodality模块化多模态对话大模型mPLUG-

10、Owl-多语言版本mPLUG-Owl:Modularization Empowers Large Language Models with Multimodality模块化多模态对话大模型mPLUG-Owl-多语言版本mPLUG-Owl:Modularization Empowers Large Language Models with Multimodality中文视频对话模型mPULG-VideomPLUG-Owl:Modularization Empowers Large Language Models with Multimodality模块化多模态对话大模型mPLUG-Owl-消融实

11、验mPLUG-Owl:Modularization Empowers Large Language Models with Multimodality训练策略和多模态指令微调数据的有效性中文视频预训练数据集Youku-mPULGYouku-mPLUG:A 10 Million Large-scale Chinese Video-Language Dataset for Pre-training and Benchmarks.中文视频预训练数据集Youku-mPULGYouku-mPLUG:A 10 Million Large-scale Chinese Video-Language Datas

12、et for Pre-training and Benchmarks.DataFunSummit#202303ModelScope实战分享ModelScope-mPLUG中英模型mPLUG-Owl:Modularization Empowers Large Language Models with Multimodalityhttps:/ Empowers Large Language Models with Multimodalityhttps:/ Empowers Large Language Models with Multimodalityhttps:/ Empowers Large Language Models with Multimodalityhttps:/