1、数字人建模和动画关键技术金小刚浙江大学CAD&CG国家重点实验室教授个人主页：http:/ MULTIMEDIAORAL2021)Xiangjun Tang,Wenxin Sun,Yongliang Yang,Xiaogang Jin,“Parametric Reshaping of Portraits in Videos”,ACM Multimedia2021,Oral Presentation,pp.4689-4697,2021数字化人脸双下巴去除（SIGGRAPH2021）input imagesresulting imagesYiqian Wu,Yongliang Yang,Qinji

2、e Xiao,Xiaogang Jin,“Coarse-to-Fine:Facial Structure Editing of Portrait Images via Latent Space Classifications,”ACM Transactions on Graphics(Proc.Siggraph 2021),2021,40(4):Article 46.数字化人脸双下巴去除（SIGGRAPH2021）Yiqian Wu,Yongliang Yang,Qinjie Xiao,Xiaogang Jin,“Coarse-to-Fine:Facial Structure Editing

3、of Portrait Images via Latent Space Classifications,”ACM Transactions on Graphics(Proc.Siggraph 2021),2021,40(4):Article 46.数字化人脸肖像去头发（CVPR 2022）Yiqian Wu,Yongliang Yang,Xiaogang Jin,“HairMapper:Removing Hair from Portraits Using GANs”,2022 IEEE/CVF Conference on Computer Vision and Pattern Recognit

4、ion(CVPR2022),June,pp.4227-4236.几个月前，美国视频博主索菲亚莱加在网络广告上看中了一件衣服，当她想要下单时，却发现衣服注明着“仅提供数字版本”，她不清楚这是什么意思。顺着广告链接，索菲亚发现这件衣服的商家是专门出售“虚拟服饰”的公司客人购买虚拟服装后，把自己的照片传给商家，随后商家会用修图软件把新衣服合成到照片里的人物身上。客人得到自己“穿上”虚拟服装的照片后，可以发布到社交媒体上，仿如真的换了一套新衣服。元宇宙里的新时尚买件“虚拟服装”，用电脑合成最潮的模样联合国欧洲经济委员会在2018年指出，时装业在可持续发展中存在着大量问题。委员会估算，时装业产生了全球2

5、0%的废水和全球10%的碳排放量。虚拟服装提供了很好的解决方案人们既可以随心所欲地“穿”新衣服，保持购买新衣服的兴奋感，同时又无需进行实际的衣物生产。元宇宙里的新时尚买件“虚拟服装”，用电脑合成最潮的模样打通物理和虚拟世界凌笛科技STYLE3D国内男装公司使用Style3D对来图生产的虚拟验证火星上的时装秀STYLE3D火星上的时装秀STYLE3DVOGUE（美国时尚杂志）：未来的时装，是依然由布料织就，还是由画笔勾勒？助力百度AI数字人希加加以百变造型亮相2022百度世界大会百度和央视新闻联合举办02用于睫毛抠图的数据集与基线方法|研究背景Background|(a)本杰明巴顿奇事(b)战斗

6、天使:阿丽塔(a)虚拟整容示例(b)Siren 虚拟人|研究背景Background研究背景2022/8/1320研究背景Background|人脸几何与毛发重建1,5,6等 眼睑、眼球重建2,8,9,眼睛运动捕捉3,4,7等2022/8/13211.Beeler et al.High-quality passive facial performance capture using anchor framesJ.ACM Trans.Graph.,2011,30(4):75:175:10.2.Bermano et al.Detailed spatio-temporal reconstructio

7、n of eyelidsJ.ACM Trans.Graph.,2015,34(4):44:1 44:11.3.Brard et al.High-quality capture of eyesJ.ACM Trans.Graph.,2014,33(6):223:1223:12.4.Brard et al.Lightweight eye capture using a parametric modelJ.ACM Trans.Graph.,2016,35(4):117:1117:12.5.Beeler et al.Gross.Coupled 3d reconstruction of sparse fa

8、cial hair and skinJ.ACM Trans.Graph.,2012,31(4):117:1117:10.6.Nam et al.Strand-accuratemulti-viewhaircaptureC.CVPR 2019,155164.7.Laura et al.Modeling and animating eye blinksJ.ACM Trans.Appl.Percept.,2011,8(3):17:117:17.8.Wen et al.Real-time 3d eye performance reconstruction for RGBD camerasJ.IEEE T

9、rans.Vis.Comput.Graph.,2017,23(12):25862598.9.Wen et al.Real-time 3d eyelids tracking from semantic edgesJ.ACM Trans.Graph.,2017,36(6):193:1193:11.相关工作：高精度人脸重建Related Work 人脸几何与毛发重建1,5,6等 眼睑、眼球重建2,8,9,眼睛运动捕捉3,4,7等 然而，目前尚没有可行的方案可以准确地编辑睫毛2022/8/1322相关工作：高精度人脸重建Related Work1.Beeler et al.High-quality p

10、assive facial performance capture using anchor framesJ.ACM Trans.Graph.,2011,30(4):75:175:10.2.Bermano et al.Detailed spatio-temporal reconstruction of eyelidsJ.ACM Trans.Graph.,2015,34(4):44:1 44:11.3.Brard et al.High-quality capture of eyesJ.ACM Trans.Graph.,2014,33(6):223:1223:12.4.Brard et al.Li

11、ghtweight eye capture using a parametric modelJ.ACM Trans.Graph.,2016,35(4):117:1117:12.5.Beeler et al.Gross.Coupled 3d reconstruction of sparse facial hair and skinJ.ACM Trans.Graph.,2012,31(4):117:1117:10.6.Nam et al.Strand-accuratemulti-viewhaircaptureC.CVPR 2019,155164.7.Laura et al.Modeling and

12、 animating eye blinksJ.ACM Trans.Appl.Percept.,2011,8(3):17:117:17.8.Wen et al.Real-time 3d eye performance reconstruction for RGBD camerasJ.IEEE Trans.Vis.Comput.Graph.,2017,23(12):25862598.9.Wen et al.Real-time 3d eyelids tracking from semantic edgesJ.ACM Trans.Graph.,2017,36(6):193:1193:11.=+1 ：图

13、像，0,1：前景蒙版，：前景图像，：背景图像2022/8/13231 Yaoyi Li,Hongtao Lu.Natural image matting via guided contextual attentionC.AAAI,2020,1145011457.基于三分图的自然图像抠图示例。从左到右分别是，输入图像，三分图，Li 和 Lu1的结果，蒙版真值。相关工作：图像抠图算法与抠图数据集Related Work2022/8/13幻灯片标题241 Rhemann et al.A perceptually motivated online benchmark for image matting

14、C.In CVPR 2009,18261833.Rhemann 等人1的抠图数据集相关工作：抠图数据集Related Work2022/8/1325蓝幕抠图 1,21.数据采集：替换物体背景（绿幕、蓝幕等）2.蒙版计算：基于三角测量法求解前景蒙版1 Alvy Ray Smith,James F.Blinn.Blue screen mattingC.In SIGGRAPH 1996.ACM,1996,259268.2 Rhemann et al.A perceptually motivated online benchmark for image mattingC.In CVPR 2009,18

15、261833.相关工作：抠图数据生成方法Related Work2022/8/1326EyelashNet：睫毛抠图数据用于睫毛抠图的数据集与基线方法2022/8/1327睫毛抠图网络Qinjie Xiao,Hanyuan Zhang,Zhaorui Zhang,Yiqian Wu,Luyuan Wang,Xiaogang Jin,Xinwei Jiang,Yongliang Yang,Tianjia Shao,Kun Zhou,EyelashNet:A Dataset and A Baseline Method for Eyelash Matting,ACM Transactions onG

16、raphics(Proc.Siggraph Asia 2021),2021,40(6):Article 217.用于睫毛抠图的数据集与基线方法2022/8/1328原始重建人脸原始重建人脸的参数化模型移除睫毛后的重建人脸移除睫毛后人脸的参数化结果研究动机2022/8/1329图像抠图方法例如：Li and Lu 2020,Lin et al.2020,Qiao et al.2020等。睫毛抠图网络睫毛抠图数据集人工移除睫毛：费时费力基于Gabor滤波的方法:Beeler et al.2012,Bermano et al.2015,Nam et al.2019,等。主要挑战2022/8/1330

17、2.睫毛一直在运动并且很难保持静止。这使得采集多个严格对齐且颜色不同的睫毛图像来估计睫毛蒙版是很有挑战性的。蓝幕抠图,Rhemann et al.2009;Smith and Blinn 19961.睫毛生长在眼睑上，而且覆盖在眼球和眼睑上。这使得睫毛图像的背景例如眼睑、眼皮等无法被分离与替换。主要挑战核心思想：睫毛数据采集细化带有初始蒙版的采集数据蒙版推理网络估计RenderEyelashNet预热训练人工选择核心思想：睫毛蒙版计算|睫毛数据采集系统16个相机365nm 紫外(UVA)闪光灯（3个）紫外闪光灯补光灯采集设备我们系统的最大紫外线辐射量：./30/国际非电离辐射防护委员会提供的辐

18、射量安全阈值紫外闪光灯48个365nm、0.06W 的LED 灯珠365nm透射滤光片紫外闪光灯睫毛上色眼部定位睫毛上色与眼部定位关闭紫外闪光灯打开紫外闪光灯荧光标记原理图像形变光流场对齐后的图像FlowNet2对齐矫正对齐矫正对齐推理阶段|GCA网络示意图11 Yaoyi Li,Hongtao Lu.Natural image matting via guided contextual attentionC.AAAI,2020,1145011457.GCA网络Li and Lu 2020GCA网络:三分图中的灰色区域:三分图中的黑色区域RGB图像三分图输出：睫毛蒙版,=,=min max(,

19、0.1),10,GCA网络蒙版推理网络RGB 图像输出：睫毛蒙版睫毛遮罩(,)是睫毛遮罩中第,个像素的蓝色通道值。,=19,1,+1,1,+1蒙版推理网络蒙版推理网络RenderEyelashNetNormal image对齐后的图像Diff相减采集的图像预热网络训练Alpha mattes渐进式训练第一轮数据集(R1)RenderEyelashNetCaptured data withinitial alpha mattes人工选择视觉上正确的睫毛抠图数据蒙版推理网络预测的睫毛蒙版更新包含细分后的睫毛蒙版的采集数据渐进式训练包含细分后的睫毛蒙版的采集数据The first round dat

20、aset(R1)人工选择第二轮数据集(R2)RenderEyelashNet视觉上正确的睫毛抠图数据视觉上正确的睫毛抠图数据渐进式训练人工选择视觉上正确的睫毛抠图数据视觉上正确的睫毛抠图数据RenderEyelashNet睫毛抠图网络训练第二轮数据集(R2)渐进式训练睫毛抠图网络Test基线网络GCA网络示意图11 Yaoyi Li,Hongtao Lu.Natural image matting via guided contextual attentionC.AAAI,2020,1145011457.基线网络数据集|从12个眼睛表情（左）和15个视图（右）捕捉到的睫毛数据范例。EYELAS

21、HNETRenderEyelashNet采集测试数据集互联网测试数据集基线测试数据集方法对比|R0R1R2渐进式训练的结果真值我们的结果RenderEye-lashNetLi and Lu 2020Nam et al.2019输入三分图与最先进的方法进行比较采集测试数据集SADMSEGradConnNam et al.20196.880.010326.761.11Li and Lu 20203.680.005345.150.88RenderEyelashNet2.910.993523.571.28我们的结果2.470.002592.571.23互联网测试数据集SADMSEGradConnNam

22、 et al.201910.120.020520.311.46Li and Lu 20205.020.009310.490.83RenderEyelashNet3.80.00356.731.6我们的结果30.0014.351.48与最先进的方法进行比较与最先进的方法进行比较输入基线网络蒙版推理网络消融实验结果展示|在互联网图像上的睫毛抠图示例在采集图像上的睫毛抠图示例应用应用睫毛换颜色睫毛增长应用：睫毛美化编辑局限性 我们提出了EyelashNet，这是第一个高质量的睫毛抠图数据集，包含5400个高质量捕获的睫毛抠图数据和5,272个虚拟睫毛抠图数据。我们提出了一个专门设计的荧光标记系统来捕捉

23、高质量的睫毛图像和遮罩。我们的方法在睫毛抠图上实现了当前最先进的性能。用于睫毛抠图的数据集与基线方法总结03基于深度学习的宽松衣服实时动画|Loose-Fitting Garments 2022 SIGGRAPH.All Rights Reserved.70Xiaoyu Pan,Jiaming Mai,Xinwei Jiang,Dongxue Tang,Jingxiang Li,Tianjia Shao,Kun Zhou,Xiaogang Jin,Dinesh Manocha,“Predicting Loose-Fitting Garment Deformations Using Bone-D

24、riven Motion Networks”,Proceedings of Siggraph2022,2022,article 11.71Virtual BoneA set of bones that drive garment deformations using rigid transformations Efficient representation for complex deformations Better guidance for garment details 2022 SIGGRAPH.All Rights Reserved.Related Works 2022 SIGGR

25、APH.All Rights Reserved.Physics-Based SimulationGarment Animation Deforms the garments using laws of physics.Computational costly.Wang 2021Data-Driven ModelsLearns from a set of ground-truth garments.Faster performanceComplementary to PBSSantesteban et al.2019 2022 SIGGRAPH.All Rights Reserved.74Dat

26、a-Driven Garment AnimationLoose-fittingDynamicSim-paramVariationsSantesteban et al.2019Patel et al.2020Wang et al.2019Ours 2022 SIGGRAPH.All Rights Reserved.A deep-learning-based method for simulating complex deformations of loose-fitting garments using virtual bonesA novel method to handle the hete

27、rogeneity between simulation parameters and body motions by modeling them in different types of networks75Contributions 2022 SIGGRAPH.All Rights Reserved.Overview 2022 SIGGRAPH.All Rights Reserved.Virtual Bone GenerationLaplacianSmoothingSimulation SequenceSkinDecompositionLow-Frequency MeshesVirtua

28、l Bones&Skin Weights 2022 SIGGRAPH.All Rights Reserved.Motion NetworkVirtual Bone Motion Sequence(Red Balls)High-FrequencyDeformation AddedLow-FrequencyDeformationBody Motion Sequence(Blue Balls)2022 SIGGRAPH.All Rights Reserved.Simulation Parameter VariationsSequence of Unseen Sim-ParametersEstimat

29、ed Sequencesof Different Sim-Parameters 2134RBFRBFRBFRBF1234 2022 SIGGRAPH.All Rights Reserved.Method 2022 SIGGRAPH.All Rights Reserved.Data Preparation Simulation 81We use Houdini Vellum Solver to generate physical simulation sequence 1,2,3.Given a garment,we perform Laplacian smoothing to obtain t

30、he low-frequency part and the high-frequency part=2022 SIGGRAPH.All Rights Reserved.Data Preparation Skin Decomposition82We perform skin decomposition to decompose the low-frequency mesh sequence into an LBS(Linear Blend Skinning)model,which contains a rest pose and skin weights matrix|,and a sequen

31、ce of virtual bones motions1,2,.Virtual Bones-Non-hierarchical,have rotations and translations.-No semantic meanings 2022 SIGGRAPH.All Rights Reserved.Motion Network Low Frequency Module83GRULBSMotion Network:Only takes body motions and generates results corresponding to simulation parameters Given

32、a body motion sequence 1,2,a recurrent neural network(Gated Recurrent Unit)transfers it to the motion of virtual bones 1,2,.And we can obtain the low-frequency mesh using LBS:GLF=(;,).2022 SIGGRAPH.All Rights Reserved.Motion Network High Frequency Module84GRUGNNGlobal FeaturesLocal FeaturesMLPA GRU

33、is used to extract the global features,and a Graph Neural Network(GNN)is used to extract the local features.The features are concatenatedand processed by another MLP to get the high-frequency deformation.The final result of a motion network:=+2022 SIGGRAPH.All Rights Reserved.RBF-Based Simulation Pa

34、rameter Variations85Different motion networks generates results corresponding to different simulation parameters 1,2,We use an RBF based network to calculate the sum weight:,=exp(2222),where is a MLP projecting sim-param to the latent space.The final result:=0,RBF 2022 SIGGRAPH.All Rights Reserved.R

35、esults 2022 SIGGRAPH.All Rights Reserved.Results 2022 SIGGRAPH.All Rights Reserved.Resultsperformance For a single motion network,about 3ms(300fps)a frame on 2080Ti for about 20K vertices If the user wants to change simulation parameters interactively,we use 8 motion networks,25ms/frame(40 fps)2022

36、SIGGRAPH.All Rights Reserved.ResultsGenerated garments on unseen motionsEstimation of the same frame under different simulation parametersOursGround Truth 2022 SIGGRAPH.All Rights Reserved.How Many Virtual Bones?2022 SIGGRAPH.All Rights Reserved.Improvement on Loose Parts 2022 SIGGRAPH.All Rights Re

37、served.Quantitative Comparison 2022 SIGGRAPH.All Rights Reserved.Qualitative ComparisonLow-Frequency Module ComparisonHigh-Frequency Module Comparison 2022 SIGGRAPH.All Rights Reserved.Quantitative ResultsOur method generates results with the lowest error on loose-fitting garments 2022 SIGGRAPH.All

38、Rights Reserved.Performance on tight garmentsOn tight garments,our method generates comparable resultscompared with previous methods 2022 SIGGRAPH.All Rights Reserved.Ablation Study on methods handling varying simulation parametersRBF-based network better handles the varying simulation parameters compared with other methods.2022 SIGGRAPH.All Rights Reserved.Discussion 2022 SIGGRAPH.All Rights Reserved.Future Work98Collision handlingOther skinning methods 2022 SIGGRAPH.All Rights Reserved.|总结 高精度人脸重建和动画是众多图形应用的核心 服装覆盖了人体80%以上，也是数字人不可或缺的重要组成部分 发展方向：高度逼真、实时、低成本的新方法非常感谢您的观看|