《2. 信源信道联合编码—从图像到任务(范晓鹏).pdf》由会员分享,可在线阅读,更多相关《2. 信源信道联合编码—从图像到任务(范晓鹏).pdf(54页珍藏版)》请在三个皮匠报告上搜索。
1、范晓鹏哈尔滨业学 教授信源信道联合编码从图像到任务一背景二三四数模混合视频通信信源信道联合编码五基于深度学习的图像编码基于深度学习的多任务JSCC背景图像视频占互联网流量的90%(2013年)背景压缩效率不断提升背景容错性差问题 无线条件下如何提升视频抗噪能力信源信道联合信源编码信道编码调制式:BPSK、QPSK、MPSK、QAM信道码:LDPC、Turbo、Polar码流级具:可分级、多路码流切换帧级:MDC、冗余帧块级:Intra refresh、FMO广义的信源信道联合编码就是信源和信道编码的联合优化可以一定程度提升鲁棒性信源信道联合编码 提高鲁棒性的视频编码工具 另外还有可分级编码、l
2、oss-aware RDO等信源信道联合编码 信道编码工具面临的问题悬崖效应 无信道码保护时Xiaopeng Fan,Junhao Song,Daniel P.Palomar,Oscar Au,Universal Binary Semidefinite Relaxation for ML Signal Detection,IEEE Trans.Communications,vol.61,no.11,pp.4565-4576,2013面临的问题悬崖效应 信道码保护后出现悬崖面临的问题悬崖效应 视频数据的抗噪能力弱Xiaopeng Fan,Oscar C.Au,Mengyao Ma,Ling Ho
3、u,Jiantao Zhou and Ngai Man Cheung,“Transcoding Based Robust Streaming of Compressed Video,”in IEEE International Conference on Acoustics,Speech,and Signal Processing(ICASSP 2009),Apr.2009.悬崖效应信道SNRC=R信源信道联合编码信源编码信道编码信源信道联合编码=12log!xx=!#!+#!悬崖效应信道SNR分离的编码信源信道联合编码可分级编码+信道码+HM视频数据信道码可分级编码+信道码+HM数模混合视频
4、通信SoftcastS.Jakubczak,H.Rahul,and D.Katabi.One-Size-Fits-All Wireless Video.In Proc.Eighth ACM SIGCOMM HotNets Workshop,New York City,NY,October 2009数模混合视频通信 存在的两个问题S.Jakubczak,H.Rahul,and D.Katabi.One-Size-Fits-All Wireless Video.In Proc.Eighth ACM SIGCOMM HotNets Workshop,New York City,NY,October
5、2009VS数模混合视频通信DCastu陪集码的能量分配u运动矢量的能量分配u能量失真优化马尔可夫随机场功率谱密度Xiaopeng Fan,Feng Wu,Debin Zhao,Oscar C.Au,Distributed Wireless Visual Communicaon with Power Distoron Opmizaon,IEEE Trans.Circuits Syst.Video Technol.,vol.23,no.6,pp.1040 1053,2013.(pdf)数模混合视频通信LayerCastXiaopeng Fan,Feng Wu,Debin Zhao,Oscar C
6、.Au,Distributed Wireless Visual Communicaon with Power Distoron Opmizaon,IEEE Trans.Circuits Syst.Video Technol.,vol.23,no.6,pp.1040 1053,2013.(pdf)u块之间的能量分配数模混合视频通信其他Cast Chaofan He,Yang Hu,Yan Chen,Xiaopeng Fan,Houqiang Li,Bing Zeng,“MUcast:Linear UncodedMultiuser Video Streaming with Channel Assi
7、gnment and Power Allocation”,IEEE Trans.on CSVT,30(4):1136-1146(2020)Hangfan Liu,Ruiqin Xiong,Xiaopeng Fan,Debin Zhao,Yongbing Zhang,Wen Gao,“CG-Cast:Scalable Wireless Image SoftCast Using Compressive Gradient”,IEEE Trans.On CSVT,29(6):1832-1843(2019)Chaofan He,Huiying Wang,Yang Hu,Yan Chen,Xiaopeng
8、 Fan,Houqiang Li,Bing Zeng,“MCast:High-Quality Linear Video Transmission with Time and Frequency Diversities”,IEEE Trans.on Image Processing,27(7):3599-3610(2018).Wenbin Yin,Xiaopeng Fan*,Yunhui Shi,Ruiqin Xiong,Debin Zhao,“Compressive Sensing Based Soft Video Broadcast Using Spatial-Temporal Sparsi
9、ty”,Mobile Networks and Applications,21(6),pp 1002-1012,2016.Hagag,Ahmed,Xiaopeng Fan*,and Fathi E.Abd El-Samie,“Hyperspectral image coding and transmission scheme based on wavelet transform and distributed source coding”,Multimed Tools Appl,vol 76,Issue 22,pp 2375723776,2017.Hagag,Ahmed,Xiaopeng Fa
10、n*,and Fathi E.Abd El-Samie,“Hyperspectral satellite Image Broadcasting with Band Ordering Optimization”Elsevier Journal of Visual Communication and Image Representation(JVCI),Volume 42,January 2017,Pages 14-27.Hagag,Ahmed,Xiaopeng Fan*,and Fathi E.Abd El-Samie,“Satellite Images Broadcast based on w
11、ireless SoftCast scheme,”International Journal of Computer Science,2017.Ruiqin Xiong,Feng Wu,Jizheng Xu,Xiaopeng Fan,Chong Luo,Wen Gao,“Analysis of DecorrelationTransform Gain for Uncoded Wireless Image and Video Transmission”,IEEE Trans.Image Processing.Vol.25,no.4,pp 1820-1833,2016.基于深度学习的图像编码技术 基
12、于变分自动编码器的可训练模型来实现端到端图像压缩和重构,先使用卷积神经网络提取图像特征,然后对特征进行量化和熵编码。Ball J,Minnen D,Singh S,et al.Variaonal image compression with a scale hyperpriorJ.arXiv preprint arXiv:1802.01436,2018.基于深度学习的端到端视频编码方案 提出一个全新的端到端的视频编码框架,包括光流估计网络,运动补偿网络、残差变换编码网络,码率估计熵编码网络。Lu G,Ouyang W,Xu D,et al.Dvc:An end-to-end deep vide
13、o compression frameworkC/Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2019:11006-11015.Neural Video Compression with Diverse Contexts(2023)Jiahao Li and Bin Li and Yan Lu,Neural Video Compression with Diverse Contexts,arXiv2023Neural Video Compression with Diverse Co
14、ntexts(2023)Jiahao Li and Bin Li and Yan Lu,Neural Video Compression with Diverse Contexts,arXiv2023!#$#%&%#&()*&+*,-.)/*0!#$#%&%#&()*&+*,-.)/*05678678%#%#9:;9:;Siarohin,Aliaksandr,et al.First order motion model for image animation.NeurIPS 2019.5678678%#%#9:;9:;Siarohin,Aliaksa
15、ndr,et al.First order moon model for image animaon.NeurIPS 2019.BITRATE/PSNR/SSIMGTDecodeHM/QP38HM/QP451.004Kbps/22.318/0.7364 6.760Kbps/34.387/0.9197 2.903Kbps/30.133/0.8474基于深度学习的信源信道联合编码 面向图像重构的信源信道联合编码模型Bourtsoulatze E,Kurka D B,Gndz D.Deep joint source-channel coding for wireless image transmis
16、sionJ.IEEE Transactions on Cognitive Communications and Networking,2019,5(3):567-579.基于深度学习的信源信道联合编码 带反馈的Deep JSCCKurka D B,Gndz D.Deepjscc-f:Deep joint source-channel coding of images with feedbackJ.IEEE Journal on Selected Areas in Informaon Theory,2020,1(1):178-193.基于深度学习的信源信道联合编码Mingze Ding,Jiah
17、ui Li,Mengyao Ma,Xiaopeng Fan,“SNR-adaptive deep joint source-channel coding for wireless image transmission”,ICASSP 2021.SNR自适应Deep JSCC基于深度学习的信源信道联合编码Xu J,Ai B,Chen W,et al.Wireless image transmission using deep source channel coding with alenon modulesJ.IEEE Transacons on Circuits and Systems for
18、 Video Technology,2022.基于Attention的Deep JSCC基于关键点的人类视频编码与动作检测关键点+关键帧目标检测+压缩Human Video Compression via Sparse Point-Guided Temporal Propagaon,ISO/IEC JTC1/SC29/WG11 MPEG2020/M52136,129th MPEG meeng,January 2020,Brussels,Belgium带有语义结构码流的检测驱动图像编码 提出了一种语义结构编码(SSC)框架 码流中增加语义信息 语义信息可帮助重建图像,也可以直接完成各种分析任务,
19、如姿态估计和分类。Detecon-driven Image Compression with Semancally Structured Bit-stream,ISO/IEC JTC1/SC29/WG11 MPEG2020/M52273,129th MPEG meeng,January 2020,Brussels,Belgium面向分类任务的信源信道联合编码 BottleNet+在AWGN 信道和BEC信道分别实现了64 倍和256倍的维度压缩(分类精度下降2%以内)Shao J,Zhang J.Bollenet+:An end-to-end approach for feature comp
20、ression in device edge co-inference systemsC/2020 IEEE Internaonal Conference on Communicaons Workshops(ICC Workshops).IEEE,2020:1-6.基于深度学习的多任务图像特征提取技术 Fast RCNNMask-RCNNK.He,G.Gkioxari,P.Dollar,and R.Girshick,“Mask r-cnn,”in 2017 IEEE Internaonal Conference on Computer Vision(ICCV),2017,pp.29802988
21、.多任务Deep特征压缩 基于特征融合的多尺度端到端特征压缩方法MSFC:DEEP FEATURE COMPRESSION IN MULTI-TASK NETWORKC/2020-2021 IEEE International Conference on Multimedia&Expo(ICME).IEEE,2021:多任务Deep特征压缩 基于特征融合的多尺度端到端特征压缩方法object deteconinstance segmentaonMSFC:DEEP FEATURE COMPRESSION IN MULTI-TASK NETWORKC/2020-2021 IEEE Internat
22、ional Conference on Multimedia&Expo(ICME).IEEE,2021:OPENG image dataset多任务Deep JSCCMengyang Wang,Zhicong Zhang,Jiahui Li,Mengyao Ma,Xiaopeng Fan*,“Deep Joint Source-Channel Coding for Multi-Task Network,”IEEE Signal Processing Letters,(2021).Deep JSCC的星座图设计Mengyang Wang,Jiahui Li,Mengyao Ma,Xiaopeng
23、 Fan*,“Constellaon Design for Deep Joint Source-Channel Coding,”IEEE Signal Processing Lelers,29:1442-1446(2022).Deep learning enabled semantic communication H.Xie,Z.Qin,G.Y.Li,and B.-H.Juang,“Deep learning enabled semantic communication systems,”IEEE Transactions on Signal Processing,vol.69,pp.2663
24、2675,2021.SNN!#$%&()*+,-.#$/012!#$/012SNN-SCMengyang Wang,Xiaopeng Fan,2210.06836v1 S-JSCC:A Digital Joint Source-Channel Coding Framework based on Spiking Neural Network(arxiv.org)SNN-SCMengyang Wang,Xiaopeng Fan,2210.06836v1 S-JSCC:A Digital Joint Source-Channel Coding Framework based on Spiking N
25、eural Network(arxiv.org)SNN vs Resnet50SNN vs VGG16SNN SC for Rerieval1.Wenrui Li,Zhengyu Ma,Liangjian Deng,Xiaopeng Fan*,Yonghong Tian,“Neuron-Based Spiking Transmission and Reasoning Network For Robust Image-Text Retrieval,”,IEEE Trans.On CSVT,2022.SNN SC for Rerieval1.Wenrui Li,Zhengyu Ma,Liangji
26、an Deng,Xiaopeng Fan*,Yonghong Tian,“Neuron-Based Spiking Transmission and Reasoning Network For Robust Image-Text Retrieval,”,IEEE Trans.On CSVT,2022.SNN SC for Rerieval1.Wenrui Li,Zhengyu Ma,Liangjian Deng,Xiaopeng Fan*,Yonghong Tian,“Neuron-Based Spiking Transmission and Reasoning Network For Rob
27、ust Image-Text Retrieval,”,IEEE Trans.On CSVT,2022.SNN SC for Rerieval1.Wenrui Li,Zhengyu Ma,Liangjian Deng,Xiaopeng Fan*,Yonghong Tian,“Neuron-Based Spiking Transmission and Reasoning Network For Robust Image-Text Retrieval,”,IEEE Trans.On CSVT,2022.SNN SC for Rerieval1.Wenrui Li,Zhengyu Ma,Liang-J
28、ian Deng,Penghong Wang,Jinqiao Shi,Xiaopeng Fan*,“Reservoir Computing Transformer for Image-Text Retrieval”,ACM Multimedia 2023.SNN SC for Rerieval1.Wenrui Li,Zhengyu Ma,Jinqiao Shi,Xiaopeng Fan,“The style transformer with common knowledge optimization for image-text retrieval”,IEEE Signal Processin
29、g Letter,2023.发表的相关论文-传统JSCC Chaofan He,Yang Hu,Yan Chen,Xiaopeng Fan,Houqiang Li,Bing Zeng,“MUcast:Linear UncodedMultiuser Video Streaming with Channel Assignment and Power Allocation”,IEEE Trans.on CSVT,30(4):1136-1146(2020)Hangfan Liu,Ruiqin Xiong,Xiaopeng Fan,Debin Zhao,Yongbing Zhang,Wen Gao,“C
30、G-Cast:Scalable Wireless Image SoftCast Using Compressive Gradient”,IEEE Trans.On CSVT,29(6):1832-1843(2019)Chaofan He,Huiying Wang,Yang Hu,Yan Chen,Xiaopeng Fan,Houqiang Li,Bing Zeng,“MCast:High-Quality Linear Video Transmission with Time and Frequency Diversities”,IEEE Trans.on Image Processing,27
31、(7):3599-3610(2018).Wenbin Yin,Xiaopeng Fan*,Yunhui Shi,Ruiqin Xiong,Debin Zhao,“Compressive Sensing Based Soft Video Broadcast Using Spatial-Temporal Sparsity”,Mobile Networks and Applications,21(6),pp 1002-1012,2016.Hagag,Ahmed,Xiaopeng Fan*,and Fathi E.Abd El-Samie,“Hyperspectral image coding and
32、 transmission scheme based on wavelet transform and distributed source coding”,Multimed Tools Appl,vol 76,Issue 22,pp 2375723776,2017.Hagag,Ahmed,Xiaopeng Fan*,and Fathi E.Abd El-Samie,“Hyperspectral satellite Image Broadcasting with Band Ordering Optimization”Elsevier Journal of Visual Communicatio
33、n and Image Representation(JVCI),Volume 42,January 2017,Pages 14-27.Hagag,Ahmed,Xiaopeng Fan*,and Fathi E.Abd El-Samie,“Satellite Images Broadcast based on wireless SoftCast scheme,”International Journal of Computer Science,2017.Ruiqin Xiong,Feng Wu,Jizheng Xu,Xiaopeng Fan,Chong Luo,Wen Gao,“Analysi
34、s of DecorrelationTransform Gain for Uncoded Wireless Image and Video Transmission”,IEEE Trans.Image Processing.Vol.25,no.4,pp 1820-1833,2016.发表的相关论文-面向任务JSCC Wenrui Li,Zhengyu Ma,Liang-Jian Deng,Penghong Wang,Jinqiao Shi,Xiaopeng Fan*,“Reservoir Computing Transformer for Image-Text Retrieval”,ACM M
35、ultimedia 2023.Wenrui Li,Zhengyu Ma,Liangjian Deng,Xiaopeng Fan*,Yonghong Tian,“Neuron-Based Spiking Transmission and Reasoning Network For Robust Image-Text Retrieval,”,IEEE Trans.On CSVT,2023.Wenrui Li,Zhengyu Ma,Jinqiao Shi,Xiaopeng Fan,“The style transformer with common knowledge optimization fo
36、r image-text retrieval”,IEEE Signal Processing Letter,2023.Mengyang Wang,Jiahui Li,Mengyao Ma,Xiaopeng Fan*,“Constellation Design for Deep Joint Source-Channel Coding,”IEEE Signal Processing Letters,29:1442-1446(2022).Mengyang Wang,Zhicong Zhang,Jiahui Li,Mengyao Ma,Xiaopeng Fan*,“Deep Joint Source-Channel Coding for Multi-Task Network,”IEEE Signal Processing Letters,28:1973-1977(2021).感谢参与THANKS