《自动驾驶论坛-【雷鸣丨英特尔】-《AI Sensor Fusion for Intelligent Transportation》.pdf》由会员分享，可在线阅读，更多相关《自动驾驶论坛-【雷鸣丨英特尔】-《AI Sensor Fusion for Intelligent Transportation》.pdf（18页珍藏版）》请在三个皮匠报告上搜索。

1、Ming Lei(雷鸣),PhDSenior Platform AIntel CorporationAI Computing Conference(AICC),Beijing,ChinaNov.29,2023AI Sensor Fusion forIntelligent TransportationOutline Roadside Sensing Usage scenarios System architecture Traffic object attributes Pros&cons of traffic sensors Sensor Fusion(categorized by integ

2、ration level)Integrated&Distributed Sensor Fusion(categorized by involved sensor types)C+R:Camera+mmWave Radar C+L:Camera+LidarRoadside Sensing Usage Scenarios Roadside Sensing mainly relies on three types of sensors:cameras,mmWave radars,and lidars Edge Computing(e.g.,MEC deployed at roadside or mo

3、bile network edge)processes raw data from Roadside Sensing Deep learning(e.g.,object classification based on neural networks for video images or 3D point clouds)Traditional computer vision(e.g.,color space conversion for video images or clustering for 3D point clouds)Radar signal processing(e.g.,FFT

4、s for range,velocity,and AoA estimation)Roadside Sensing System ArchitectureRoadside Sensing Traffic Object AttributesRTSs(Road Traffic Signs)RTIs(Road Traffic Incidents)RTPs(Road Traffic Participants)RTS attributes defined in standards:Data Frame:DF_RTSData YD/T 3709-2020 T/CSAE 53-2020 Associated

5、RTS types GB 5768.2-2009RTI attributes defined in standards:Data Frame:DF_RTEData YD/T 3709-2020 T/CSAE 53-2020 Associated RTI types GB/T 29100-2012RTP attributes defined in standards:Data Frame:DF_ParticipantData YD/T 3709-2020 T/CSAE 53-2020Roadside Sensing Traffic Sensors Pros&ConsTraffic Sensor

6、Type(Strengths in detecting Traffic Object Attributes)AdvantagesDisadvantagesCamera(Object classification and estimation of pixel coordinates on a 2D plane)1)Rich in details,excellent discernibility;2)Can accurately capture the contour,texture,color distribution,and so on as well as generate semanti

7、c information which can facilitate the object classification under non-extreme ambient light conditions;3)Can recognize static and plane traffic signs such as traffic lanes and zebra crossing;4)High lateral resolution can be used to estimate relative lateral velocity;5)Video and image processing tec

8、hnologies are relatively mature;6)Relatively low cost and long product life.1)Susceptible to ambient light conditions(e.g.,low light at night,strong sunlight,and so on);2)Susceptible to weather conditions(e.g.,rain,snow,fog,haze,smoke,dust,and so on);3)Relatively high workload of video analytics bas

9、ed on DL;4)Lack of depth information,difficult for obtaining accurate 3D info;5)Low positioning accuracy;6)Decreased resolution at longer distance;7)Difficult to estimate relative radial/longitudinal velocity.mmWave Radar(Estimation of range,velocity and AoA)1)Capable of estimating the range,velocit

10、y(relative radial/longitudinal velocity),micro-Doppler&AoA of the target through conventional signal processing(much lower computing power than that of DL);2)Generates images with 4D radar or SAR,potential for object classification;3)Relatively long range;4)Performance is stable as range increases;5

11、)All weather conditions(e.g.,rain,snow,fog,haze,smoke,dust,etc.);6)Not affected by light conditions;7)Relatively low cost and long product life.1)Difficult for accurate object classification(sparse point cloud);2)Difficult to distinguish stationary objects(e.g.,stopped vehicles)from background(e.g.,

12、guardrails,manhole covers,etc.);3)Difficult to detect pedestrians and small objects at long range;4)Unable to recognize traffic signs and traffic signals(lights);5)Detection accuracy of the objects lateral position is not high;6)Serious lack of environmental details;7)Ghost detections&false detectio

13、ns(high false alarm probability)caused by multipath,clutter,interference&noise;8)Difficult for tracking.Lidar(High-precision positioning and estimation of 3D dimensions of objects)1)High ranging resolution and precision;2)High angular resolution;3)Wide field of view(FoV),360 FoV for most mechanical

14、lidars;4)Dense enough data that AI inference(with NN model)can make use of;5)Strong recognition abilities:capable of 3D imaging(obtaining accurate contours of pedestrians and even smaller objects)and estimation of 3D dimensions of the objects(length,width and height)and other info;6)Accurate positio

15、ning of multiple targets;7)Strong tracking ability;8)Not affected by ambient light conditions.1)Limited performance in recognizing information such as the colors of the objects,traffic signs and traffic signals;2)DL inference on the 3D point cloud generated by lidar requires relatively high computin

16、g power;3)Susceptible to certain weather conditions(e.g.,rain,snow,fog);4)Resolution&accuracy drop as range increases(not like radar);5)Measurement affected by platform movement/vibration;6)High cost.No one single type of traffic sensor can meet all requirements of roadway transportation,and thats w

17、here Sensor Fusion comes in.Multimodal sensors are not just complementary,but provide redundancy.Sensor Fusion Categories ISF&DSF(by Integration Level)C:Camera R:mmWave Radar L:LidarDSF(Distributed Sensor Fusion):Sensing(at least 4Cs/4Cs+4Rs/4Cs+2Ls for an intersection)&Computing are in multiple spl

18、it devices connected via networkISF(Integrated Sensor Fusion):a.k.a.AIO(All-In-One),Sensing(1C+1R/1C+1L)&Computingare integrated in one deviceSoC ProcessorsSoC ProcessorsiGPUiGPUdGPUFPGASensor Fusion C+R ConceptM.Lei,D.Yang and X.Weng,Integrated Sensor Fusion Based on 4D MIMO Radar and Camera:A Solu

19、tion for Connected Vehicle Applications,in IEEE Vehicular Technology Magazine,vol.17,no.4,pp.38-46,Dec.2022,doi:10.1109/MVT.2022.3207453.Impact factor of 13.609(from the Thomson Reuters Journal Citation Reports)Download:https:/ieeexplore.ieee.org/document/9913510Sensor Fusion C+R IntelSW RISoC Proce

20、ssoriGPU(in SoC)IntelSW RI(Reference Implementation):Garnet Park(codename)Computing platform Processor:IntelCeleron6305E SoC Processor(with iGPU:integrated GPU)Software toolkits IntelDistribution of OpenVINO Toolkit InteloneAPI Math Kernel Library(oneMKL)IntelCeleron6305E A Single Processor for All

21、WorkloadsIntel Celeron SoC ProcessorParameterCPU#of Cores2#of Threads2Base Frequency1.8 GHzCache4 MBBus Speed4 GT/sMemoryMax Memory Size64 GBMemory TypesDDR4-3200,LPDDR4x-3733Max#of Memory Channels2Processor GraphicsProcessor GraphicsIntel UHD GraphicsGraphics Max Dynamic Frequency1.25 GHz#of EUs48M

22、edia1 VDBOXExpansion OptionsPCIe for CPUPCIe Gen 3 4 lanesPCIe for PCHPCIe Gen 3 4 lanesOthersThunderbolt 4/USB 4 4IntelCeleron6305E SoC ProcessorProduct link of Intel Celeron 6305E SoC:https:/ Coding/DecodingRadar signal processingFusion of resultsDeep LearningTraditional Computer VisioniGPU(Integr

23、ated GPU in SoC)Sensor Fusion C+R IntelSW RI PerformanceHardware configurationHost SoC processorIntelCeleron6305E SoC Processor(2 cores,2 threads)iGPUIntelUHD Graphics48 EUs,1.25 GHzMemory8GB,DDR4,3200 MT/sSoftware configurationOSUbuntu 22.04IntelDistribution of OpenVINO Toolkit2022.3.0InteloneAPI M

24、ath Kernel Library(oneMKL)2023.2.0NN modelYOLOX-SPerformanceThroughput(average)22.61 FPSProcessing latency(T1-T0)(average)80.17 msCPU loading rate(average)85.3%(Full loading rate is 200%for 2 CPU threads)iGPU loading rate(average)42.7%(Full loading rate is 100%)System Configuration&PerformanceSoC Pr

25、ocessoriGPU(in SoC)Sensor Fusion C+R IntelSW RI Demo VideoDemo Video of Sensor Fusion C+R on IntelProcessorObject classification result is from video analyticsRadar Cartesian Coordinate System Targets position is derived from range&angle measured by radar Velocity&heading measured by radar are visua

26、lized by a directional line segmentObject-level results generated by video analytics and radar signal processing are fused in the radars Cartesian coordinate systemSensor Fusion C+R IntelSW RI Demo VideoSensor Fusion C+L Solution White PaperDownloadEnglish Version下载中文版English version is also availab

27、le:https:/ et al.,Lidar 3D Point Cloud Processing and Sensor Fusion based on Intel Architecture for C-V2X,Intel Solution White Paper,Dec.2021.Pipeline of Sensor Fusion C+L(Camera+Lidar)Categories,3D positions(ranges),speeds,headings,and 3D sizes of objectsCategories and 2D positionsSoCiGPUdGPUSensor

28、 Fusion C+L PipelineSensor Fusion C+L All-in-One Roadside SensingLeishen All-in-One Roadside Sensing Equipment(Lidar&Camera)Lidar ModelLeishen CH128X1Ranging MethodToF(Pulsed Lidar)Laser Wavelength905 nmNumber of Lines(Laser Beams)128Maximum Range160m 10%(Reflectivity)Range Resolution3 cmData Rate(S

29、ingle Echo)760,000 points/secField of View(FoV)Vertical-18-7Horizontal120Angle ResolutionVertical0.125(Central ROI Region)0.25(Side Regions)Horizontal0.1(5 Hz)0.2(10 Hz)0.4(20 Hz)Lidar used to generate 3D point cloudSource:M.Lei et al.,Lidar 3D Point Cloud Processing and Sensor Fusion based on Intel

30、 Architecture for C-V2X,Intel Solution White Paper,Dec.2021.Sensor Fusion C+L Roadside MECProcessor SKUIntel Core i7-1165G7Intel Core i7-1185GRECPU4 cores,8 threadsConfigurable TDP-up Freq.:2.80 GHz4 cores,8 threadsConfigurable TDP-up Freq.:2.80 GHzProcessor Graphics(iGPU)Intel Iris Xe Graphics,96 E

31、Us,Graphics Max Dynamic Freq.:1.30 GHzIntel Iris Xe Graphics,96 EUs,Graphics Max Dynamic Freq.:1.35GHzAI Performance(FP32)1.996 TFLOPS(iGPU)0.358 TFLOPS(CPU)2.073 TFLOPS(iGPU)0.358 TFLOPS(CPU)AI Performance(INT8)7.987 TOPS(iGPU)1.433 TOPS(CPU)8.294 TOPS(iGPU)1.433 TOPS(CPU)Memory8GB,DDR4,Speed:2400

32、MT/s8GB,LPDDR4,speed:4267 MT/sBIOSAmerican Megatrends Inc.2.21.1278Intel CorporationTGLSFWI1.R00.4024.A01.2101201730Operating SystemUbuntu 20.04.1 LTSLinux Kernel5.8.0-43-genericPyTorch1.7.1+CPUOpenVINO2021.3Configurations of the 11th-Gen Intel Core processors(Tiger Lake)The maximum performance of t

33、he two processors is detailed in the product specifications:Intel Core i7-1185GRE Processor:link Intel Core i7-1165G7 Processor:linkJHCTech Roadside MEC Equipment(KMDA-3301)Specs:http:/ and Software Configuration(Used in Performance Evaluation)SoCiGPUSource:M.Lei et al.,Lidar 3D Point Cloud Processing and Sensor Fusion based on Intel Architecture for C-V2X,Intel Solution White Paper,Dec.2021.Ming Lei,PhDSenior Platform AIntel CorporationThank you!