1、 The KhronosGroup Inc.2023-Page 1This work is licensed under a Creative Commons Attribution 4.0 International LicenseOpen Standards Powering the Future of Embedded Vision Embedded World ChinaJune 2023Neil TrevettPresident,KhronosVP Developer Ecosystems,NVIDIA The KhronosGroup Inc.2023-Page 2This wor

2、k is licensed under a Creative Commons Attribution 4.0 International LicenseTopicsThe need for parallel offload of vision and inferencing processingThe need for open standard APIs for embedded visionIntroduction to Khronos and its family of open API standardsKhronos and Safety Critical APIsIntroduct

3、ion to the new Kamaros Embedded Camera API in development How to get involved!The KhronosGroup Inc.2023-Page 3This work is licensed under a Creative Commons Attribution 4.0 International LicenseKhronos Connects Software to SiliconOpen,royalty-free interoperability standards to harness the power of G

4、PU,XR and multiprocessor hardware3D graphics,augmented and virtual reality,parallel programming,inferencing and vision accelerationNon-profit,member-driven standards organization,open to any companyProven multi-company governance and Intellectual Property FrameworkFounded in 2000 200 Members|40%US,3

5、0%Europe,30%Asia The KhronosGroup Inc.2023-Page 4This work is licensed under a Creative Commons Attribution 4.0 International LicenseIncreasing Embedded Vision Processing DemandsIncreasing Number and Diversity of Sensors per SystemIncluding camera arrays and depth sensors such as LidarIncreasing Sop

6、histication of Sensor and Vision ProcessingIncluding inferencingRapidly increasing demand for higher performance vision processing The KhronosGroup Inc.2023-Page 5This work is licensed under a Creative Commons Attribution 4.0 International LicenseThe Need for Parallel ProcessingSingle ProcessorSimpl

7、e to programbutMay not provide enough performanceespeciallyAs Moores Law frequency/power scaling is slowing Multi-ProcessorAdditional processors can process expanded workloads butAdds complexity to system design and programming,as there is the need to:(i)Distribute workload across processors(ii)Sync

8、hronize use of compute and memory resources(iii)Communicate intermediate data and resultsCan open standards help solve this complexity?The KhronosGroup Inc.2023-Page 6This work is licensed under a Creative Commons Attribution 4.0 International LicenseOpen Standards Make Technology PervasiveOpen Stan

9、dard=Shared SpecificationImplementationImplementationImplementationImplementationImplementationContributorContributorContributorContributorContributorOpen Source=Shared ImplementationOpen standards with rigorous conformance testing enable consistency across multiple implementations that can meet the

10、 needs of diverse markets,price points,and use casesOpen standards often use open source to spread the implementation effort for sample implementations,tools,samples,conformance tests,validators etcINTEROPERABILITY standards define precise COMMUNICATIONE.g.,software to hardware,client to server The

11、KhronosGroup Inc.2023-Page 7This work is licensed under a Creative Commons Attribution 4.0 International LicenseOpen Standard Whys,Whens and HowsWhyGrow MarketsBy reducing consumer confusion and increasing capabilities and usabilityReduce CostsBy sharing ecosystem development between many companies

12、and driving volumeSpeed Time to MarketWith well-proven functionality,testing and interoperabilityEncourage InnovationFree companies to compete on value differentiators:quality,performance,power etc.WhenTechnologies are ProvenDont do R&D by standards committeeConsensus NeedThe downside of not having

13、a standard is clear to allHowMulti-company governanceAvoid single-company control or dependencyWell-defined IPR Policy Royalty-free standards have clearer path to wide adoptionWith Extensibility Enable a responsive innovation pipeline to meet customer and market needsThoughtful Level of AbstractionF

14、ocus on interoperability and avoid over specifying implementation which stifles innovation The KhronosGroup Inc.2023-Page 8This work is licensed under a Creative Commons Attribution 4.0 International LicenseKhronos Active Standards3D GraphicsDesktop,Mobileand Web3D Assets Authoringand DeliveryPortab

15、le XRAugmented and Virtual RealityParallel ComputationVision,Camera,Inferencing,Machine LearningSafety Critical APIs Cross-platform software reusability Decoupled software and hardware for easier development and integration of new components Cross-generation reusability and field upgradability Indus

16、try-wide ecosystem of tools and librariesOpen Standard APIs in Embedded Markets The KhronosGroup Inc.2023-Page 9This work is licensed under a Creative Commons Attribution 4.0 International LicenseKhronos Compute Acceleration StandardsGPUGPU rendering+compute accelerationHeterogeneous compute acceler

17、ationSingle source C+programming with compute accelerationGraph-based vision and inferencing accelerationIntermediate Representation(IR)supporting parallel execution and graphicsHigher-level Languages and APIsStreamlined development and performance portabilityGPUFPGADSPCustom Hardware GPUCPUCPUCPUAI

18、/Tensor HWLower-level Languages and APIsExplicit hardware controlNeural Network Exchange FormatTrained NetworksMultiple programming abstractions to meet the needs of diverse software stack architecturesThird party vision,streaming and inferencing librariesApplications,libraries,and higher-level lang

19、uages and APIs use lower-level Khronos standards to access hardware acceleration The KhronosGroup Inc.2023-Page 10This work is licensed under a Creative Commons Attribution 4.0 International LicenseOpenCL Low-level Parallel ProgramingComplements GPU-only APIsSimpler programming modelRelatively light

20、weight run-timeMore language flexibility,e.g.,pointersRigorously defined numeric precisionOpenCL Kernel CodeOpenCL Kernel CodeOpenCL Kernel CodeOpenCL C Kernel CodeGPUDSPCPUCPUFPGAOpenCLDevicesHost CPUNN HWRuntime OpenCL API to compile,load and execute kernels across devicesProgramming and Runtime F

21、ramework for Application AccelerationOffload compute-intensive kernels onto parallel heterogeneous processorsCPUs,GPUs,DSPs,FPGAs,Tensor ProcessorsOpenCL C or C+kernel languagesPlatform Layer APIQuery,select and initialize compute devicesRuntime APIBuild and execute kernels programs on multiple devi

22、cesExplicit Application ControlWhich programs execute on what deviceWhere data is stored in memories in the systemWhen programs are run,and what operations are dependent on earlier operations The KhronosGroup Inc.2023-Page 11This work is licensed under a Creative Commons Attribution 4.0 Internationa

23、l LicenseExecuting OpenCL ProgramsContextProgramsKernelsMemoryObjectsCommandQueuesSend for executionProgramsKernel0Kernel1Kernel2ImagesBuffersIn order&out of orderCreate data&argumentsCompileA kernel program is the basic unit of executable code(similar to a C function)An OpenCL program is a collecti

24、on of kernels and functionsAn OpenCL command queue is used by the host application to send kernels and data transfer functions to a device for execution.By enqueueing commands into a command queue,kernels and data transfer functions may execute asynchronously and in parallel with application host co

25、deAs an open standard,OpenCL is a well proven design,available from many silicon vendors with an extensive ecosystem of available tools,compilers,libraries and educational materials The KhronosGroup Inc.2023-Page 12This work is licensed under a Creative Commons Attribution 4.0 International LicenseG

26、rowing Need for APIs for Functional Safety1990sAvionics2010sAutomotive2020sEverywhereGrowing demand for accelerated graphics and compute where functional safety is paramountAutonomous vehicles and devices,avionics,medical,industrial,and energyIndustry safety-critical standards includeRTCA DO-178C(av

27、ionics)|ISO 26262(automotive)IEC 61508(industrial)|IEC 62304(medical)Safety-critical APIs are designed to reduce system-level certification effort and costs 1)Streamlined to reduce documentation and testing surface area2)Deterministic behavior to simplify system design and testing3)Unambiguous and c

28、omprehensive fault handling The KhronosGroup Inc.2023-Page 13This work is licensed under a Creative Commons Attribution 4.0 International LicenseKhronos Safety Critical Standards EvolutionOpenGL ES 1.0-2003Fixed function graphicsOpenGL ES 2.0-2007Programmable ShadersOpenGL SC 1.0-2005Safety-critical

29、 subsetOpenGL SC 2.0-2016Safety-critical subsetVulkan 1.2-2020Explicit Graphics,Compute and DisplayVulkan SC 1.0-2022Safety-critical subsetSYCL 2020C+-based heterogeneous parallel programmingMarch 2022SYCL SC Working Group announcedKhronos has 20 years experience in standards for safety-critical mar

30、kets Leveraging proven mainstream standards with shipping implementations and developer tooling and familiarityA choice of abstraction levels to suit different markets and developer needsOpenVX SC Extension 2017Vision/inferencing OpenVX 1.3 2019SC Extension integrated into core OpenVX The KhronosGro

31、up Inc.2023-Page 14This work is licensed under a Creative Commons Attribution 4.0 International LicenseNeed for Embedded Camera API StandardEmbedded SystemCameraSensor andISPVision andInferencingAccelerationSensor StreamApplication controlling sensor stream generation and processing in real timeChoi

32、ce of multiple open standard,cross-vendor APIsNo widely adopted,open standard,cross-vendor APIsBenefits of a Cross-vendor Embedded Camera APIPortability of camera/sensor code for easier system integration of new sensorsPreservation of application code across multiple generations of cameras and senso

33、rsSophisticated control over sensor stream generation increases effectiveness of downstream accelerated processing An effective open,cross-vendor open standard for camera,sensor and ISP control is an ecosystem gap compared to processing APIs The cost and time to integrate and utilize sensors in embe

34、dded systems using proprietary APIs has become a major constraint on innovation and efficiency in the embedded vision market The KhronosGroup Inc.2023-Page 15This work is licensed under a Creative Commons Attribution 4.0 International LicenseKamaros Embedded Camera APIFrameworks&MiddlewareOSGStreame

35、rSensorsPhysical DevicesLensesLightsProcessorsKamaros System RuntimeWindows Linux Android Chrome RTOSTransportCSI-2USBEthernetApplicationOpenVXLibrariesAPI in scopeSome libraries may be in scopeKamaros APINames of transport layers,framework and operating systems are illustrative examples Camera Prog

36、ramming Model based on Pipeline TemplatesDefines available streams through sensor,ISP and other processing elementsExposes static and per-frame controls-ranging from simple to more complex Balances simplicity and portability and detailed HW controlTested pipeline templates can be vendor-supplied The

37、 KhronosGroup Inc.2023-Page 16This work is licensed under a Creative Commons Attribution 4.0 International LicenseYou are Welcome to Get Involved!Any company is welcome to join Khronos to influence standards developmenthttps:/www.khronos.org/members/or email memberserviceskhronosgroup.orgMore information on any Khronos APIshttps:/www.khronos.org/Khronos is at booth A015 in the exhibition hall!We are happy to answer any questions!Khronos is developing a growing family of open,royalty-free API standards for embedded and safety-critical markets