1、 ASAM Test Specification Study Group Report 2022 1 EVOLVING LANDSCAPES OF COLLABORATIVE TESTING FOR ADAS&AD ASAM Test Specification Study Group Report 2022 Version 1.0.0 Date:2022-02-02 by ASAM e.V.,2022 Table of Contents ASAM Test Specification Study Group Report 2022 2 Disclaimer This document is

2、the copyrighted property of ASAM e.V.Any use is limited to the scope described in the license terms.The license terms can be viewed at Table of Contents ASAM Test Specification Study Group Report 2022 3 Table of Contents 1 Executive Summary.6 2 Scope.9 3 Terms and Definitions.10 3.1 Test Method.10 3

3、.2 Test Case.11 3.3 Scenario.13 3.4 Test Specification.14 3.5 The Relation between Test Case and Scenario.15 4 Automotive Industry Insights.19 4.1 Homologation and the Transforming Role of Technical Services.20 4.1.1 Testing and type approval current and future regulations.20 4.1.2 Possible impact o

4、f the study group and defined test strategy blueprint.21 4.1.3 Examples with special relevance for type approval.22 4.1.4 The trustworthiness of test environments.23 4.2 Data-driven Development and Shifting Responsibilities.24 4.2.1 How does automated driving change the electrical/electronic(E/E)dev

5、elopment process?.24 4.2.2 How does automated driving change the release and homologation process?.26 4.3 Existing Standards and Current Research Activities.27 4.3.1 Infrastructure Representation.28 4.3.2 Domain Representation&Taxonomies.34 4.3.3 Interface Definitions.40 4.3.4 Test Specification.50

6、4.3.5 Data Handling.62 4.3.5.1 ODS and MDF.62 4.3.6 Methods and Processes.70 4.3.7 Research Projects Ongoing or recent.79 5 A Blueprint for New ADAS/AD Test Strategies.84 5.1 The Derived Blueprint.85 5.2 User Journeys and Use Cases.88 5.2.1 Requirements Based Testing on Closed Loop HIL+Requirements

7、Based Test SIL.89 Table of Contents ASAM Test Specification Study Group Report 2022 4 5.2.2 Requirements Based Testing MIL.91 5.2.3 Fault Injection Testing MIL.95 5.2.4 Scenario-based Open Road Testing.99 5.2.5 Scenario-based Testing on Proving Grounds.102 5.2.6 Hardware Reprocessing/Data Replay.106

8、 5.2.7 Requirements-based Vehicle-in-the-Loop Testing.109 5.2.8 Scenario-based SIL Closed Loop Testing.111 5.3 AI Specific Aspects.117 6 Test Data Management.121 6.1 Data Modeling Based on Test Strategy and Use Cases.122 6.2 Impact on Homologation and Type Approval Process.126 6.3 Proposal for the T

9、echnical Application in the ASAM ODS Standard.127 6.3.1 Preview of an ODS application model.127 7 Recommendations.131 1 Executive Summary ASAM Test Specification Study Group Report 2022 5 Foreword A main focus of collaborative efforts across the automotive industry needs to be testing strategies and

10、 how to adapt them.Software-defined vehicles and autonomous driving functions are continuously and drastically increasing the complexity of testing.To tackle this challenge,a network of experts across all branches of the automotive industry have joined forces under the umbrella of ASAM e.V.The ASAM

11、Test Specification Study Group created a comprehensive overview of the testing and standardization landscape and developed a test methodology blueprint that embraces the multi-pillar approach needed to prove the safety of autonomous driving functions.A potential basis for future standardization.The

12、results are a statement,a recommendation,and a definite invitation to shape this future together.Table of Contents ASAM Test Specification Study Group Report 2022 6 1 1 Executive Summary Executive Summary Although sometimes it may feel like we have a frustratingly limited number of ways to steer a v

13、ehicle through traffic,the reality is that driving is tremendously complex,with an infinite number of possible scenarios particularly when it comes to testing requirements.One question the current developments will soon pose is:who exactly is behind the steering wheel?Today,we see the adoption of SA

14、E Level 0 to Level 2 advanced driver assistance systems(ADAS)throughout the industry,including features such as Brake Assist,Lane Keeping Assist,Electronic Stability Control,Blind Spot Indication,or Parking Assist.While these features do not eliminate the role of the driver,the interaction with the

15、environment increasingly transfers from driver to vehicle.In other words,the transition from assisted driving to fully automated driving is on the horizon.Consequently,the automotive industry is undergoing the biggest changes in its history.This paradigm shift affects everyone and brings opportuniti

16、es and challenges to the industry.Levels of Autonomy and the Difficulty of Proving Safety The future of transportation promises to make life safer and more mobile for everyone,with positive economic results.But to realize that promise,it is first necessary to test new vehicles and every part of thei

17、r architecture,especially as the parts become smarter and highly complex.However,the increased complexity necessitates a radical change of test methods and new concepts for comprehensive vehicle validation in both the physical and the virtual world.Still,many players appear to have neglected to cons

18、ider the best path for developing and implementing suitable strategies.Without toolkits independent of manufacturers or domains,and without intensive exchange across all subsectors,highly autonomous driving simply cannot be realized.We are on the verge of ushering in a new era of transportation safe

19、ty through autonomous mobility,but this shift raises big questions.What constitutes state-of-the-art testing of driving functions in a rapidly evolving environment?How will the automotive industry a)consider best practices for testing and safe deployment while b)at the same time integrating new tech

20、nologies like Artificial Intelligence and c)complying with regulations that are not yet developed?In February 2021,the UNECE(United Nations Economic Commission for Europe)presented the New Assessment/Test Method for Automated Driving(NATM)a framework and potential game changer.Its multi-pillar appro

21、ach considers the topics high level of complexity and poses new opportunities for more comprehensive requirement sets as a focus of technical services.However,these still need to be created and standardized.Why the Joint Effort?To expedite this process,a transdisciplinary network of experts was form

22、ed,the ASAM Test Specification Study Group,including representatives of technical services,software providers,manufacturers and OEMS,engineering and test data management professionals.Under the banner of ASAM e.V.experts have collaborated for more than twenty years in order to create standards for t

23、he development and testing of automotive electrical/electronic systems.They succeeded in reducing costs and efforts at OEMs and Tier-1 companies for the development,maintenance,and administration of development tool chains.Currently the ASAM portfolio Table of Contents ASAM Test Specification Study

24、Group Report 2022 7 comprises more than 30 trusted standards that are applied in automotive development across the globe.While this means that many pieces of the puzzle when it comes to standardized scenario-based testing workflows are being tackled,the overall interplay between those pieces had not

25、 been examined in such close detail.The following report documents the results of this new joint effort and aims to define a valid basis for follow-up activities and projects.Main goals of the ASAM Test Specification Study Group project:Provide overview of test methods in the field of ADAS/AD Develo

26、p a potential basis for future testing,the Test Strategy Blueprint Detailed use cases for the implementation of a test strategy Alignment with current standardizations Provide recommendations for stakeholders and proposals for further standardization activities A New Strategy Blueprint Based on the

27、experience of the many experts working together during the course of the ASAM Test Specification Study Group project,a blueprint to meet the challenges of testing has been developed.Test strategies are becoming more complex,but this blueprint makes it possible to set a sensible starting point.A holi

28、stic best practice that can be tailored according to the specific requirements of manufacturing and other projects,but one that meets regulatory,legal,and technical requirements.This blueprint is to be understood as an invitation.An invitation to use,develop,and critically engage with.An opportunity

29、 to work together on the challenges of an increasingly complex automotive industry.The implementation of new testing strategies is an intricate task.Therefore,we defined concrete workflows and identified the need for a uniform test data management.In addition,we provide initial solutions so that use

30、rs are not left alone.Despite not being able to analyze and highlight all aspects,we were able to define concrete recommendations for action and follow-up activities.Current test strategies,which are often heterogeneous and have grown over several vehicle generations,must be viewed holistically and

31、also used during vehicle approval.The various test procedures and test environments highlighted in the blueprint can also be clearly anchored in the data-driven process.It is important to understand that the phases are no longer strictly separated from one another,but that transitions are smooth and

32、 continuous.Iterations and changes between the phases occur constantly and merge into one another.In the following chart you can see a possible and reasonable combination to fulfill test coverage for a software-centric and(partially)autonomous vehicle,which is sufficient for release and homologation

33、.Table of Contents ASAM Test Specification Study Group Report 2022 8 The derived blueprint,exemplifying test coverage for a software-centric vehicle The Vision of Seamless Exchange The following chapters and sub-sections elaborate on the current standardization landscape and possible future developm

34、ents,describe relevant use cases and how the new test strategy blueprint can be used.The report aims to provide overview and to additionally clarify the role of data-driven development and the importance of test data management.The final chapter summarizes the study groups findings,deducing recommen

35、dations for further action.Competition stimulates progress.If a specific technology is not effective for its users,however,competition loses its benefits.It leads to incompatible systems rather than better products.That is one reason why ASAM pursues a vision of the free interconnection of developme

36、nt process chain tools and the seamless exchange of data.The combined efforts of the ASM Test Specification Study Group bear witness to the fertility of this concept.Table of Contents ASAM Test Specification Study Group Report 2022 9 2 2 Scope Scope Mapping a Changing Landscape When attempting to ma

37、p the standardization landscape in its entirety,the first step is to define what is meant by“entirety.”Identifying the boundaries is an important part of the process,before examining the relevance of all aspects to achieve a comprehensive overview.The study group decided on a sequential project appr

38、oach,a phase and milestone plan with intermediate results that build on each other.Methods and use cases were clustered to facilitate structure and exchange in subworkgroups.The report examines the relevant test techniques and use cases for testing and homologation of the ADAS/AD Domain in the autom

39、otive industry with a special focus on software-centric cars with modern E/E architectures.The goal is to identify relevant standards,potential workflows,and their variants,and the overall interplay between these parts to form a cohesive whole.This analysis leads to a documented set of overarching u

40、se cases for testing and homologation,a set of potential workflows implementing them,and an overview of relevant users,standards,and their application.Additionally,the study group identifies gaps in the workflows,leading to the identification of potentially necessary additions to existing standards,

41、liaisons between standards,or completely new standards.Recommendations for these standards or additions are collected and documented.The goal is to define a valid basis for follow-up activities and projects.As the safety arguments and homologation of modern vehicles are the key aspects,security-spec

42、ific aspects such as fuzzing,pen testing,and TARA-driven testing are not currently receiving much attention.In order to cover a large part of the automotive test landscape in the ADAS/AD domain,the group pursued a broad scope when considering possible test techniques and test environments.The goal i

43、s to prioritize these test techniques and to map them to user journeys and workflows in order to map their relevance(and also for standardization).It is important to us that we cover large parts of the test map,even though we will focus on certain aspects as we go along.For follow-up projects,we bel

44、ieve it is necessary to provide as complete an overview as possible.From our point of view,the test procedures presented represent a large part of the relevant test procedures,which,through combinations of test environments,enable us to provide an almost complete representation of the relevant tests

45、 from testing during development to homologation.Table of Contents ASAM Test Specification Study Group Report 2022 10 3 3 Terms and DefinitionsTerms and Definitions An Outline of Crucial Concepts Based on the following definitions,the study group were able to derive a structure and find an adequate

46、level of abstraction to measure up to the complexity of the topic as well as to the applicable compilation of a set of methods,use cases and test aspects.3.13.1 Test MethodTest Method The term“test method”is not defined in any of the common standards like the ISTQB glossary or ISO 29119,but there ar

47、e some hints how the term could be used.The American Society for Testing and Materials defines“test method”as:“a method for a test in science or engineering,such as a physical test,chemical test,or statistical test.It is a definitive procedure that produces a test result.In order to ensure accurate

48、and relevant test results,a test method should be explicit,unambiguous,and experimentally feasible,as well as effective and reproducible.”Similarities to other standards like software methodology are object-oriented,sequential,data-driven,and agile software development methods.In the context of high

49、ly automated driving we speak also about the PEGASUS Method for Assessment of Highly Automated Driving Function(HAD-F)propagated from the PEGASUS project(https:/www.pegasusprojekt.de/en/).A test method is any procedure that fulfils test goals and defines,for example,applicable test techniques or pra

50、ctices as a part of this specific method.Following this idea,requirements-based testing,for example,is a test practice,but not a test method in itself.There are many methods to conduct requirements-based tests to obtain test results and fulfil test goals.A test method may consist of Choice of test p

51、ractices(requirements-based testing,scenario-based testing,model-based testing)Test level(system test level,integration test level)Test environments(HIL,SIL,MIL,VIL,proving ground,real road testing)Test techniques(equivalence partitioning,state-based testing,failure injection test(?)Applicable proce

52、dures/processes(postprocessing,risk assessment)Choice of coverage criteria So,a test method represents a partial implementation of the test strategy for an aspect of the test item to be tested.Table of Contents ASAM Test Specification Study Group Report 2022 11 3.23.2 Test CaseTest Case Test cases a

53、re one of the most important building blocks of automotive testing and are used in all different kinds of testing methodologies.A test case defines an autonomous unit that describes one particular test that can be specified,executed,and assessed.In the following,we will present three different but w

54、ell-used definitions of“test case”and compare them.We will not choose the right definition out of these terms,since they are quite similar and the exact choice depends on usage.Using the current definition of the International Software Testing Qualification Board(ISTQB),a test case can be defined as

55、:“A set of preconditions,inputs,actions(where applicable),expected results,and postconditions,developed based on test conditions.”The current ISO standard(ISO 29119)omits the postconditions(which are optional and could be results of the test)and defines a test case as:“A set of test case preconditio

56、ns,inputs(including actions,where applicable),and expected results,developed to drive the execution of a test item to meet test objectives,including correct implementation,error identification,checking quality,and other valued information.”This can be shortened to:“A set of preconditions,inputs,and

57、expected results,developed to drive the execution of a test item to meet test objectives.”The previously mentioned actions are part of the general inputs of the test case.The latter definition is a very probable candidate for being used in the new ISO standard.As stated in the second paragraph,these

58、 three definitions are quite similar,with only the necessity of stating postconditions and actions as part of a test case being debatable and without effect on the usage of the term in the rest of this document.If a scenario-based test approach is used test cases may reference scenarios for environm

59、ent simulation.Table of Contents ASAM Test Specification Study Group Report 2022 12 Referenced Standards:ISTQB ISO/IEC/IEEE 29119 Table of Contents ASAM Test Specification Study Group Report 2022 13 3.33.3 ScenarioScenario A scenario describes how the world or scene changes with time,usually from a

60、specific perspective.In the context of vehicles and driving,this will encompass the developing view of both world-fixed elements such as the road layout and static furniture,world-changing elements such as weather and lighting,and dynamically moving elements such as other vehicles,objects,people,or

61、animals.In addition,a scenario may contain scenario validity criteria.At the time of writing,the group were made aware of a new definition for a test scenario defined in ISO 34501.This may have an impact on future revisions of the workflows defined herein,but it was not considered for this revision.

62、Table of Contents ASAM Test Specification Study Group Report 2022 14 3.43.4 TestTest SpecificationSpecification Test case specifications are important documentation types in automotive testing across all different kinds of testing methodologies(from requirements-based to interface testing,fault inje

63、ction,resource usage,and so on)and have a direct impact on the testing process.The test specification describes everything that is required for the test execution.This includes both the test environment and the sequence that describes the test execution.As part of the testing process,several steps a

64、re carried out and include the following activities:Test planning Test monitoring and control Test analysis and test design Test implementation Test execution Test completion So,the test specification is the complete documentation of test design and test implementation and includes test cases and ot

65、her necessary information to run those test cases.As part of the start-up phase,the preparations,follow-up work,and actions for“cleaning up”are defined.In the context of test design,the focus is on identifying and defining the required test cases.The aim is to define what exactly we want to test and

66、 to recognize defined scenarios.As a result,we get a wrapper for the test cases as the sequence to be executed.For this purpose the generic structure of the test specification is broken down into three components.1.The first of them is the test model specification or test design specification,which

67、describe the target of testing and the model used by the tester to derive test cases:a.Traditional feature sets b.Test coverage items c.Test models 2.The test case specification,which describes the test cases 3.The test procedure specification,which describes how test cases are grouped with addition

68、al information necessary to perform testing The ISTQB standard defines the test specification as a document that consists of a test design specification,test case specification(describing one or more test cases),and/or test procedure specification.Table of Contents ASAM Test Specification Study Grou

69、p Report 2022 15 Referenced standards:ISTQB ISO/IEC/IEEE 29119 Retired IEEE 829 3.53.5 The The Relation between Test CaseRelation between Test Case and Scenarioand Scenario Test case specifications and scenario descriptions are important document types in automotive testing across all different kind

70、s of testing methodologies(from requirements based to interfaces testing,fault injection,resource usage,and so on).For example,in requirements-based testing methodology,the use of test case descriptions has proven successful in specifying requirements,in test bench configuration,and in defining test

71、 steps and expected result of test runs.As another example,scenarios have become increasingly important in the context of scenario-based testing using simulation(also known as virtual testing)and other relevant test environments for the homologation of ADAS/AD(reference to the test matrix in the rep

72、ort).The scenario-based testing methodology for ADS is evolving,and different standards and regulations are being formed with respect to it.Virtual testing environments based on scenario specifications have been demonstrated by few AV-focused companies.In scenario-based testing,one or more scenarios

73、 are referenced from a test case specification.Scenarios contain the behavioral specifications for vehicles,pedestrians,etc.acting in the(virtual)world.Scenarios may include scenario validity criteria that may impact the test results.At the end,there is a wide spectrum of potential mutual relationsh

74、ips between the test and the scenarios.Below are several examples of such relationships:The test may affect scenario parametrization by posing additional constraints on it The scenario may result in events that are used by the test And many others.There is always a trade-off between the amount of in

75、formation flowing back and forth between the test and the scenario and the reusability of different scenarios across different tests.That is,independent of the choice of scenario language and test language,the ability to exchange information and reuse tests and scenarios should be allowed as much as

76、 possible.This leads to a need for a standardized interface between the test and scenario that clearly specifies the interaction rules.Table of Contents ASAM Test Specification Study Group Report 2022 16 Standardized Interface Due to the heterogeneous architecture of both test instances and simulati

77、on environments in the industrial environment,both document types must be formulated generically with respect to the underlying hardware and software architecture.This is necessary to enable the consistent use of test cases and scenarios across different test instances and test environments througho

78、ut the development process and within homologation processes.Table of Contents ASAM Test Specification Study Group Report 2022 17 In scenario-based testing,one or more scenarios are referenced from a test case specification.This requires an interface within the test case description to implement hig

79、her-level concepts(e.g.parametrization)consistently in tests and scenarios.To enable the use of different scenario description languages(from the ASAM Open Standards),this interface must be designed in an abstract and modular manner.Standardizing the interface simplifies the synchronization of the d

80、evelopments and reduces integration risks.Minimum requirements need to be defined regarding what should be handled between test case and scenario.Reference to scenarios(from within test cases)Parametrization and metrics of scenarios and test cases(e.g.changing the initial velocity of a vehicle)shall

81、 be consistent Table of Contents ASAM Test Specification Study Group Report 2022 18 Specification of interfaces for runtime interaction within test cases with scenarios(access to language elements,e.g.events and success criteria)with environment simulation and the test environment(interface examples

82、:see image tracetronic)e.g.evaluation and modification of numeric values during test execution(model parameters,ECU values,bus data,etc.)Interfaces should be bidirectional Table of Contents ASAM Test Specification Study Group Report 2022 19 4 Automotive Industry Insights The Status Quo of Test Strat

83、egies and Homologation Developments Due to the high complexity of the driving task in a so-called open world,requirement-and specification-driven approaches tend to have limits.Up until now,the set of methods in requirement-based engineering were not able to keep up with this increasing complexity.T

84、o address this issue,the term“safety of the intended function”(SOTIF)has been created.In order to recover the reduced capabilities of the left side of the V-model,the activities in the verification and validation branch need to relate to each other in order to provide a solid foundation for the safe

85、ty argumentation.Table of Contents ASAM Test Specification Study Group Report 2022 20 4.14.1 Homologation and the Transforming RoleHomologation and the Transforming Role of Technical Services of Technical Services Out of the lab onto the road?The New Assessment/Test Method for Automated Driving(NATM

86、)published in February 2021 provides a framework composed of a scenarios catalog and five validation methodologies to which this report refers,underlining the fact that the homologation process will develop further over time.While future strategies and type approval for new vehicles and ADAS functio

87、ns will vary,the common denominator is certain:today and tomorrow the assessment of an ADS and its ability to demonstrate safe behavior when operating in the real world needs to combine three pillars;real-world testing,physical certification tests,and extensive auditing.The high complexity of the dr

88、iving task and the quest for the best approach require a technical inspection to go beyond a static set of test cases.Additionally,the complexity of the automation systems in terms of architecture and components require checks beyond a static set of technical checklist items.To complement well-estab

89、lished static test cases,checks of procedures and fulfillment of objectives are needed(e.g.through audits).4.1.1 Testing and type approval current and future regulations The aforementioned need to relate individual test results to the remaining tests and results finds its expression in the“multi-pil

90、lar”approach put forward by GRVA(UNECE Working Party on Automated/Autonomous and Connected Vehicles)and the effort of the New Assessment/Test Method for Automated Driving(NATM)at the UNECE.Thus,in the latest and in upcoming regulations,one can expect a more comprehensive set of requirements as compa

91、red to the past.At this point in time,the following items are discussed:Scenario catalog Simulation/virtual testing Track testing Real-world testing Audit/assessment In-service monitoring and reporting Note that there are topics that can only be addressed by combinations of the above items.Some exam

92、ples are:Trustworthiness(simulation/virtual testing+track testing+real-world testing)Qualification of tools(audit/assessment+testing+in-service monitoring)Coverage of hazardous scenarios(audit/assessment+scenario catalog)Table of Contents ASAM Test Specification Study Group Report 2022 21 Sufficient

93、 exploration of unknown hazardous scenarios(real-world testing+in-service monitoring+scenario catalog)It is these objectives that are in the focus of inspections by technical service providers.4.1.2 Possible impact of the study group and defined test strategy blueprint 1.The blueprint of the strateg

94、y can be used to argue residual risk estimations,performed scenarios,and considered weaknesses and mitigations The aforementioned objective to achieve coverage of all hazardous scenarios is obviously rooted in the need to estimate the residual risk and to evaluate whether the achieved values can be

95、accepted.A test strategy blueprint that links hazard analyzes and exploration of scenarios with the verification of analysis results and effectivity of implemented risk mitigation measures is one key element to achieving the goal of an evaluation of residual risks.2.A testing strategy blueprint defi

96、ned by the study group can serve as basis for an assessment of OEM-/Tier-1-specific test strategies As outlined above,the verification and validation challenge for automated driving is also rooted in the fact that different tests,which make use of different methods and Test Environments,are interdep

97、endent:the full strength of the verification and validation evidence and argument can only be unfolded if the various results make reference to each other.A testing strategy blueprint might show the most relevant types of interrelation.It might thus serve as a basis for setting up a verification and

98、 validation strategy at OEM or Tier-1,as well as a basis for assessments and release.3.The different methods in the test strategy can be reviewed in different ways:document review,witness testing,or retest at technical service The assessment of the test activities and test results produced in terms

99、of agreement with the verification and validation strategy,as well as requirements from standards and regulations,will probably require several approaches.These approaches(very much similar to the test methods themselves)have to trade off effort and depth of investigation.On the low-effort end,docum

100、ent reviews might be used to confirm that,for example,passed test results are truly confirming safety and failed tests have been properly fed back into the development cycle.For deeper insight into the tests performed,witnessed tests are especially valuable in case very specific and specialized test

101、 equipment is required.To achieve fully independent confirmation of test results on the high-effort end,a fully independent retest performed by a technical service might complete the range of confirmation of the test activities.Table of Contents ASAM Test Specification Study Group Report 2022 22 4.1

102、.3 Examples with special relevance for type approval Scenario-Based Approval of ALKS(Automated Lane Keeping Systems)using X-in-the-Loop Scenario-based test approaches have several features that make them interesting for assessment,release,and type approval of automated driving functions.The first as

103、pect to mention here is the analogies to methods used for risk evaluation.These analogies allow the use of in-the-loop methods to confirm cases identified during analysis.This might include aspects like severity ratings or input parameters for controllability considerations,as well as identification

104、 of edge cases.Secondly,scenario-based in-the-loop methods might be used to build a positive risk argument along the lines of UNECE R157(ALKS)Annex 4,Appendix 3,by comparing the system of interest with an(existing)benchmark.Such a benchmark might also be a driver model as suggested in UNECE R157.Thi

105、rdly,to confirm the safety of the intended function(SOTIF from ISO/DIS 21448 or“operational safety”in UNECE sources),it will be necessary to argue the absence of unknown hazardous scenarios(refer to“area 3”from ISO/DIS 21448).This endeavor is sometimes also referred to as the exploration of the unkn

106、own.In case this exploration requires a parameter variation,in-the-loop techniques should complement real-world observations.SW/HW Reprocessing/Data-Replay for Perception Systems With the currently available methods and experiences,the argument to control risk cannot end after product release.Instea

107、d,the final evidence for acceptable risk(e.g.,in terms of a positive risk balance)can only be provided in the field.Thus,field observation has become an essential part of the UNECE regulations.Due to limitations in connectivity and in-vehicle data storage capacity,testing capabilities to reproduce a

108、dverse interactions in real-world application are required to operate an automated vehicle fleet safely.Reprocessing of recorded(or reconstructed)sensor data might allow a reconstruction,analysis,and future mitigation of such adverse effects.VIL and Proving Ground Previously,the outstanding role of

109、in-the-loop and reprocessing techniques has been discussed.However,the key element that makes these methods efficient namely partial modeling(and thus simplification)of the real world is also a point of weakness once it comes to systematic deviations.To keep these potential systematic issues under c

110、ontrol,a close alignment with other test results that are closer to the real-world application of automated vehicles on public roads is required.This means vehicle-in-the-loop and proving ground tests are key to model validation in other in-the-loop approaches.Beyond this,proving ground tests are in

111、tegrated in the UNECE multi-pillar approach as a mandatory part,requiring the accomplishment of a minimum set of challenges.Table of Contents ASAM Test Specification Study Group Report 2022 23 4.1.4 The trustworthiness of test environments ISO 26262 Tool Qualification As an automation system has con

112、trol over the major actuations of the vehicle longitudinal and lateral control its decisions are safety critical.Thus,automation systems are subject to ISO 26262.As ISO 26262 affects requirements for testing and the use of SW tools that are applied to cover objectives from the standard.Therefore,all

113、 parts of the test tool chain used to verify or validate safety-critical decisions will be subject to tool qualification.This means the tool use cases need to be specified and compliance with these specifications has to be shown in agreement with the provisions in ISO 26262.Model Validation Similar

114、to the in-vehicle automation function that has functional safety and SOTIF aspects,models used for verification and validations also need to be executed with integrity and need to provide a certain minimum performance in terms of agreement with the real world(e.g.ISO 11010).In the previous paragraph

115、 on proving ground tests,it was already outlined that there is the need to support the lower-level in-the-loop tests in terms of validation of the deployed models.By nature,a model is a simplification of the real world.The trick with modeling is thus to remove complexity that is not relevant for the

116、 process of interest,keeping the relevant parts sufficiently realistic.It needs to be confirmed whether this abstraction process has been successful.This confirmation is typically done by validating the models.This means the model results are confronted with results obtained in real-world experiment

117、s.Only with such input on the trustworthiness of the tool chain can the results be valued correctly during homologation.Test Environment Adequacy and Fit-for-Purpose Checks Verification and validation strategies have a full product life cycle impact.Thus,tool environments have to face requirements r

118、elated to reliability,long-term support,and reproducibility of results.This can only be accomplished if the test environments are used based on a solid quality management system and the recurrent prove of suitability(hardware calibrations).Risk Assessments,Expert Knowledge,Systems Designs It has bee

119、n discussed above that testing for automated driving is strongly interrelated with other development activities.This implies that a test strategy blueprint cannot be a static,stand-alone document.Instead,it needs to adapt to other sources.Such sources are risk assessment methods to be deployed,avail

120、able expert knowledge on the particular automation system,and also the choice of system architecture and design.Table of Contents ASAM Test Specification Study Group Report 2022 24 4.24.2 DataData-driven Development and Shifting Responsibilitiesdriven Development and Shifting Responsibilities How(on

121、ly)a collaborative approach will enable autonomous driving Just as software is the key to most of todays automotive breakthroughs,data-driven development is key to establishing new automated driving functions.However,with Big Data,shifting paradigms,and AI,how will players in the automotive industry

122、 ensure the safety of functions and the future development of autonomous driving(AD)?A close examination of the situation shows the necessity of the early integration of best practices and comprehensive strategies into the process a matter of intensive exchange,investment,and strong partnerships.4.2

123、.1 How does automated driving change the electrical/electronic(E/E)development process?The standard approach to electrical/electronic development works with automated driving but only to a limited extent,as the event chains of the driving function are clearly becoming more complex and contain elabor

124、ate environmental sensors.Newly established organizations are function-oriented,rather than component-oriented.That means that responsibilities shift,and not just within individual organizations.It also changes the relationship between manufacturers and suppliers,which then,of course,changes develop

125、ment processes and test strategies.At the same time,priorities are shifting as well.Where software algorithms and electronics were previously the sole focus,now AI comes into play.That results in changed process models and the emergence of new roles in data processing.Data train AI,and data are cruc

126、ial for verifying and validating AI.That requires intensive investments in AI infrastructure in order to develop the right solutions and to support the associated data processes.This encompasses the processing and handling of data,from the acquisition of data logging in vehicles to data enrichment a

127、nd big data management.Due to the high complexity of systems and their environments the importance of data reaches a new level.Established methods of model development reach their limits when they encounter data-driven development:AI-learned behavior cannot be modeled directly.A large amount of info

128、rmation must be made available in a systematic way before a sufficiently accurate model can be synthesized.Thus,AI know-how must be connected with the know-how of electrical/electronic development in the automotive sector.While the introduction of AI is accepted,it cannot lead to weakened competence

129、 in development or testing of automotive industry.This is why data-driven development requires a network of partners with an overall view of the process landscape a partnership that acts independently of domain and thus with a focus on overall development to support the advancement of holistic appro

130、aches.Table of Contents ASAM Test Specification Study Group Report 2022 25 Where previously closed-loop control(prototyping)and validation(closed-loop XIL)were used,data-driven development is enforcing a paradigm shift:data are also being used to stimulate interfaces and subsequent evaluation.This c

131、alls for test systems that reproduce recorded and enriched data from test drives or simulations to validate AI algorithms and AI-based ECUs,purely simulatively,based on software or AI algorithms,or directly with the ECU hardware under strict,real-time requirements.Ultimately,testing functional and s

132、tructural requirements using synthesized and enriched data leads to new challenges in validation.New technologies and methods must be integrated into a holistic process to make this data processing as safe as possible and efficiently connected to all the integration steps along the electrical/electr

133、onic development process.This covers everything from data process to AI training to overall integration and is based on continuously repeated processes that achieve system maturity with increasing amounts of data.Providers of development solutions must,therefore,be involved in the development proces

134、s much earlier.They will take on the role of development partner to integrate new methods into established processes and to define testing goals and achieve them as efficiently as possible.Table of Contents ASAM Test Specification Study Group Report 2022 26 4.2.2 How does automated driving change th

135、e release and homologation process?Parallel to this shift to data-driven development,as we have established here,the release and homologation of automated driving is also changing.In the past,homologation was the last step during or after development.Random samples were made on vehicles to test over

136、all system behavior.Given the complexity of new systems,this is no longer sufficient.The residual risk of such a highly automated system cannot be determined at the overall system level.In the future,the development process and the(mostly simulative)methods used there will be the basis for the homol

137、ogation of automated driving functions.The development of new data-driven development processes should thus also aim to achieve homologation as efficiently and safely as possible,and to fulfill compliance requirements.This means by implication that the requirements of the relevant current and future

138、 safety standards must be met.As such,automated-driving releases will be based on two pillars,largely implemented by simulation:firstly,verification,in which all measures to prevent known risks are checked,and secondly,validation,in which the system is pushed to its limits in relevant use cases to i

139、dentify unknown risks.Both pillars combine simulation with vehicle testing.Through combining the available methods of HIL and SIL safeguards,a statement can be made about overall performance and residual risk.The methods will range from classic HIL tests to highly scalable cloud-based simulation sol

140、utions that can check an almost infinite number of possible combinations of scenarios.Tool manufacturers must make their products cloud-capable and more flexible than ever before to be able to address the higher number of iterations between verification and validation and the system design.Offerings

141、 such as SaaS(safety as a service)might play a role in the future.The changes and transformations have led as never before to the need to standardize and further explore fundamentals.The complexity arising from software-centric vehicles,the challenges of autonomous driving,and data-driven developmen

142、t can only be managed if standards are applied wherever possible.To make this visible and to put current standards and current research into this context,this needs to be analyzed globally.Table of Contents ASAM Test Specification Study Group Report 2022 27 4.34.3 Existing Standards and Current Rese

143、arch ActivitiesExisting Standards and Current Research Activities A 360-degree angle to drive alignment Analyzing the direction of future test strategies,we need to consider the following:the standardization landscape is not a one-way street.It requires comparing existing standards and comprehensive

144、ly identifying interrelations,gaps,and phases of development.This report aims to offer insight into the effect of new strategies and current activities,and to ask how the study groups results might affect ongoing efforts.The overall goal:to compile all relevant information and develop a clear-sighte

145、d blueprint for testing strategies,adopt advance standardization in the ADAS/AD domain,and provide a common pool of knowledge.ASAM aims to use existing standards where possible and if necessary and feasible to adapt them to the new requirements.Within this study,several of ASAMs industry-proven stan

146、dards,as well as important standards from other organizations,shall be evaluated for their usefulness for the validation of autonomous vehicles.To structure the analysis of individual standards in the next sub-chapters,this report applies the division of standards as follows:Table of Contents ASAM T

147、est Specification Study Group Report 2022 28 4.3.1 Infrastructure Representation These are standards that are used to describe some or all aspects of the simulation or test specific infrastructure.4.3.1.1 ASAM OpenODD Description In order to establish the true capabilities and limitations of automat

148、ed driving systems(ADSs),we need to first define their operational design domain(ODD).ODD refers to the operating environment(road type,weather conditions,traffic conditions)in which a vehicle can drive safely.For example,for low-speed automated driving(LSAD)systems such as pods and shuttles,the ODD

149、 may include urban areas with predefined routes that include pedestrians and cyclists.On the other hand,for a highway chauffeur system,an ODD may include a four-lane divided freeway and dry conditions only.The types of scenarios a vehicle may encounter will be a function of its defined ODD,making th

150、is fundamental to any safety evaluation and scenario identification.A more formal definition of ODD as defined by SAE J3016(2018)is as follows:“Operating conditions under which a given driving automation system or feature thereof is specifically designed to function,including,but not limited to,envi

151、ronmental,geographical,and time-of-day restrictions,and/or the requisite presence or absence of certain traffic or roadway characteristics.”In order to enable stakeholders to share,compare,and reuse ODD definitions,there is a need for standards to provide guidance to the stakeholders on both the att

152、ributes to be used for the definition of ODD and a format for defining the ODD using those attributes.BSI PAS 1883(UK)provides a taxonomy for ODD.Additionally,ISO 34503 uses the taxonomy to provide a high-level definition format for ODD.While these standardization activities address the important ne

153、eds of the industry,a gap still exists in the industry for a definition of ODD adapted for simulation.ASAM OpenODD is a representation of the abstract ODD specification in a more well-defined syntax and semantics which enables machines to interpret and perform the required analysis.Additionally,the

154、ASAM OpenODD specification shall be measurable and verifiable for the attributes it specifies.ASAM OpenODD focuses on a machine-readable format that is:Searchable Exchangeable Extensible Machine-readable Measurable and verifiable Table of Contents ASAM Test Specification Study Group Report 2022 29 H

155、uman-readable Current role and relevance with regards to ADAS/AD In the scenario-based testing workflow ASAM OpenODD will play a very important role supporting the test description,defining the boundaries of what to test,and achieving a good test coverage of the operational design domain and its bor

156、ders.It allows for the statistical quantification of:1.exposure of certain attribute values of the ODD,2.the performance of a CAV and its systems against the ODD,3.the tightening or expansion of the ODD over time.Status The concept paper was released in November 2021,with a follow-up standardization

157、 project starting in March 2022.View online Potential implications of the Operational Design Domain and associated testing standards The study group expects the need to quantify and use ODDs for testing in ADAS/AD to grow significantly over the coming decade,with the need for common ground through,f

158、or example,standards,growing proportionately.The operational design domain is a term that has emerged recently out of the need to more stringently restrict or more clearly delineate the design domain for ADAS/AD functionality.In the past,formal ODDs played more of a minor role in testing,particularl

159、y outside the ADAS/AD domain.Their usage was limited to natural-language formulations for on-road testing,with verification or validation against these being limited to the undefined format.Table of Contents ASAM Test Specification Study Group Report 2022 30 The growing need to be able to specify op

160、erational design domain limits in a clear,exchangeable manner is a critical factor in the deployment of AD systems,whose ODDs will presumably increase as the industry advances.Being able to state these limits is,of course,just one facet.As of the authoring date of this report there is but one regula

161、tion for a Level 3+AD system(UNECE 157 ALKS).As regulatory bodies continue making inroads into defining regulations for AD systems and the automotive industry with the development and testing of AD systems,the demand for structured methods and processes to verify against such specifications will soa

162、r.This is directly supported by new and upcoming standards such as OpenODD,which provides an underlying exchange format,as well as taxonomy-driven standards like BSI PAS 1883,ISO 34503,or ASAM OpenXOntology.ODDs will be integrated into a majority of the steps across the testing workflow,from the ini

163、tial system design and risk assessments to the test specification,where the expected operating limits of a function might be described by an abstract ODD,all the way to being the accompanying artifact for the test case and scenario definition process,with ever-increasing levels of concreteness in th

164、e ODD.Validation steps,such as checking scenarios and results against an ODD,will become an integral part of a testing workflow.Test results must begin including ODD criteria,including KPIs to measure degradation of the ODD.Exactly how ODDs will be integrated into tests remains to be determined and

165、will become clearer as the infrastructure and standards surrounding their definition evolves.Table of Contents ASAM Test Specification Study Group Report 2022 31 4.3.1.2 ASAM OpenSCENARIO Description OpenSCENARIO defines the dynamic content of the world,for example the behavior of traffic participan

166、ts and how they are expected to interact with each other and the environment.It is used in:Virtual development Tests Validation of functions for:Driver assistance Automated driving Autonomous driving Testing on test tracks or proving grounds Testing in a mixed environment(HIL)Decoding real-world dri

167、ving data The OpenSCENARIO standard occupies a unique position amongst the OpenX set of standards in that ASAM currently actively develops two parallel versions of it.The two versions occupy different positions in the application toolchain.V1.0 is a low-level and concrete specification format,primar

168、ily designed to be read by simulation tools,whereas V2.0.0 allows those users that create maneuver descriptions and tests to define scenarios at a higher level of abstraction as well as providing alternative expression methods to the current XML format of OpenSCENARIO 1.0.0 via a domain-specific lan

169、guage(DSL).OpenSCENARIO 2 defines an abstract road syntax that allows for the description of road geometry at varying levels of abstraction.Version 1.1 does not support the description of such static content;instead,it must be referenced through,for example,OpenDRIVE descriptions.Status V1.1.0 relea

170、se date:March 2021 V1.2.0 release date:March 2022 V2.0.0 release date:December 2021 Current role and relevance with regards to ADAS/AD The complexity of the systems being tested for high levels of automation has(and will continue to)grown exponentially.Whilst the full implications of this are yet to

171、 be understood,these levels of complexity need to be reduced at some stage or other in the testing workflow.This can be done at the test specification level(potentially leading to more tests)or at the scenario level.This increasing complexity is directly reflected in the capabilities of the scenario

172、 description languages being developed.It is possible that this complexity can be abstracted out significantly better through powerful scenario descriptions that can cover large solution spaces through higher levels of abstraction,which also serve to support increased automation.Further tools to red

173、uce complexity at various points include the robust use of standards as well as ensuring a seamless Table of Contents ASAM Test Specification Study Group Report 2022 32 exchange of data.To better understand the target groups,the ASAM OpenSCENARIO 2.0.0 standard documents use cases and users for scen

174、ario descriptions in detail.These include various use cases specifically targeting test engineers and regulators(see the ASAM OpenSCENARIO V2.0.0 standard for details).Whilst only OpenSCENARIO 2 explicitly targets test engineers and regulators,both versions are deemed to be relevant to testing and t

175、his report.OpenSCENARIO 1 is currently the only standardized format for scenario descriptions and is hence a key part of current scenario-based testing workflows.This will likely become less so as OpenSCENARIO 2 is released and gains traction in the industry.This is expected to be driven in part by

176、the need to have higher levels of scenario abstraction,for example for regulatory purposes.It is hoped that a standardized format capable of a full spectrum of abstraction levels like OpenSCENARIO 2 will greatly support the use of,as well as the shared understanding of,scenarios in legislature.The r

177、oadmap at ASAM for OpenSCENARIO estimates a convergence of the two versions to one in some form within the next three years.Study group summary of ASAM OpenSCENARIO As detailed in section 3.5.“The Relation between Test Case and Scenario”,there is a very close interaction between a scenario and a tes

178、t,which necessitates an interface within the test case description to implement higher-level concepts(e.g.parametrization)consistently in tests and scenarios.This interface must be represented in the scenario as well.It is thus expected that this standard will be extended to support such an interfac

179、e.Additionally,a scenario needs to have some interface to an ODD description.Further possible implications include a clearer separation of test-relevant parameters and fields from generic scenario fields,but this is likely to be part of a separate testing-focused standard that can be applied with th

180、e language features of OpenSCENARIO.Possibly this could be implemented as a feature set of OpenSCENARIO that is testing-specific.Other such extensions could include bindings to external languages(in addition to the aforementioned testing interface)or testing libraries.OpenSCENARIO V1 has applied ext

181、ensive focus in its ongoing development to clarifying the interaction of a scenario description with a scenario engine.Such an engine is then responsible for interacting with a simulation.It is expected that such clarifications will need to be extended to both versions,particularly with an emphasis

182、on how one or more tests activate one or more scenarios.Table of Contents ASAM Test Specification Study Group Report 2022 33 4.3.1.3 ASAM OpenDRIVE Description The OpenDRIVE format provides a common base for describing road networks with Extensible Markup Language(XML)syntax,using the file extension

183、.xodr.The data that is stored in an OpenDRIVE file describes the geometry of roads,lanes,and objects,such as markings on the road,as well as features along the roads,for example signals.The road networks that are described in the OpenDRIVE file can either be synthetic or based on real data.The main

184、purpose of OpenDRIVE is to provide a road network description that can be fed into simulations for development and validation purposes of AD and ADAS features.Status The latest version of OpenDRIVE is V1.7.1,released in June 2021.Insights into how this standard might evolve Various use cases leverag

185、e road network descriptions without necessarily requiring a scenario.One example is traffic simulations with intelligent traffic agents,which interact directly with the road network without strictly needing a scenario.These use cases will still require some level of interaction between the road netw

186、ork description and the test case or the ODD characteristics.It is plausible to expect that such an interaction could still be realized using the interfaces being defined in a scenario description(see OpenSCENARIO recommendations),with an otherwise empty scenario(no dynamic content like maneuvers).H

187、owever,it is also possible that a direct need to extend OpenDRIVE with a similar interface to the test and the ODD will evolve.Table of Contents ASAM Test Specification Study Group Report 2022 34 4.3.2 Domain Representation&Taxonomies These are standards that are used to prescribe taxonomies and ont

188、ologies for the relevant domains.4.3.2.1 ASAM OpenLABEL Description OpenLABEL V1.0.0 standardizes the annotation format and labeling methods for objects and scenarios.The use of a standardized format will help save costs and resources in converting annotated data.OpenLABEL will be represented in a J

189、SON format and therefore can be easily parsed by tools and applications.OpenLABEL specifies which coordinate systems are being used as a reference for the label.OpenLABEL also provides methods to label objects in a scene(one point in time,frame)and over a period of scenes,enhancing this with the lab

190、eling of actions,intentions,and relations between objects.The figure shows the concepts related to data annotation for scenario-tagging.ASAM OpenLABEL covers the definition of the annotation schema for scenario-tagging and an ontology for tags.Scenario-tagging use cases focus on raw data that is use

191、d in the development,testing,and validation process of ADAS and AD functions,for example test scenarios or simulation scenarios.Often the format of such raw data is OpenSCENARIO,GEOscenario or other domain-specific languages or formats used to describe and store simulation and test scenarios.Status

192、The first version of this standard was released in November 2021.Table of Contents ASAM Test Specification Study Group Report 2022 35 4.3.2.2 ASAM OpenXOntology Description The OpenX standards describe traffic participants,road networks,driving maneuvers,and test scenarios for driving and traffic si

193、mulation,labeling sensor data,and other use cases.The standards share concepts from the domain of road traffic,such as roads,lanes,and traffic participants.OpenXOntology provides an ontology-based architecture for these concepts and thus a common definition of the domain model for the OpenX standard

194、s.OpenXOntology covers the following concepts from the domain of road traffic:Road infrastructure,meaning roads and lanes Traffic infrastructure,meaning traffic signs,signals,etc.Temporal changes in the road and traffic infrastructures,meaning road maintenance,diversions,etc.Dynamic traffic agents,s

195、uch as cars,pedestrians,and cyclists Environment,meaning weather,time of day,etc.Vehicle-to-everything(V2X)communication objects ASAM OpenXOntology aims to structure and formalize knowledge about the existence of things in the domain of road traffic.This is done as independently as possible of the i

196、ntended applications in ASAM standard development so that the ontology can be reused for different types of applications.The current application ontologies are focusing on ODD and scenarios.Nevertheless,the Study Group Test Specification sees the need to investigate the potential benefit of a testin

197、g application ontology.The complexity of creating and using domain-specific ontologies can be reduced by dividing knowledge into multiple areas via modularization.OpenXOntology is divided into the following modules:Core ontology basic concepts such as physical objects,temporal states Domain ontology

198、 central concepts of the road traffic domain Application ontologies application-or standard-specific content that is not relevant to the full domain Table of Contents ASAM Test Specification Study Group Report 2022 36 4.3.2.3 ISO/DIS 34501 Road Vehicles Terms and Definitions of Test Scenarios for Au

199、tomated Driving Systems Description The standard specifies terms and definitions of test scenarios for automated driving systems(ADSs).The contents are intended to be applied to Level 3 and higher ADSs defined in ISO/SAE 22736.The standard for terms and definitions is the basis for test scenarios fo

200、r automated driving systems,and the development of appropriate international standards will play a key role in supporting the testing,evaluation,and management of automated driving systems.This standard is used to harmonize and standardize the terminology and definition of test scenarios for automat

201、ed driving systems on a global scale.Status This standard is currently a Draft International Standard(DIS registered,stage 40.40)at ISO.4.3.2.4 ISO/AWI 34503 Road vehicles Taxonomy for Operational Design Domain for Automated Driving Systems Description This document specifies the basic requirements

202、for a hierarchical taxonomy for defining the operational design domain(ODD)for an ADS.The ODD includes static and dynamic attributes that can be used to develop test scenarios in which an ADS is designed to operate.This document also defines basic test procedures for attributes of the ODD.This docum

203、ent is applicable to automated driving systems of Level 3 and higher as defined in ISO/SAE 22736.ISO/SAE 22736 also defines the concept of ODD.The definition of ODD is fundamental in ensuring safe operation of the ADS as it defines the operating conditions of the ADS.While different kinds of ADSs ma

204、y be developed in the industry worldwide,there is a need to provide guidance on a framework for the definition of ODD for manufacturers,operators,end users,and regulators to ensure safe deployment of ADSs.This document will assist manufacturers of ADSs in the incorporation of minimum attributes for

205、the Table of Contents ASAM Test Specification Study Group Report 2022 37 definition of ODD and allow end users,operators,and regulators to reference a minimum set of attributes for the definition.Status This standard is currently in the Preparatory Stage(under development)at ISO.Table of Contents AS

206、AM Test Specification Study Group Report 2022 38 4.3.2.5 ISO/AWI 34504 Road Vehicles Scenario Attributes and Categorization Description This document defines packages concerning the different attributes of a scenario that convey information required to construct a complete test scenario.These attrib

207、utes may refer to complementary data sources,such as the related road network,3D scenes,and models.To ensure a certain quality standard,measures for the integrity and integrability of these attributes are defined.Additionally,this work item defines a categorization of the scenarios by providing tags

208、 that carry qualitative or quantitative information about the scenarios.Status This standard is currently in the preparatory stage(under development)at ISO.4.3.2.6 Potential Implications of Taxonomy-,Labeling-,and Ontology-Based Standards for Testing Amongst others,this subsection considers:OpenXOnt

209、ology OpenLABEL ISO 34501 ISO 34503 ISO 34504 An underlying requirement for collaboration and data exchange is a shared understanding of the terminology being used.Standardized taxonomies such as those mentioned above provide this baseline in the domain(or a subset of it).The overlap,differences,and

210、 relationships amongst the terms being used need to be clear for exchange to be successful.Worth emphasizing is that it is not necessary to have only one taxonomy as long as the overlaps and differences in the terms are clear.OpenXOntology currently does this,for example,by clearly mapping terms acr

211、oss the OpenX standards,rather than defining one single taxonomy.We also recommend the development of an application-specific layer to the OpenXOntology standard for testing that takes all of the conclusions made in this report into consideration.The natural-language-like reasoning enabled by an ont

212、ology could also be used to define another layer to scenario and test descriptions that is natural-language-like.This would aim to address,for example,the functional scenario layer when applied to scenario descriptions.Such a layer would enable machine-readable,natural-language-like descriptions in

213、legislature that can easily be converted to more concrete descriptions,thus being less subject to ambiguity or misinterpretation.The aforementioned taxonomies enable further steps in the testing-specific workflows.A standardized set of tags for scenarios,as defined in OpenLABEL for example,allows a

214、test to specify scenarios based on a specific label.An ontology like OpenXOntology allows for further automation of such a workflow,allowing for scenario selection based on similar or related tags.Table of Contents ASAM Test Specification Study Group Report 2022 39 This has similar implications for

215、the interaction between an ODD and a scenario database,where tags specified in the ODD can be used to provide the search parameters for matching scenarios.The OpenLABEL standard with its standardized format for labels for objects of interest(for sensors)is extremely beneficial to testing workflows t

216、hat leverage different test techniques.Recorded data from a test drive on an open road for example can be used to automate the specification of scenarios for scenario-based testing(see Section 10.2 of the ASAM OpenXOntology standard for more details on this use case).It can be expected that a standa

217、rdized set of labels generally eases coverage determination,allowing for the implementation of automata that speed up the process.Study group summary of this standard As initial standards and guidelines on terminology for ADAS/AD are published it is expected that significant alignment efforts will b

218、e needed to realize the goals and advantages described above.These efforts will need to take place in and amongst the standardization organizations to prevent adding to the confusion in the industry.It is recommended that a coordinated effort to align the different taxonomies is initiated initially

219、at ASAM internally and in second step across organizations.Table of Contents ASAM Test Specification Study Group Report 2022 40 4.3.3 Interface Definitions These are standards that are used to define interfaces or protocols,such as APIs or messaging formats between test relevant models or elements.4

220、.3.3.1.1.1.1.1.1 ASAM OSI(Open Simulation Interface)Description To achieve a widespread use of driving simulators for function developers,the connection between the function development framework and the simulation environment has to rely on generic interfaces.ASAM OSI provides easy and straightforw

221、ard compatibility between automated driving functions and the variety of driving simulation frameworks available.It allows users to connect any sensor,via a standardized interface,to any automated driving function and to any driving simulator tooling.It simplifies integration and thus significantly

222、strengthens the accessibility and usefulness of virtual testing.ASAM OSI(Open Simulation Interface)started as a generic data exchange interface compliant with the ISO 23150 logic interface for the environmental perception of automated driving functions in virtual scenarios.In tandem with packaging s

223、pecifications,such as the ASAM OSI Sensor Model Packaging(OSMP)specification,ASAM OSI provides solutions for simulation model data exchange across different implementations.ASAM OSI contains an object-based environment description using the message format of the protocol buffer library developed and

224、 maintained by Google.ASAM OSI defines top-level messages that are used to exchange data between separate models.Top-level messages define the GroundTruth interface,the SensorData interface,and,since ASAM OSI version 3.0.0,the SensorView/Sensor-View configuration interfaces and the FeatureData inter

225、face.The GroundTruth interface provides an exact view on the simulated objects in a global coordinate system,the ground truth world coordinate system.The FeatureData interface provides a list of simple features in the reference frame of the respective sensor of a vehicle for environmental perception

226、.It is generated from a GroundTruth message and may serve as input for a sensor model that simulates object detection or feature fusion of multiple sensors.The following figure shows the interfaces and models involved in modeling a sensor.Table of Contents ASAM Test Specification Study Group Report

227、2022 41 OSI also defines interfaces for traffic participant models.The TrafficCommand interface makes it possible to send commands to traffic participant models.The TrafficUpdate interface makes it possible to receive the updated state from traffic participant models.The following figure shows the i

228、nterfaces of a generic traffic participant.Traffic participant models may use other OSI interfaces internally,for example to model autonomous vehicles.The following figure shows a more advanced use case for traffic participants.Status The ongoing development activity at ASAM aims to further extend A

229、SAM OSI in order to fulfill the requirement of being able to use it as a standardized interface between further simulation entities.The latest version of OSI,version 3.4.0,was released in November 2021.Potential implications of OSI for testing As a standardized interface for models in a test,includi

230、ng sensor models and traffic participant models,OSI provides significant interchangeability and reusability of models across different test techniques such as HIL,MIL,or SIL.This becomes ever more relevant as multiple techniques are employed to satisfy a test goal.As touched on in the discussion aro

231、und OpenODD,the systems being tested are ever growing in complexity,and it is becoming increasingly important to reduce the complexity of tests for transparency.Maintaining the same model across tests provides one less variable to take into account.Table of Contents ASAM Test Specification Study Gro

232、up Report 2022 42 Further details on the interaction of models with a test are discussed in the section on the Functional Mockup Interface(FMI).The strong interplay between OSI and FMI needs to be taken into account through the development of such testing workflows.Insights into how this standard mi

233、ght evolve OSI is continually being developed to support further model types in a simulation or test environment,one example being as an interface to a vehicle-in-the-loop.It is possible that further test-specific or testing-technique-specific messages are implemented in OSI,whether as an extension

234、to existing messages to account for test-specific data or as additional interfaces,for example to an ODD or test layer.Performance is seen as a limiting factor for many applications of OSI.Prior to extending the current OSI standard further it is recommended that its relatively high performance over

235、head is improved.Similar considerations have been introduced in the current development project,which aims to investigate alternative,more performant formats to Protobuf.As it is also being developed as an interface between many of the features in the OpenX standards(for example between a scenario e

236、ngine and a traffic participant or to a traffic signal),any changes made to the other OpenX standards will need to be considered and integrated into OSI.One such change could be extending the TrafficCommand messages to support the exchange of test-specific information with traffic entities.Table of

237、Contents ASAM Test Specification Study Group Report 2022 43 4.3.3.2 ASAM XIL Description The ASAM XIL standard(see https:/ details)defines interfaces that test automation tools can use to connect to and communicate with test benches.Since the ASAM XIL standard supports not only HIL but also MIL and

238、SIL systems,it can be used in all phases of the development and test cycles.Test automation tools that are ASAM XIL compliant enable interoperability between different test tool and test bench vendors.The standard provides two levels of abstraction that are built on top of each other:the test bench

239、layer and the framework layer.The test bench layer defines interfaces for access to different technical areas of a test bench,the so-called test bench ports.Currently available ports are:The model access port(MAPort)for access to the simulation model.It is possible to read and write parameters,to ca

240、pture/record signals,and to generate signals.The ECU measurement port(ECUMPort)and ECU calibration port(ECUCPort)provide access to measurement and calibration tools.They allow the reading and capturing/recording of measurement variables of an ECU as well as the reading and writing of internal calibr

241、ation values of the ECU.The network access port(NetworkPort)provides access to network communication,such as frames,PDUs,and signals on CAN or Flexray.The diagnostic port(DiagPort)communicates with a diagnostic system to read data via diagnostic services from an ECU or functional group.The electrica

242、l error simulation port(EESPort)controls electrical error simulation hardware.It allows different types of electrical errors to be set.Table of Contents ASAM Test Specification Study Group Report 2022 44 The framework layer is built on top of the test bench ports and provides access to the test benc

243、h at a higher level while abstracting more vendor-specific details and configuration options.The purpose of the XIL framework is to achieve effective decoupling between test cases and test benches to improve test reuse.The XIL framework provides port-independent reading,writing,and capturing of vari

244、ables,conversion of units and data types,mapping of variable identifiers,and configuration and life cycle management of the test bench ports.Therefore,the XIL framework uses functionality of the XIL test bench ports.The XIL framework also allows the configuration of data acquisition from different d

245、ata sources,that is,from different XIL test bench ports.The time traces of variables from these different data sources are assembled on a common time basis.With both XIL test bench and XIL framework,data capturing/recording can be performed based on memory for online processing or based on files int

246、o standardized measurement formats(ASAM MDF 4.0 or higher)for data reuse in a later process stage.References https:/ The standard is well established.The current version is XIL 2.2.Version XIL 3.0 is under development(see https:/ to ASAM XIL 3.0.0 project page(https:/ are the new features for XIL 3.

247、0 to be developed:Table of Contents ASAM Test Specification Study Group Report 2022 45 Introduction of a new value container with generic complex data types Introduction of a new test bench port for access to service-oriented communication(SOC):this covers tests of service provider or service consum

248、er as well as integration tests in the network and storage of SOC data in ASAM MDF Traceability of test case execution,including logging of events Usage of complex setup routines to react to status events of the test system Monitoring of variables and states by streaming instead of polling Error inj

249、ection modification to stimulate function signals Access to system description via an API Rework of the existing ECUM and ECUC test bench ports to one common ECU port Usage of the standard in Linux environments All in all,these are many general improvements,but some have a strong relation to other s

250、tandards and ADAS/AS development needs.Current role and relevance with regard to ADAS/AD Current relations to other standards:ASAM MDF(Measurement Data Format)Currently,the XIL standard requires that measurement results be stored in ASAM-MDF-compliant files of version 4.0 or higher.XIL 3.0 is target

251、ing a solution for the storage of complex data from service-oriented communication in MDF.This may overlap with current MDF activities on the support of image,radar,lidar,and sensor logging(see https:/ Mock-up Interface)XIL relates to the FMI standard in terms of variability of simulation variables.

252、Furthermore,the growing relevance of(distributed)cosimulation-based test systems raises challenges that are relevant to both standards and may require alignment(esp.regarding system configuration and simulation control).The relation of FMI to the ASAM XIL standard needs to be evaluated.Future relati

253、ons to other standards:AUTOSAR Adaptive Platform and SOME/IP The future introduction of an XIL SOC(service-oriented communication)port will eventually have a strong relation to the standards AUTOSAR Adaptive Platform and SOME/IP(Scalable Service-Oriented Middleware over IP).However,it is expected th

254、at the new SOC port interfaces will encapsulate and abstract the technical details of these standards,similar to what is already being done today with the existing XIL test bench ports.ASAM OSI The increasing usage of ASAM OSI for environment models(coupling of sensors,agents,etc.)may raise the need

255、 for the XIL standard to extend its functionality accordingly to support MIL,SIL,and Table of Contents ASAM Test Specification Study Group Report 2022 46 HIL testing for future AD applications.Especially the new data types and the binary data type used by OSI may need some adaptions allowing OSI dat

256、a to be captured,recorded,or stimulated.In addition,the specifics of configuring and controlling scenario-based environment simulations may raise further requirements of the XIL standard.The relation of ASAM OSI to the ASAM XIL standard needs to be evaluated.Additional study group findings on ASAM X

257、IL The study group has identified interrelations between the standards FMI,ASAM MDF,ASAM OSI,and ASAM XIL as described in this section.The recommendation is to check on the coordination activities between the different standardization projects.4.3.3.3 FMI Functional Mock-up Interface Description The

258、 Functional Mock-up Interface(FMI,https:/fmi-standard.org/)is a free standard that defines a container and an interface to exchange dynamic models using a combination of XML files,binaries,and C-code zipped into a single file.It is supported by more than 150 tools and maintained as a Modelica Associ

259、ation Project on GitHub.Status The standard is well established.The current version is FMI 2.0.2.The version FMI 3.0 is under development and will be released soon.Currently,a prerelease beta 1 version is available.The current version of FMI has several limitations and was released a while ago(FMI 2

260、.0 release in July 2014).Nevertheless,the upcoming version FMI 3.0 will resolve many shortcomings.According to the FMI standard website(https:/fmi-standard.org/faq/#what-will-be-the-new-features-of-fmi-30),these are the new features for FMI 3.0(preliminary):Ports and icons Array variables Clocks and

261、 hybrid cosimulation Binary data type Intermediate variable access Source code FMUs Numeric variable types Extra directory Table of Contents ASAM Test Specification Study Group Report 2022 47 Current role and relevance in regards to ADAS/AD All in all,these are general improvements,but some have a s

262、trong relation to other standards and ADAS/AD development needs.The figure below shows some of these dependencies.FMI 3 and OSI With the FMI 3 binary port,the implementation of OSI interfaces for FMUs(functional mock-up units)will be more robust and replaces an existing work-around(OSMP OSI Sensor M

263、odel Packaging).The binary data type is an opaque binary data type to FMU variables to allow,for instance,the efficient exchange of complex sensor data(OSI)or also field bus data(V-ECU).(Note:The OSI standard itself is discussed in an extra section of this document.)FMI 3 and V-ECUs Currently,no sta

264、ndardization project for V-ECUs exists.Definitions for V-ECUs and the need for standardization currently are documented in a PROSTEP white paper:WhitePaper_V-ECU.pdf.The idea is to implement a V-ECU standard as a layered standard on top of FMI 3 to be usable for all kinds of ECU projects(AUTOSAR Cla

265、ssic,AUTOSAR Adaptive,and non-AUTOSAR ECUs).Besides the FMI 3 binary port for bus data,some more FMI 3 features are important for this layered approach.The FMI 3“extra directory”adds a new folder in the ZIP archive representing an FMU,providing additional data to travel with the FMU that can be modi

266、fied by different tools,allowing for layered standards.The FMI 3“clocks and hybrid cosimulation”feature introduces clocks for synchronization of variable changes across FMUs and allows cosimulation with events.Thus,the realization of most V-ECU input/output signals with FMI 3.0 will be possible.The

267、following aspects need to be evaluated and then would determine the content for a V-ECU layered standard:Bus ports(CAN,LIN,FlexRay,Ethernet):Maybe possible via FMI 3 binary ports in combination with MIME types for each bus system,but multiple frames per channel must be possible Events and callback f

268、unctions:Events essential for bus communication or simulated hardware events,for instance;callbacks for data measurement services outside of V-ECU;maybe possible via clocks in FMI 3.0 Table of Contents ASAM Test Specification Study Group Report 2022 48 Address based access to calibration and measure

269、ment variables(with A2L files).Not possible with get/set functions,but via direct memory access;extension of FMI especially for V-ECUs necessary Simulation performance:Events/callbacks via several clocks;if no direct memory access possible,additional memory copies necessary,which reduces performance

270、 FMI 3,OSI,V-ECU,and XIL API Some new features of FMI 3.0 along with the increasing usage of OSI for environment models(coupling of sensors,agents,etc.)raise the need for the XIL API to extend its functionality accordingly to support MIL,SIL,and HIL testing for future AD applications.Especially the

271、new data types and the binary data type used by OSI need some adaptions to allow them to capture,to record,or to stimulate OSI data.The relation of a V-ECU standard to the XIL API ports(MA,ECUC,ECUM,EES,DIAG,NETWORK)needs to be evaluated.(Note:The XIL API standard itself is discussed in an extra sec

272、tion of this document.)Study group summary of FMI 3 The findings of the study group do not conflict with the FMI 3 planned features.Nevertheless,the study group has identified the interrelations between the standards FMI 3,OSI,V-ECU,and XIL API as described in this section.The recommendation is to c

273、heck on the coordination activities between the different standardization projects.Table of Contents ASAM Test Specification Study Group Report 2022 49 4.3.3.4 ISO/WD 22133-1 ISO/WD 22133“Road vehicles Test object monitoring and control for active safety and automated/autonomous vehicle testing Func

274、tional requirements,specifications and communication protocol part 1”Road Vehicles Test Object Monitoring and Control for Active Safety and Automated/Autonomous Vehicle Testing Part 1:Functional Requirements,Specifications,and Communication Protocol Description ISO 22133 consists of the following pa

275、rts,under the general title“Road Vehicles Test Object Monitoring and Control for Active Safety and Automated/Autonomous Vehicle Testing”:Part 1:Functional Requirements,Specifications,and Communication Protocol Part 2:Test Scenario Description Formats The testing of collision avoidance systems,active

276、 safety functions,and more advanced autonomous functions in vehicles requires testing on proving grounds.The purpose is to expose the vehicle under test to potentially dangerous traffic situations in a safe manner.The evaluation is done during development,and in voluntary and mandatory test procedur

277、es.To orchestrate these traffic scenarios,various impactable targets representing traffic actors have to be controlled.The number of controlled targets may be one or many depending on the required traffic scenario.Multiple requirements are important,such as safety,position and speed precision,and lo

278、gging capabilities.This document specifies requirements,functionality,and a protocol allowing for multi-vendor target carrier systems to be controlled according to the required traffic scenario,to report expected information for logging purposes and other functions required.Scope This document speci

279、fies requirements,procedures,and message formats for the controlling and monitoring of test targets,used for the testing of active safety functions and autonomous vehicles.The document specifies functionality and messaging for the monitoring and controlling of test objects by a control center facili

280、tating an interoperable test object environment.This document does not specify the internal architecture of the test object and control center.This document does not specify how the testing of the vehicles shall be performed.Table of Contents ASAM Test Specification Study Group Report 2022 50 4.3.4

281、Test Specification These are standards that are used to define or automate test specifications and their definition.4.3.4.1 ASAM ATX Description The ASAM ATX standard(Automotive Test Exchange Format,see https:/ details)defines a format for a standardized exchange of test data between different test

282、systems.ATX supports the ISTQB“Certified Tester”syllabus methodology and can be used for many activities in the test process,e.g.test specification,test planning,test execution,and test evaluation.The ATX test exchange format allows the reuse of existing test cases in different test automation softw

283、are systems.It enables the description of manual or automated test specifications.References:https:/ Activities in the Test Process:Test Specification The Test specification activity realizes the general testing objectives by the creation of concrete test conditions and test cases.The major tasks ar

284、e the following:Define and specify the possible test objects Define and specify the test base(such as requirements,failure mode and effects analysis,risk analysis reports,architecture,design,interface specifications)and evaluate the testability of the test base and test objects Identify and prioriti

285、ze the test cases based on analysis of test items and the specification,behavior,and structure of the software Design high-level test cases(also called logical or abstract test cases)and model the test case flow by defining test steps and test actions Document test level,quality criteria,and the use

286、d test design technique for the test cases Create bidirectional traceability between test base and test cases Identify the necessary test parameters to support the configuration of test cases,test steps,or test actions Specify the possible test environment setup and document the required infrastruct

287、ure and test tools Test Preparation Table of Contents ASAM Test Specification Study Group Report 2022 51 Test preparation is the activity where test suites are specified by combining the test cases in a particular order and including any other information needed for test execution.Test preparation h

288、as the following major tasks:Identification and definition of the concrete test object Identification and definition of the test environment Identification,definition,and/or creation of test data(test parameter values)Creation of test execution plans(test suites)from the test cases Implementation of

289、 unimplemented test actions in the chosen implementation language Test Execution In this activity the test environment is set up and the tests are executed.The major tasks are the following:Generation of code for the test cases identified by the test execution plan by incorporating the specified tes

290、t data Compilation of the generated code for the test cases Setup and verification of the test environment.Optionally,preparation of the test harnesses Execution of test procedures either manually or by using test execution tools,according to the sequence in the test execution plan Logging of the ou

291、tcome and the verdict of test execution.Recording the identities and versions of the unit under test,test tools,etc.Test Evaluation The test evaluation activity collects the results of a test execution and analyzes the outcome.The major tasks are:Comparing results with expected results Reporting the

292、 differences as incidents in the test report Reporting noticeable problems in the test report and analyzing them Attaching test logs,result data,etc.to the test report Repeating test activities if necessary Test Environment A test environment describes the hardware,instrumentation,simulators,softwar

293、e tools,and other support elements needed to conduct a test.Test Object A test object describes the component or system to be tested.A test object may consist of different parts.Test Base A test base represents a chunk of information from which the requirements of a component or Table of Contents AS

294、AM Test Specification Study Group Report 2022 52 system can be inferred.This might be for example complete documentation,a requirements document,or a single requirement exported out of a requirement management or issue tracking system.Quality Attributes There are a number of quality attributes avail

295、able with which product quality can be described,such as functionality,reliability,usability,efficiency,maintainability,and portability.Test Design Technique A test design technique is a method used to derive or select test cases.Test Specification A test specification()is a container for a set of t

296、est cases and a lot of things which they have in common,e.g.the test base,test objects,or test environments.Test Case A test case is an essential part of the test specification.It allows the definition of tests and a set of input values,execution preconditions,expected results,and execution postcond

297、itions,developed for a particular objective or test condition,such as to exercise a particular program path or to verify compliance with a specific requirement.Test Step A test step is part of a specific sequence that serves to divide a test case into steps.They can be documented individually and wi

298、ll show up in the test report.Test Action A test action is a single operation inside a test step.Test Parameter Values The element is used to provide the values for test parameters defined within a.Test Execution Plan A is a test suite describing scope,environment,and schedule of intended test activ

299、ities.It identifies the test objects,the test environment,the test parameter values,the planned test cases,and their execution sequence.It is a record which can be referenced in a test planning process.The results of the execution of a test execution plan are recorded in a test report.Test Report Th

300、e test report contains the consequence/outcome of the execution of a test execution plan.For Table of Contents ASAM Test Specification Study Group Report 2022 53 all executed test cases it includes verdicts and result data like outputs to streams,data changes,logs,or result files.Relation to other s

301、tandards ATX uses the terminology defined by ISTQB.ATX reuses patterns and XML structures defined by the following standards:MSR Concepts of Application Profile(MSR-TR-CAP)ASAM Harmonized Data Objects(ASAM-HDO)ASAM AE Functional Specification Exchange Format(ASAM AE-FSX)AUTOSAR V4.0 AUTOSAR Generic

302、Structure Template AUTOSAR Software Component Template Table of Contents ASAM Test Specification Study Group Report 2022 54 By using ASAM XIL API(see section XXX)as interface to test benches/hardware the test exchange flexibility is enhanced due to the abstraction from each specific test bench.Statu

303、s The first version of the ASAM ATX standard 1.0.0 was published in 2012.The current version is 1.0.1 from September 25,2018.In this release only the TEST-OBJECT-SET was extended.At the current time there are no further development projects.Source https:/ group summary of ATX Although the activity i

304、s limited,the standard offers interesting aspects in terms of test specifications,metadata,test data,and test plans.However,the standard does not currently support any special features of scenario-based testing.The focus is on classic,requirements-based,and functional testing.Main benefits of ATX:Un

305、ified abstract test specification to be shared among different authorities,OEMs,suppliers,and tools Development of the test system is independent of test libraries(efficiency increase on end-user side)ASAM ATX is frequently used in conjunction with ASAM HIL in hardware-in-the-loop test systems.Sourc

306、e:ASAM e.V.ASAM ATX,Programmers Guide,Version 1.0.1,Date:2018-09-25 Table of Contents ASAM Test Specification Study Group Report 2022 55 4.3.4.2 OTX(ISO 13209)“Open Test sequence eXchange format”Description OTX is a proven industrial standard for test sequences and is as such very important in the f

307、ield of scenario-based testing.Using OTX,test specification and test logic can be described in a standardized format.OTX(ISO 13209 and ASAM OTX Extensions)OTX is first and foremost a programming language,especially related to the domain of testing within the automotive industry.Therefore,OTX is a so

308、-called domain-specific language(DSL).OTX stands for“Open Test Sequence Exchange”and has been an ISO standard since 2012(ISO 13209).OTX is executable and the test logic can be displayed graphically,where the strict separation of test logic and runtime implementation allows the integration of OTX int

309、o any target system(see figure).OTX has harmonizing effects,because it can bring together previously separated standards.It can be extended by any new functionality based on a standardized extension mechanism.OTX has a measurable quality based on about 150 semantic checker rules to improve process r

310、eliability.OTX is industry proven,is stable,and is continuously being developed by means of ASAM OTX extensions.In summary,one can say that “OTX is a domain-specific programming language for the process-reliable description of exchangeable and executable test logic in the automotive industry.”One of

311、 the main features of OTX is the standard-compliant exchangeability of executable test logic between development,production,after-sales,and embedded systems inside the car,for example as a so-called on-board tester.This shows that like no other current standard,“OTX can ensure that the same unchange

312、d test logic can be executed in any target system at any time and comes to the same results.”Table of Contents ASAM Test Specification Study Group Report 2022 56 OTX consists of a stand-alone executable core(see figure).Inside the core the entire data model for a complete programming language is def

313、ined.These are procedure calls,assignments,branches,loops,activities for parallel execution and error handling as well as all extension interfaces which new OTX extensions can implement.Each OTX extension extends the core by means of new domain-specific functionality.This is especially relevant for

314、the vehicle diagnostics domain,but also for the communication between different external systems and for other functionality which is necessary for a complete language,like file access,logging,string handling,date and time handling,and so on.Table of Contents ASAM Test Specification Study Group Repo

315、rt 2022 57 In the ongoing ASAM project“OTX Extensions”the following improvements have already been implemented or are still being planned(see figure).The core has been expanded by new extension points,so that OTX can support not only the test logic,but also the test description.Based on this a new U

316、nitTest extension was implemented as a prototype(see below for details).A new range extension extends OTX by a set of new simple data types to describe values with validity ranges,for example a float value with an upper and/or lower limit as well as a blacklist and whitelist.Furthermore,OTX can be e

317、xtended by an ASAM XIL extension to get access to the test bench via the XIL model access port.OTX will also support the upcoming ASAM standard for service-oriented vehicle diagnostics(SOVD)with one or more new OTX extensions.And finally,OTX can be extended for the new ASAM OpenX standards,or for ex

318、ample for the domain of machine learning or big data.OTX UnitTest Extension The UnitTest extension is a prototype to show the possibilities of the new extension points.The extension can be used as a base for further extensions.The UnitTest extension adds a new procedure type to the core,the“TestCase

319、Procedure.”A test case procedure contains one or more test cases and can be structured in test suites(see figure for an example).Table of Contents ASAM Test Specification Study Group Report 2022 58 The simple example of a division of two float values shows some of the main features.The first test ca

320、se divides 10 by 2 and the expected result is 5.The second test case does the same with minus values.The last test case expects an exception because of the division by zero.The extension covers about 90 percent of the well-known NUnit testing framework(see NUnit.org),e.g.oneTimeSetupProcedure,setupP

321、rocedure,tearDownProcedure,and oneTimeTearDownProcedure.References ISO 13209-1:2011 Road vehicles Open Test sequence eXchange format(OTX)Part 1:General information and use cases,https:/www.iso.org/standard/53507.html ISO 13209-2:2012 Road vehicles Open Test sequence eXchange format(OTX)Part 2:Core d

322、ata model specification and requirements,https:/www.iso.org/standard/53509.html ISO 13209-3:2012 Road vehicles Open Test sequence eXchange format(OTX)Part 3:Standard extensions and requirements,https:/www.iso.org/standard/55599.html ISO 13209-4:2021 Road vehicles Open Test sequence eXchange format(O

323、TX)Part 4:Expanded extensions interface definition,https:/www.iso.org/standard/74090.html ISO/DIS 13209-2:2022,https:/www.iso.org/standard/76976.html(under development)ISO/DIS 13209-3:2022https:/www.iso.org/standard/76977.html(under development)ASAM OTX extensions,https:/ Relation to Other Standards

324、 OTX has a close relationship to the standards for vehicle diagnostics but it can also be used outside these areas.ISO 22900-3:2012 Road vehicles Modular vehicle communication interface(MVCI)Part 3:Diagnostic server application programming interface(D-Server API),https:/www.iso.org/standard/56615.ht

325、ml ISO 22901-1:2008 Road vehicles Open diagnostic data exchange(ODX)Part 1:Data model specification,https:/www.iso.org/standard/41207.html ASAM SOVD Service oriented vehicle diagnostics,https:/ ATX,https:/ ASAM XIL,https:/ necessary,OTX can be expanded to support existing standards.Status The standa

326、rd is well established.It is mature and comprehensive enough to replace existing solutions in development,production,and the workshop.Most German vehicle manufacturers have been using the standard productively for years or are planning to use it.We currently estimate around 100,000 installations in

327、development,production,after-sales and within the vehicle,and the trend is growing rapidly.OTX is a process issue.It has a strong influence on optimizing processes.Table of Contents ASAM Test Specification Study Group Report 2022 59 Numerous well-known tool suppliers already offer a solution or are

328、in the process of developing one.Table of Contents ASAM Test Specification Study Group Report 2022 60 Table of Contents ASAM Test Specification Study Group Report 2022 61 Study group summary of OTX The OTX standard(ISO and ASAM)is a domain-specific programming language(DSL)for the process-reliable d

329、escription of exchangeable and executable test logic in the automotive industry.OTX can ensure that the same unchanged test logic can be executed in any target system at any time and comes to the same results.OTX can be expanded to meet the requirements of scenario-based testing.OTX is industry prov

330、en and has the necessary maturity.It is recommended to examine OTX more closely as a description language for scenario-based testing.The creation of a new ASAM working group“OTX Extensions for Scenario-based Testing”is proposed.Table of Contents ASAM Test Specification Study Group Report 2022 62 4.3

331、.5 Data Handling These are standards that define formats or processes for data management and storage 4.3.5.1 ODS and MDF Description ASAM MDF(Measurement Data Format)is a binary file format for storing recorded or calculated data for postmeasurement processing,off-line evaluation,or long-term stora

332、ge.ASAM MDF is not only used for the storage of sensor,ECU,bus,or monitoring data,it is also used for the compact storage of large data amounts as external components managed by an ODS(Open Data Services)Mixed-Mode Server.In addition to the classical bus data like Flexray,LIN,CAN,or Ethernet,new typ

333、es of data are recorded in the ADAS/AD areas(ground truth data(e.g.GNSS),GigEVision,SerDes/GMSL,FPD Link,or MIPI.These new types of data should be managed and stored in a similar way to the classical bus data in future MDF versions.Therefore,ASAM is currently planning to extend the ASAM MDF standard by an associated standard,“Image Radar Lidar Sensor Logging,”that describes how to store image stre