《Qt:安全关键程序的代码覆盖率的指南和注意事项白皮书(2022)(12页).pdf》由会员分享,可在线阅读,更多相关《Qt:安全关键程序的代码覆盖率的指南和注意事项白皮书(2022)(12页).pdf(12页珍藏版)》请在三个皮匠报告上搜索。
1、Qt 公司安全关键程序的代码覆盖率|2021 年 12 月 13 日1WHITEPAPERCode Coverage for Safety-Critical ProgramsMetrics&ConsiderationsThis white paper summarizes the different implications and considerations surrounding code coverage for safety-critical programs,along with code coverage requirements mandated by four major s
2、tandards governing safety-critical software in various industry contexts.First,we discuss the emergence of safety-critical software as part of modern systems.Next,we define code coverage analysis,how it works,and why it is used as a pre-requisite for achieving certification.Third,we define the most
3、encountered coverage metrics in software testing,against which test coverage can be measured.We describe their advantages and disadvantages in the context of quality assurance,and finally,state their relevance to the four safety standards.安全关键程序的代码覆盖率指标与注意事项本白皮书总结了围绕安全关键程序代码覆盖的不同定义和考虑因素,以及不同行业管理安全关键
4、软件的四种主要标准中所规定的代码覆盖率要求。首先,我们对作为现代系统一部分的安全关键程序的出现进行了讨论。接下来,我们定义了代码覆盖率分析,介绍它工作原理以及为什么它被用作获得认证的先决条件。第三,我们定义了软件测试中最常见的覆盖率指标,可以根据这些指标来衡量测试覆盖率。我们描述了它们在质量保证中的优、缺点,并在最后指出了它们与四种安全标准的相关性。白皮书Qt 公司安全关键程序的代码覆盖率|2021 年 12 月 13 日2Table of contents1.Safety-Critical Systems as Software.32.Code Coverage Requirements f
5、or Safety-Critical Programs .43.Coverage Metrics.53.1 Function Coverage.53.1.1 Definition.53.1.2 Relevance for Safety Standards.53.2 Line Coverage.63.2.1 Definition.63.2.2 Formatting Dependency.63.2.3 Disguised Control Flow.73.2.4 Relevance for Safety Standards.73.3 Statement Coverage.73.3.1 Definit
6、ion.73.3.2 Relevance for Safety Standards.83.4 Decision(Branch)Coverage.83.4.1 Definition.83.4.2 Relevance for Safety Standards.93.5 Modified Condition/Decision Coverage(MC/DC)Coverage.93.5.1 Definition.93.5.2 Relevance for Safety Standards.103.6 Multiple Condition Coverage(MCC)Coverage.113.6.1 Defi
7、nition.113.6.2 Relevance for Safety Standards.114.Conclusion.12目录1.安全关键系统即软件 .32.安全关键程序的代码覆盖率要求 .43.覆盖率指标.5 3.1 函数覆盖率 .5 3.1.1 定义 .5 3.1.2 安全标准的相关性.5 3.2 行覆盖率.6 3.2.1 定义 .6 3.2.2 格式化依赖性 .6 3.2.3 伪装的控制流 .7 3.2.4 安全标准的相关性.7 3.3 语句覆盖率 .7 3.3.1 定义 .7 3.3.2 安全标准相关性 .8 3.4 判定(分支)覆盖率 .8 3.4.1 定义 .8 3.4.2 安
8、全标准的相关性.9 3.5 修正条件/判定覆盖(MC/DC)覆盖率 .9 3.5.1 定义 .9 3.5.2 安全标准的相关性.10 3.6 多条件覆盖(MCC)覆盖率 .11 3.6.1 定义 .11 3.6.2 安全标的准相关性 .114.结论 .12Qt 公司安全关键程序的代码覆盖率|2021 年 12 月 13 日32Table of contents1.Safety-Critical Systems as Software.32.Code Coverage Requirements for Safety-Critical Programs .43.Coverage Metrics.5
9、3.1 Function Coverage.53.1.1 Definition.53.1.2 Relevance for Safety Standards.53.2 Line Coverage.63.2.1 Definition.63.2.2 Formatting Dependency.63.2.3 Disguised Control Flow.73.2.4 Relevance for Safety Standards.73.3 Statement Coverage.73.3.1 Definition.73.3.2 Relevance for Safety Standards.83.4 Dec
10、ision(Branch)Coverage.83.4.1 Definition.83.4.2 Relevance for Safety Standards.93.5 Modified Condition/Decision Coverage(MC/DC)Coverage.93.5.1 Definition.93.5.2 Relevance for Safety Standards.103.6 Multiple Condition Coverage(MCC)Coverage.113.6.1 Definition.113.6.2 Relevance for Safety Standards.114.
11、Conclusion.123Safety-Critical Systems as SoftwareA system is“safety-critical”if a failure in its operation could result in human fatality(or severe injury)or significant damage to property or the environment.Such systems are becoming increasingly computer-based.(Take,for example,the RATPs ongoing de
12、velopment of fully autonomous subway lines in the Paris railway network.)In this digital transformation,standards in the field of safety engineering have emerged,which set requirements on the software development of software-based safety-critical systems.Under the IEC 61508 standard,which governs fu
13、nctional safety of electrical/electronic/programmable electronic safety-related systems,the probability of a dangerous failure is such that,less than one failure one human life lost is probable every 114,115 years of continuous system operation at the top Safety Integrity Level(SIL).Apart from their
14、 macroscopic view of human,property,and environmental safety,these standards primary goal is to ensure the software quality and fitness at the source code level.In other words,the low(or zero)defect rate requirement extends to software operation.A method to achieve this is through quality assurance
15、testing.1.安全关键系统即软件若系统运行故障可能导致人员死亡(或重伤)或严重损害财产、周边环境,则该系统被称为“安全关键系统”。此类系统对计算机的依赖性正不断增加(例如RATP 正在为巴黎轨道网络开发的全自动地铁线路)。在数字化转型过程中,安全工程领域的标准已应运而生,这些标准对基于软件的安全关键系统的开发提出了一系列要求。根据管理电气/电子/可编程电子安全相关系统功能安全的 IEC 61508 标准,发生危险故障的概率为:在最高安全完整性等级(SIL)下,系统每 114,115年连续运行可能发生不到一次故障 一人死亡。除对人员、财产和环境安全的宏观规定外,这些标准的主要目标是确保源代
16、码层的软件质量和适用性。换言之,低(或零)缺陷率的要求延伸到了软件运行中。通过质量保证测试是实现这一目标的方法。Qt 公司安全关键程序的代码覆盖率|2021 年 12 月 13 日42.安全关键程序的.代码覆盖率要求每一种安全标准均规定了一系列软件测试要求,其一便是程序的代码覆盖率。由此产生了三个问题:1.什么是代码覆盖率?2.如何量化和统计代码测试覆盖率?3.为什么安全关键系统对代码覆盖率有要求?代码覆盖率是一种分析方法,用于衡量一个或多个测试所执行的源代码函数、语句和条件的百分比。代码覆盖率数据使用检测程序代码的工具测量和分析得出,这是在代码中插入指令来跟踪执行情况的预编译步骤。当我们对被
17、检测的二进制文件运行完一整套测试,即可获得覆盖率数据。代码覆盖率分析在确保安全关键程序及其分支系统的适用性方面发挥着至关重要的作用。这种分析会告知开发团队先验测试未触及哪些(潜在的关键)程序区域。代码覆盖率分析工具使开发者能够识别并清除无用代码、检测并解决 bug、重构现有代码以提高效率、消除冗余测试等。在受监管的行业中,代码覆盖率在降低生产中发生严重缺陷的可能性方面发挥着至关重要的作用。代码覆盖率能够帮助确保软件质量,有助于确保软件可信度。若要获得认证,必须达到特定标准所规定的代码覆盖率水平。每种标准都有其独特要求。现在,我们将以四个主要标准为例,介绍覆盖率指标要求。这些标准是:ISO 26
18、262 道路车辆 功能安全 IEC 61508 电气/电子/可编程电子安全相关系统的功能安全 DO-178C 机载系统中的软件注意事项 EN 50128 铁路控制和保护系统软件在以下章节中,我们提供了一个 C+示例程序以及调用示例,以介绍可以衡量测试覆盖率的不同覆盖率指标。度量标准包括函数覆盖率、行覆盖率、语句(块)覆盖率、判定(或分支)覆盖率、修正条件/判定覆盖率(MC/DC)和多条件覆盖率(MCC)。我们的示例程序能够检测测试用例的输入是否为实数。它展示了所有的覆盖率指标,包含在特定的测试中有些指标低于其他指标。Qt 公司安全关键程序的代码覆盖率|2021 年 12 月 13 日5 53.
19、3.13.1.1Coverage MetricsFunction CoverageDefinitionThe function coverage of a program counts how many functions were called(and how often).Here,the count includes member functions(or methods)in object-oriented programming languages like C+.Consider our sample program,with an example function call:No
20、te that this metric reports only that a function was called;it does not report the execution of the body of the function.Relevance for Safety Standards3.1.2*architectural levelISO 26262*IEC 61508DO-178CEN 50128A =+1=+-B=+2=+-C =+3=+-D =+4=+-Table 1Key:+denotes recommended;+denotes highly recommended
21、.A-D,1-4 are the respective standards Safety Integrity Levels(SILs).Function coverage is generally useful as an initial assessment of a projects coverage,but more robust metrics are required for in-depth analyses.Indeed,within the safety standards discussed in this paper,function coverage is require
22、d always with stricter metrics.3.覆盖率指标3.1 函数覆盖率3.1.1 定义程序的函数覆盖率指有多少函数被调用(以及调用频率)。计数包括 C+等面向对象的编程语言中的成员函数(或方法)。请看我们的示例程序,带有一个函数调用示例:请注意,此指标只报告一个被调用的函数,而不报告函数主体的执行情况。3.1.2 安全标准相关性*架构水平函数覆盖率通常可用作对项目覆盖率的初步评估,但更深入的分析则需要更严格的衡量指标。事实上,在本白皮书中探讨的安全标准中,函数覆盖率总是需要更严格的衡量标准。表格 1图例:+代表推荐;+代表强烈推荐,A-D,1-4 为对应标准的安全完整性
23、等级(SIL)。Qt 公司安全关键程序的代码覆盖率|2021 年 12 月 13 日6 6Line CoverageDefinitionLine coverage is the number of executed source code lines divided by the total number of source code lines.Depending on the code coverage toolchain in use,only lines that contain executable statements may be considered,not those wit
24、h pure declarations.Other tools count pure declarations as executable code,for example:int x=0Formatting DependencyThis metric is unstable because it depends strongly on a programs code formatting.Consider our original program above,with a sample invocation:3.2.2This results in 80.00%line coverage.I
25、f you would reformat line 16 like this:The same invocation would result in 81.82%line coverage for a decision which is only partially executed.Intermediate line coverage is meaningless in quality assessments,and therefore only complete,100%coverage is considered.One can imagine writing an entire fun
26、ction on one line or breaking the statements into multiple lines to change the coverage percentages systematically,without any increase in the quality of the tests.3.2 行覆盖率3.2.1 定义行覆盖率为已执行的源代码行数除以源代码总行数。根据使用的代码覆盖率工具链不同,可能只计算包含可执行语句的行而不计算纯声明的行。其他工具会将纯声明行计为可执行代码,例如:3.2.2 格式依赖此指标在很大程度上取决于程序的代码格式,因此该指标结
27、果不稳定。参考前文所述的原始程序,其中包括一个示例调用:结果是行覆盖率为 80.00%。若将 16 行修改为以下格式:相同的调用,判定仅被部分执行,得出的行覆盖率为 81.82%。在质量评估中,一般的行覆盖率毫无意义,因此只考虑完全覆盖,即100%覆盖率。可以想象,将整个函数写入一行,或者将语句分解为多行以系统改变覆盖率,却不会提升测试的质量。Qt 公司安全关键程序的代码覆盖率|2021 年 12 月 13 日7Disguised Control FlowConsider our original code listing before we demonstrated the formatti
28、ng dependency.Lets modify the code to move the subsequent return statement in line 16,as follows:3.2.3With the following invocation:Our coverage increases to 88.89%.Note that line 16 is displayed as covered.But our test has not touched the return false statement;for decisions,or branching of code,li
29、ne coverage does not detect the missing test.Relevance for Safety StandardsNone,due to the limitations listed above.Statement coverage,addressed in section 3.3 remediates the formatting dependency described above.Decision coverage,described in section 3.4,addresses the issue of branching seen in the
30、 example.3.2.4Statement CoverageDefinitionStatement coverage tracks the executed program statements.Statement coverage is calculated by dividing the number of executed statements by the total number of statements.Depending on the code coverage toolchain in use,the metric may be reported using simple
31、 or compound statements,the latter also known as blocks or statement blocks.A block groups a sequence of simple statements.The compiler treats such blocks as a single statement.C+,for example,uses curly braces for grouping.Achieving 100%statement coverage gives 100%statement block coverage,and vice
32、versa.3.33.3.13.2.3 伪装的控制流在演示格式依赖之前,请考虑我们的原始代码清单。为了将之后的 return 语句移到第 16 行,让我们修改代码如下:通过以下调用:覆盖率提升到 88.89%。请注意第 16 行显示为已覆盖。但测试并未触及 return false 语句;对于判定或分支代码,行覆盖率不会检测到缺少的测试。3.2.4 安全标准相关性无,因为有上述的限制。第 3.3 节介绍的语句覆盖率修正了上述格式依赖。第 3.4节介绍的判定覆盖率解决了示例中出现的分支问题。3.3 语句覆盖率3.3.1 定义语句覆盖率跟踪可执行的程序语句。语句覆盖率的计算方式是用已执行的可执行语
33、句数除以可执行的语句总数。根据使用的代码覆盖率工具链,该指标可使用简单或复合语句报告,复合语句也称为块或语句块。块对一连串简单语句进行分组。编译器将此类块视为单个语句。例如,C+使用大括号 进行分组。实现 100%语句覆盖率则可以得到 100%的语句块覆盖率,反之亦然。Qt 公司安全关键程序的代码覆盖率|2021 年 12 月 13 日8Statement coverage remediates the formatting dependency seen earlier in Line Coverage.In block coverage,complete statement(block)c
34、overage subsumes complete line coverage.The disadvantage of statement(block)coverage is its weakness to simple-if statements.Simple-if structures have no else clause.Thus,complete statement coverage can be achieved for simple-if structures,regardless of the decisions truth outcome.3.3.2Relevance for
35、 Safety StandardsISO 26262*IEC 61508DO-178CEN 50128A =+1=+A=+0=+B=+2=+B=+1=+C =+3=+C =+2=+D =+4=+D =N/A3=+-E =N/A4=+Table 2Key:+denotes recommended;+denotes highly recommended.A-D,1-4,A-E,0-4 are the respective standards Safety Integrity Levels(SILs).8Statement coverage remediates the formatting dep
36、endency seen earlier in Line Coverage.In block coverage,complete statement(block)coverage subsumes complete line coverage.The disadvantage of statement(block)coverage is its weakness to simple-if statements.Simple-if structures have no else clause.Thus,complete statement coverage can be achieved for
37、 simple-if structures,regardless of the decisions truth outcome.3.3.2Relevance for Safety StandardsISO 26262*IEC 61508DO-178CEN 50128A =+1=+A=+0=+B=+2=+B=+1=+C =+3=+C =+2=+D =+4=+D =N/A3=+-E =N/A4=+Table 2Key:+denotes recommended;+denotes highly recommended.A-D,1-4,A-E,0-4 are the respective standar
38、ds Safety Integrity Levels(SILs).Statement coverage is mandatory under the ISO 26262 standard for lower SILs,which do not highly recommend more stringent coverage levels.(For example,for ASIL D,Modified Condition/Decision Coverage(MC/DC)is required,which subsumes 100%statement coverage.)The DO-178C
39、standard requires no statement coverage(or any higher-order metric)for levels D or E,where there are minor failure conditions or no effect on the system,respectively.Decision(Branch)CoverageDefinitionThe decision(or branch)coverage is the number of executed statement blocks and decisions divided by
40、the total number of statements and decisions.Here,each decision counts twice:once for the true case and once for the false case.We can achieve 100%decision coverage in our program with a minimum of two invocations:3.43.4.1语句覆盖率弥补了行覆盖率章节所介绍的格式依赖问题。在块覆盖率中,完整的语句(块)覆盖率包括了完整的行覆盖率。语句(块)覆盖率的缺点在于无法正确识别简单if语
41、句。简单if语句中没有else子句。因此,无论判定的真假结果如何,都可以对简单 if 结构可以实现完整的语句覆盖。3.3.2 安全标准相关性根据 ISO 26262 标准,对于较低 SIL 来说,语句覆盖是强制性的,该标注并不强烈推荐更严格的覆盖级别。(例如,对于 ASIL D,需要修正条件/判定覆盖(MC/DC),其中就包含 100%语句覆盖率)。DO-178C 标准不要求 D 或 E 级的语句覆盖率(或更高阶的指标),因为这两个级别分别存在轻微的故障条件或对系统没有影响。3.4 判定(分支)覆盖率3.4.1 定义判定(或分支)覆盖率指已执行的语句块和判定数量除以语句和判定的总数。此处每个判
42、定计算两次:一次为真,一次为假。我们可以通过至少两次调用在程序中实现 100%的判定覆盖率:注意,这将自动导致 100%的语句、行和函数覆盖率。达到 100%的判定覆盖率可确保满足所有判定结果,这是语句覆盖率固有的缺陷。也就是说,这个指标不考虑由逻辑运算符产生的布尔表达式的分支,例如,&,|。表格 2图例:+代表推荐;+代表强烈推荐,A-D,1-4 为相应标准的安全完整性等级(SIL)。Qt 公司安全关键程序的代码覆盖率|2021 年 12 月 13 日9The Qt CompanyCode Coverage for Safety-Critical Programs|December 13,2
43、0219Taking a look at line 16:A truth table reveals the untested conditional expressions:Cond.Truth val.c =0T()F(X)c=0c=0c=0 c=0 c=9 c=.判定真.假.真.假.对于这种情况,我们可选择下个章节讨论的条件覆盖率(MCC)。3.5.2 安全标准相关性表 4图例:+代表推荐;+代表强烈推荐,A-D,1-4 为相应标准的安全完整性等级(SILs)。在 EN 50128 标准中,MC/DC(或 MCC)建议用于 SIL 级别 1、2,强烈建议用于 SIL 级别 3、4。Qt
44、公司安全关键程序的代码覆盖率|2021 年 12 月 13 日1111Multiple Condition Coverage(MCC)CoverageDefinitionIn Multiple Condition Coverage(MCC),every combination of condition outcomes within a decision occurs at least once to reach full coverage.The coverage is measured by taking the number of executed statement blocks an
45、d condition combinations divided by their total number in the program.With MCC,a complete decision table would be needed for full coverage.To determine the required tests in the decision table,substitute the number of conditions for N in 2N.Our above invocations for complete MC/DC coverage results i
46、n 93.750%MCC coverage.In our program,it is not technically possible to write a test which completes the decision table(owing to the single variable c which is included in every condition.)Relevance for Safety StandardsOf the four standards in this paper,the DO-178C and EN 50128 standards recommend M
47、CC(or MC/DC)in their requirements.Generally,the MC/DC metric requires N +1 tests,where N is again the number of conditions.Required tests in the MCC metric can explode exponentially with large numbers of conditions.The MC/DC metric was created to compromise between plain condition/decision coverage
48、and MCC.3.63.6.13.6.23.6 多条件覆盖(MCC)覆盖率3.6.1 定义在多条件覆盖(MCC)中,一个判定中的每个条件结果组合至少会出现一次,以达到完全覆盖。多条件覆盖率的计算方法是用已执行语句块和条件组合的数量除以它们在程序中的总数。使用 MCC,需要一个完整的决策表来进行全面的覆盖。要确定判定表所需的测试,需要将 2N 中的 N 替换为条件数量。上述对完整MC/DC覆盖的调用可实现93.750%的多条件覆盖率。在我们的程序中,从技术的角度,不可能写出一个完成判定表的测试(因为每个条件中都包含单个变量 c)3.6.2 安全标准相关性在本白皮书的四个标准中,DO-178C
49、和 EN 50128 标准在其要求中推荐 MCC(或MC/DC)。通常,MC/DC 指标需要进行 N+1 测试,这里的 N 同样指的是条件数量。条件数量庞大时,MCC 指标中所需要的测试数量会呈指数级“爆炸”。创建 MC/DC 指标是为了在简易的条件/判定覆盖和多条件覆盖之间进行折衷。在测试效率和开销权衡的前提下,已经进行了比较 MC/DC 和 MCC 之间的错误检测概率的研究。我们鼓励读者回顾这项研究,因为它实用于他们自己的程序。Qt 公司安全关键程序的代码覆盖率|2021 年 12 月 13 日12124.ConclusionThe four standards presented in
50、this paper are unique in their coverage metric requirements,but all share the common thread of minimizing system failures to prevent human fatalities.Software development of safety-critical systems requires sophisticated code coverage tools to permit a“test smart”vs.“test more”methodology not only t
51、o achieve safety certification but to deliver products within increasingly constrained frameworks.Automating the code coverage,where possible,is key to reducing human errors,which are the base cause of software defects.Two things hold true about safety-critical software.First,their systems cannot be
52、 made safer once they are already in use.Second,due largely to continuing technological advancements in computing,these systems will play an increasingly ubiquitous part in human life.Therefore,prioritizing quality assurance as a means to ensuring confidence and fitness of the software for use is paramount.4.结论本白皮书中所介绍的四种标准都有其独一无二的覆盖率指标要求,但它们都有一个共同点,即旨在最大限度地减少系统故障以防止人员伤亡。安全关键系统的软件开发需要尖端的代码覆盖工具以兼顾“智能测试”与“详尽测试”,不仅能获得安全认证,还能在日益受限的框架内交付产品。尽可能自动化代码覆盖是减少人为错误的关键,人为错误是软件缺陷的根本原因。安全关键软件有两个特点。首先,一旦投入使用,其系统就无法变得更安全。其次,由于计算技术的持续进步,这些系统将在人类生活中发挥越来越普遍的作用。因此,将质量保证作为确保软件使用的信心和适用性的一种手段是至关重要的。