1、World Digital Technology Academy(WDTA)Generative AI Application Security Testingand Validation StandardWorld Digital Technology Academy StandardWDTA AI-STR-01Edition:2024-04 WDTA 2024 All rights reserved.The World Digital Technology Standard WDTA AI-STR-01 is designated as a WDTAnorm.This document i

2、s the property of the World Digital Technology Academy(WDTA)and isprotected by international copyright laws.Any use of this document,including reproduction,modification,distribution,or republication,without the prior written permission of WDTA,isprohibited.WDTA is not liable for any errors or omissi

3、ons in this document.Discover more WDTA standard and related publications at https:/wdtacademy.org/.Version History*Standard IDVersionDateChangesWDTA AI-STR-011.02024-04Initial ReleaseForewordWorld Digital Technology Academy(WDTA)is dedicated to becoming a trailblazer in global digitaltech innovatio

4、n,aligned with the United Nations framework as an NGO.Upholding its 3S principleSpeed,Safety,SharingWDTA strives to accelerate the creation of digital norms,spearhead research,encourage international cooperation,and maintain leadership in technological advancement.Through collaborative efforts,WDTA

5、is dedicated to advancing digital technology for the bettermentof society.The AI STR(Safety,Trust,Responsibility)program,a core part of WDTAs internationalinitiatives,addresses the complex challenges brought about by the proliferation of AI systems.Recognizing the rapid expansion and integration of

6、AI technologies worldwide,AI STR stands at theforefront of global technological progression.This standard document provides a framework for testing and validating the security of Generative AIapplications.The framework covers key areas across the AI application lifecycle,including BaseModel Selectio

7、n,Embedding and Vector Database in the Retrieve Augment Generation designpatterns,Prompt Execution/Inference,Agentic Behaviors,Fine-Tuning,Response Handling,and AIApplication Runtime Security.The primary objective is to ensure AI applications behave securely andaccording to their intended design thr

8、oughout their lifecycle.By providing a set of testing andvalidation standards and guidelines for each layer of the AI Application Stack,focusing on securityand compliance,this document aims to assist developers and organizations in enhancing the securityand reliability of their AI applications built

9、 using LLMs,mitigating potential security risks,improvingoverall quality,and promoting responsible development and deployment of AI technologies.AI STR program represents a paradigm shift in how we approach the development and deployment ofAI technologies.Championing safety,trust,and responsibility

10、in AI systems,lays the groundwork fora more ethical,secure,and equitable digital future,where AI technologies serve as enablers ofprogress rather than as sources of uncertainty and harm.Generative AI Application Security Testingand Validation Standard is one of the AI STR standards.Founding Chairman

11、 of WDTAExecutive Chairman of WDTAAcknowledgmentsCo-Chair of WDTA AI STR Working GroupKen Huang(CSA GCR)Nick Hamilton(OpenAI)Josiah Burke(Anthorphic)Lead AuthorsKen Huang(CSA GCR)Heather Frase(Georgetown University)Jerry Huang(Kleiner Perkins)Leon Derczynski(Nvidia)Krystal(A)Jackson(University of Ca

12、lifornia,Berkeley)Patricia Thaine(Private AI)Govindaraj Palanisamy(Global Payments Inc)Vishwas Manral(Precize.ai)Qing Hu(Meta)Ads Dawson(OWASP Foundation)Amit Elazari(OpenPolicy)Apostol Vassilev(National Institute of Standards and Technology)Bo Li(University of Chicago)ReviewersCari Miller(Center fo

13、r Inclusive Change)Daniel Altman(Google)Dawn Song(University of California,Berkeley)Gene Shi(Learning-Genie)Jianling GUO(Baidu)Jing HUANG(iFLYTEK)John Sotiropoulos(Kainos)Josiah Burke(Anthropic)Lars Ruddigkeit(Microsoft)Guanchen LIN(Ant Group)Melan XU(World Digital Technology Academy)Nathan VanHoudn

14、os(Carnegie Mellon University)Nick Hamilton(OpenAI)Rob van der Veer(Software Improvement Group)Sandy Dunn(BreachQuest,acquired by Resilience)Seyi Feyisetan(Amazon)Yushi SHEN(NovNet Computing System Tech Co.,Ltd.)Song GUO(The Hong Kong University of Science and Technology)Steve Wilson(Exabeam)Swapnil

15、 Modal(Meta)Tal Shapira(Reco AI)Anyu WANG(OPPO)Wicky WANG(ISACA)Yongxia WANG(Tencent)Table of Contents1.Scope.12.Intended Audience.43.Normative References.54.Terms and Definitions.65.AI Applications Security and Validation Standards.95.1 Base Model Selection Testing Standards.95.1.1 Model Compliance

16、 and Context Testing.95.1.2 Data Usage Check Testing.115.1.3 Base Model Inference API Security Testing.155.2 Embedding and Vector Database.205.2.1 Data Cleaning and Anonymization Testing.205.2.2 Vector Database Security Testing.215.3 Prompt and Knowledge Retrieval with RAG.235.3.1 Prompt Constructio

17、n Testing.235.3.2 Prompt Templates Testing.255.3.3 External API Integration Testing(Function Calling,Plug-in).275.3.4 Retrieval from Vector Database Testing.285.4 Prompt Execution/Inference.285.4 1 LLM Application APIs Testing.295.4.2 Caching and Validation Testing.325.5 Agentic Behaviors.325.5.1 Pr

18、ompt Response Testing.335.5.2 Memory Utilization Testing.335.5.3 Knowledge Application Testing.345.5.4 Planning Capability Testing.345.5.6 Tools Utilization Testing.355.5.7 Excessive Agency Testing.355.6 Fine Tuning.375.6.1 Data Privacy Check Testing.375.6.2 Base Model Selection Testing for Fine Tun

19、ing.385.6.3 Base Model Storage Testing for Fine Tuning.385.6.4 Training Data Poisoning Testing.385.6.5 Model Deployment Testing Post Fine-Tuning.395.7 Response Handling.395.7.1 Grounding or Fact Check Testing.395.7.2 Relevance Check Testing.405.7.3 Toxicity Check Testing.415.7.4 Ethical Check Testin

20、g.415.7.5 Insecure Output Handling Testing.425.7.6 Back Door Attack Testing.425.7.7 Privacy and Copyright Compliance Check.435.7.8 Graceful Handling of Unknown or Unsupported Queries.445.8 AI Application Runtime Security.445.8.1 Data Protection Testing.455.8.2 Model Security Testing.455.8.3 Infrastr

21、ucture Security Testing.475.8.4 API Security Testing.485.8.5 Compliance and Audit Trail Testing.485.8.6 Real-time Monitoring and Anomaly Detection Testing.485.8.7 Configuration&Posture Management Testing.495.8.8 Incident Response Plan Testing.495.8.9 User Access Management Testing.505.8.10 Dependenc

22、y and Third-party Component Security Testing.505.8.11 Robust Testing and Validation.505.8.12 Availability Testing.515.8.13 Reconnaissance Protection Testing.515.8.14 Persistence Mitigation Testing:.515.8.15 Privilege Escalation Defense Testing:.525.8.16 Defense Evasion Detection Testing:.525.8.17 Di

23、scovery Resistance Testing:.535.8.18 Collection Safeguards Testing:.535.9 Additional Testing Specifications.535.9.1 Supply Chain Vulnerabilities Testing.535.9.2 Secure AI Application Development Process.575.9.3 AI Application Governance Testing.585.9.4 Secure Model Sharing and Deployment.625.9.5 Tra

24、nsparency in Decision-Making.641Generative AI Application Security Testingand Validation Standard1.ScopeThe Generative AI Application Security Testing and Validation Standard document outlines acomprehensive framework to test or validate the security of downstream AI applications,particularlythose b

25、uilt using Large Language Models(LLMs).It defines the scope of testing and validation acrossvarious layers of the AI Application Stack(Figure 1).Integrating a generative GenAI model into alarger AI-enabled system or downstream application can introduce security issues.Thus,alldownstream AI applicati

26、ons need security testing and standards validation,even if the base GenAImodel was thoroughly tested before integration into the downstream application.While this document serves as an initial version,its primary emphasis in this iteration is on LLM.However,its important to note that the scope exten

27、ds to GenAI.In subsequent versions of thisdocument,there will be opportunities to incorporate multi-modal and expansive GenAI modelsAI security testing and validation work together to ensure AI Applications behave securely and asintended.Robust methodologies should be employed through development li

28、fe cycles where feasible,using techniques like prompt injection,scanning,and red teaming exercises to identify issuesproactively.However,testing alone has limitations,especially with third-party components wheretesting may not be possible or limited.In this situation,engaging external experts or org

29、anizationsspecializing in auditing AI governance,processes,and procedures is extremely important to validatethe security of third-party components.Thoroughly auditing AI applications to check conformance tosecurity standards across all lifecycle deployment environments is critical.A thorough examina

30、tion of the downstream AI application ensures adherence to security standards,even when model-level assessments are inadequate.An integrated assurance approach with strongtesting practices plus ongoing validation of policies,processes,and performance provides assurancefor responsible AI outcomes as

31、systems continue autonomous learning.Together,they provideinformation about a systems strengths and weaknesses,inform appropriate versus inappropriate end-use applications,and assist with risk mitigation.2This specification covers the security testing of downstream applications built on top of base

32、LLMmodels but does not detail security test specifications for the base LLM models themselves.Aseparate document to be published in the future will cover security testing specifications specificallyfor base LLM models.This specification addresses the below key areas:Base Model Selection:Candidate mo

33、dels for a downstream AI application should be examinedbefore selection.This section covers verifying the base models compliance,appropriate data usage,and API security.The document provides guidance to ensure that the chosen model aligns with legal,ethical,and operational standards,a crucial step i

34、n ensuring the AI applications security.The scopeincludes both open-source and closed-source model selection.Embedding and Vector Database:These are critical components in most downstream AIapplications,storing and retrieving chunks of language data.This document outlines procedures fortesting data

35、integrity,quality,and anonymization processes to protect user privacy and comply withregulations.The specification provides guidelines for testing the confidentiality,integrity,andavailability of the vector database.Prompt and Knowledge Retrieval with Retrieval-Augmented Generation(RAG):RAG cansigni

36、ficantly improve the factual accuracy and reliability of generative AI applications,such as largelanguage models.It achieves this by dynamically incorporating relevant,domain-specific knowledgeextracted from external sources in real-time during the text generation.This section guides theconstruction

37、 of effective prompts,the creation and use of prompt templates,and the integration ofexternal APIs.It also covers testing the retrieval process from the vector database,ensuring that theAI Application can accurately access and utilize relevant information.Prompt Execution/Inference:The document deta

38、ils testing procedures for LLM APIs in the PromptExecution/Inference layer,including tests for caching mechanisms and validation processes tooptimize performance and accuracy.This layer also includes tests for checking prompts and ensuringLLMs are not being used to perform unauthorized actions,which

39、 are not allowed for the use case.Agentic Behaviors:These are advanced LLM application capabilities.The specification outlines testsfor prompt interpretation,memory utilization,knowledge application,planning,and action initiation.This includes testing the tools integrated into the AI Application to

40、enhance its capabilities securely.Fine-Tuning:The GenAI model is often fine-tuned for a specific downstream AI application.Thissection encompasses tests for data privacy,re-evaluation of the base model selection,and modeldeployment,ensuring the AIs continual improvement and relevance.3Response Handl

41、ing:This section involves testing for fact-checking of the AIs responses,relevance,toxicity,and ethical considerations to maintain the Trustworthy and security of the AIs interactions.AI Application Runtime Security:Runtime security involves the continuous,real-time monitoringof the AI Application.I

42、t covers data protection,model security,infrastructure security,andcompliance with audit trails.This ensures a comprehensive security approach,safeguarding the AIApplication against various threats and vulnerabilities throughout its lifecycle.Overall,the Generative AI Application Security Testing an

43、d Validation Standard document provides adetailed and structured approach to testing each layer of the AI Application Stack,ensuring that allaspects of an AI Application are rigorously evaluated for security and compliance.Figure 1:AI Application Stack42.Intended AudienceThe intended audience for th

44、is document is professionals and stakeholders involved in ensuring thesecurity and integrity of Generative AI Applications.This document is particularly relevant to:AI Security Engineers and Analysts:These individuals are primarily responsible for implementingand maintaining the security measures ou

45、tlined in the specification.They assess the AI application forthreats,design security architectures,and monitor systems to prevent,detect,and respond to securityincidents.These engineers also look at bias and threats.AI Developers,MLOps and AI Engineers:These are the people who build,maintain,and au

46、tomatethe workflow of AI applications.They use the security specification to understand and integratesecurity best practices into the application development lifecycle.Compliance Officers and Regulatory Experts:Professionals responsible for ensuring that AIapplications comply with the constantly evo

47、lving legal and regulatory standards use the specificationto guide compliance efforts,particularly in industries with strict data protection and privacyregulations.Data Protection Officers:These officers ensure that AI applications handle data securely and incompliance with data protection laws and

48、policies.The security specification provides them withguidelines for proper data management and protection strategies.IT and Network Administrators:These administrators are responsible for the underlyinginfrastructure of AI Applications.These professionals will use the security specification to secu

49、renetworks,servers,and other components against vulnerabilities that bad-actors could exploit in AI-related processes.Risk Management Professionals:These individuals assess and manage risks associated with AIapplications.The security specification aids them in identifying potential security risks an

50、dimplementing measures to mitigate them.Ethics Review Boards:Boards tasked with overseeing the ethical use of AI rely on securityspecifications to ensure that AI applications are ethically sound and secure against misuse or harmfulmanipulation.Project Managers and Product Owners in AI Projects:These

51、 stakeholders ensure that the AIprojects are delivered securely and efficiently.The security specification guides them in settingsecurity-related project goals and benchmarks.5Third-Party or External Security Auditors and Consultants:These experts provide an externalreview of the AI applications sec

52、urity posture.They use the specification as a benchmark to evaluatethe applications adherence to security best practices.End-Users or Business Stakeholders:While not directly involved in implementing security,end-users or business stakeholders of AI applications have a vested interest in the securit

53、y of thesesystems.Understanding the security specifications can help them gauge the reliability andTrustworthy of AI applications.Each of these groups plays a pivotal role in ensuring the security of AI applications,fromdevelopment through deployment and operation,using the Generative AI Application

54、 SecurityTesting and Validation Standard as a guiding framework.3.Normative ReferencesThe references listed below are essential for applying and understanding this document.They providefoundational theories,practices,legal frameworks,and guidelines critical to the secure andresponsible development a

55、nd deployment of AI applications:WDTA Declaration on Global AI GovernanceGenerative AI security:Theories and PracticesApplying the AIS Domain of the CCM to Generative AIEU AI ActBiden Executive Order on Safe,Secure,and Trustworthy Artificial IntelligenceNIST Trustworthy and Responsible AI NIST AI 10

56、0-2e2023NIST Artificial Intelligence Risk Management Framework(AI RMF 1.0)Chinese Generative AI RegulationChinese Approach to AI RegulationsConfidential Computing ConsortiumOWASP Top 10 for LLM ApplicationsCSA Cloud Controls Matrix(CCM v4)MITRE ATLAS(Adversarial Threat Landscape for Artificial-Intel

57、ligence Systems)NIST Secure Software Development Framework(SSDF)OWASP Top 10 API Security RisksMitigating Security Risks in Retrieval Augmented Generation(RAG)LLM ApplicationsOWASP AI Exchange64.Terms and DefinitionsFor this document,the following terms and definitions apply:Agentic Behaviors:The ca

58、pacity of LLM Applications to exhibit agency through actions likememory utilization,knowledge application,planning,and executing actions based on prompts.AI Application Runtime Security:The comprehensive security measures implemented to protect theAI Application during operations.It encompasses data

59、 protection,model security,and infrastructuresecurity.AI Governance:The framework,requirements,oversight,and accountability towards AI risk.Thesestructures enable mapping risk to organizational context(like teams,identities,priorities,and rules)while providing attestations,traceability,metrics,and o

60、versight.AI Response Handling:Involves processing and evaluating the AIs responses for accuracy,relevance,non-toxicity,privacy,confidentiality,and ethical alignment.API Security Check:Verifying the security measures of the APIs interfacing with the model,such asauthentication,authorization,and data

61、encryption to prevent unauthorized access and data breaches.Base Language Model:A base model(sometimes called a foundation model)is a large languagemodel that has already been trained and fine-tuned for general capabilities by its original modelbuilders using techniques like Reinforcement Learning f

62、rom Human Feedback(RLHF).These basemodels(e.g,OpenAIs GPT-4,Anthropics Claude 3,Googles Gemini 1.5,Cohere Command,Amazon Titan,or Metas open-sourced LLaMA2)are a strong foundation for further task-specificcustomization.Typically,developers tune the base model outputs to display broad linguisticprofi

63、ciency and adaptability to specialized use cases in a downstream application.Engineers andcompanies then take these base models as a starting point for efficiently developing and deployingcustomized AI solutions tailored to their precise needs and applications.The base model eliminatesthe need to tr

64、ain a full model from scratch,providing closed and open-sourced AI applicationlaunching pads.Base Model Selection:Choosing an appropriate base model with consideration of AI security.Selection involves evaluating factors like performance benchmarks,training data quality,potentialbiases,security proc

65、edures,potential harmful outputs,intended use cases,and regulatory compliancerequirements.Sound model provenance,transparency,auditing of data/training approaches,cross-functional review processes,and adherence to codes of conduct are vital considerations to upholdsecurity,compliance,and ethics stan

66、dards in deploying LLMs responsibly.7Caching:Techniques used to store inferred outputs from AI models to avoid repeated computationsduring inference.Since deep neural network models can be computationally expensive to run,cachingtheir outputs allows for quicker response times during live requests.Ty

67、pical solutions include cachingquestion-answer pairs for chatbots,classifications for computer vision models,or generated text forlarge language models.Caching Validation:Checking the cached outputs from AI Applications for accuracy,relevance,andsecurity before returning them to users.This can invol

68、ve confidence checks,semantic analysis,inputblocking for sensitive topics,or human confirmation.Validation works with caching to ensure reliableand secure real-time AI while benefiting from faster performance.Applying thoughtful validationmethods is essential when leveraging caching.Closed-Source Mo

69、del:A model whose weights,inference code,and training data inventory are notpublicly available.Data Cleaning and Anonymization:The removal of inaccuracies and inconsistencies from data andanonymizing personal or sensitive information to maintain privacy and compliance.Data Usage Check:Ensuring the d

70、ata used for training and operating the model is appropriate,ethically sourced,and compliant with data protection regulations.External API Integration:Incorporating external APIs into the LLM application to enhancefunctionality,such as accessing additional data sources or performing specialized comp

71、utations.Fine Tuning:The process of adjusting a based model for specific tasks or datasets to improveperformance,relevance,and compliance with data privacy(with futuristic techniques like fine-tuningfor unlearning sensitive data).LLM(Large Language Model):A large language model(LLM)is a neural netwo

72、rk trained on ahuge text corpus to generate intelligent text by predicting the next word or token,allowing open-ended text generation applications like conversational AI chatbots.Model Compliance Check:Assessing whether the chosen model adheres to legal,regulatory,andethical standards.This includes

73、considerations like data privacy laws and bias minimization.Keep inmind that compliance will change over time and do not assume that it is always a given.Also do notinfer compliance of one vendor to another.Models itself are rarely certified but their hosted solutions.Model Registry:A database,repos

74、itory,or system for storing,versioning,and cataloging MachineLearning/AI models and associated metadata(e.g.,model cards).A Model Garden is a curated versionof the Registry with the providers curated models.It normally requires a meta-data lineage betweenthe model and the used training data and infe

75、rence data points.8Prompt Construction and Templates:The creation of effective and secure prompts for the LLMand developing templates to standardize and streamline prompt generation.Prompt Processing:The process where the LLM interprets and processes a prompt to generate aresponse.This process invol

76、ves understanding the prompt,accessing relevant knowledge,andgenerating output based on prompts and contextual knowledge.RAG,or Retrieval-Augmented Generation:Retrieval-augmented generation improves the factualaccuracy of generative AI applications,like large language models,by augmenting them with

77、relevantknowledge extracted in real-time from a vector database.During inference,a retriever module firstuses the generators internal state vector to query a vector database storing external knowledge(texts,images,etc).The retrieved vectors most relevant to the generation context are then crossed wi

78、th theinternal state to produce the next generated outputs.This process dynamically grounds the modelsgeneration closer to reality,correcting false assumptions and reducing hallucinated content.Thisscalable retrieval infrastructure provides the generator with a continuous supply of relevant external

79、data during open-ended inference.This retrieval-augmented generation approach counteracts thegenerators knowledge limitations and tendency to fabricate information,thereby improving factualconsistency,security,and trust in open-domain generative AI applications.Vector Database:Vector databases serve

80、 as ground truth,help extend knowledge beyond trainingtime to run time,and reduce hallucination in generative AI models.They allow storing large volumesof real-world data(images,texts,molecular structures,etc.)as vector representations that capturesemantic concepts and features.These vector datasets

81、 then act as a reference for generative modelsduring inference to align their outputs closer to reality and avoid fabricating false details(hallucination).Retrieving the nearest vector matches from the database for generative model outputsprovides an automated way to detect and filter hallucinated c

82、ontent.This dataset conditioning iscrucial for security-critical applications of generative AI like drug discovery and content creation.Optimized vector search and scalability make databases like PG Vector,Milvus,Weaviate,andPinecone well-suited to enable such large-scale hallucination detection for

83、 deployed generative AIapplications in the real world.95.AI Applications Security and ValidationStandardsEnsuring the security and integrity of AI applications requires structured and rigorous testing acrossall components in the AI application stack.A comprehensive testing regime verifies that every

84、 aspect(from foundational model selection through to runtime security)of a downstream AI applicationfunctions securely,as intended,and without vulnerabilities.Meticulous testing specifications set clearrequirements,methods,and expected results,enabling transparent evaluations.This section providesde

85、tailed testing standards for each layer of the AI application architecture,as referenced in Figure 1:AI Application Stack.5.1 Base Model Selection Testing StandardsThe Base Model Selection is a crucial aspect of ensuring the security and compliance of an AIapplication.Selection involves distinct con

86、siderations for both open-source and closed-source models,recognizing that while closed-source models may have more readily available compliancedocumentation,open-source models might lack established compliance documents.Both,however,require thorough testing and validation.Its important to note that

87、 testing and validation of a base model is a continuous process,especially ifthe upstream base model changes.As the base model evolves and is updated,it is essential to re-validate the model to ensure that it still meets the required security and compliance standards.Thisongoing validation process h

88、elps maintain the integrity and reliability of the AI application,even asthe underlying base model undergoes modifications or improvements.5.1.1 Model Compliance and Context TestingChecking model compliance involves different methodologies for each model type,considering theirunique characteristics

89、and the availability of information.Requirement:Ensure the AI-based model,whether open-source or closed-source,complies withlegal,regulatory,security and ethical standards.10Method:For closed-source models,conduct a detailed review of available compliance documentationprovided by the vendor against

90、relevant laws,industry regulations,and ethical guidelines.To ensurecompliance,the models quality of the training data(fit for purpose),output behavior,operationalparameters,and community feedback should be reviewed and evaluated.Permission and access toclosed models may limit this evaluation.For all

91、 models,tools such as Model Cards1and DataStatements2provide baselines for model and data documentation.This may include consultations withlegal and industry experts to interpret compliance in areas lacking formal documentation.Test if themodels accuracy,relevance,consistency,and performance on a sp

92、ecific task meets predeterminedrequirements.Benchmark the model using preset scenarios and datasets to measure its performanceand the quality of its output.There are resources for testing,but their utility may vary and degrade over time.There are manypublicly available prompt security testing result

93、s for both closed-and open-sourced models.Becausethey use different test datasets,these security assessment efforts can produce different results for thesame model.Still,consulting multiple of publically available results can point out which types ofharmful behavior an LLM is more likely to show.Ben

94、chmarks will also change over time as newfailure models and attacks are discovered;consider and report the age/version of benchmarks used toassess the model.Additionally,identify and list known vulnerabilities from sources such as the MITRE Atlas()3,AVID4,the AI Risk Database5and the AI Incident Dat

95、abase6Expected Results:The base model fully meets all legal,regulatory,security,and ethical requirements,regardless of its source.Any areas of non-compliance are clearly identified for closed-source modelsand rigorously inferred and documented for open-source models.2.Checking the context metadata,l

96、ineage of training and fine-tuning of each model exists in themodel card.Model cards provide details of models used by each application.1https:/dl.acm.org/doi/10.1145/3287560.32875962https:/techpolicylab.uw.edu/data-statements/3https:/atlas.mitre.org/4https:/avidml.org/5https:/airisk.io/6https:/inci

97、dentdatabase.ai/11Requirement:Ensure the AI model,whether open-source or closed-source,has the model carddetailing the source of the model,data sensitivity,training datasets,and model custodian.Thisinformation should be accessible directly or indirectly through the model card.Method:Conduct a detail

98、ed review of available model cards for all models,including hosted models.Make sure the model card has details of the model lineage and ownership.The model card should beable to provide the data sets for both training and fine-tuning(if applicable).For closed-source models,get details of the model c

99、ard from the provider if known.Maintain and manage model cards as model stewards,ownership,datasets as these change with time.Expected Results:The base model,irrespective of its source,has complete metadata of the modelapplications.5.1.2 Data Usage Check Testing1.Protect user privacy in interactions

100、 with AI Applications.Requirement:Safeguarding User Privacy in PromptsMethod:Implement data anonymization or pseudo-anonymization techniques on user prompts forsensitive and personal data.Conduct regular audits to ensure that personal identifiers are effectivelyobscured.Also,make sure prompts and ou

101、tput are not stored more than what is specified by policy.This may limit both the content stored and the length of storage time.Conduct adversarial testing tocheck for data leaks,using methods(e.g.,continuation testing,cloze task testing,etc.)to fill in gaps.Expected Results:User prompts are process

102、ed without revealing personal identities,ensuring privacyand compliance with data protection laws.If needed,there are controls to prevent sensitiveinformation from being collected or sent to an API.2.Maintain ethical integrity and legal compliance in the handling and use of data.Requirement:Ethical

103、and Legal Use of DataMethod:Establish guidelines for data usage that align with ethical standards and legal requirements.Perform routine compliance checks and audits to monitor adherence to these guidelines.Continuouslyensure that model documentation is sufficient to assess compliance.12Expected Res

104、ults:Data from user prompts,fine tuning training data,and the vector database is usedethically and legally,with no instances of misuse or mishandling.3.Ensure adherence to international and local data protection regulations.Requirement:Adherence to Data Protection RegulationsMethod:Implement procedu

105、res for obtaining user consent,ensuring data transparency,and providingusers control over their data.Data minimization collects only necessary personal data and avoidsexcessive collection.Minimize the use and storage period of personal data through technical means.Take differentiated privacy protect

106、ion measures based on the sensitivity of the data.Regularly trainstaff on data protection laws and conduct compliance audits.Implement the process and procedures ofresponding to access to information requests and requests to be forgotten.Expected Results:Full compliance with data protection laws lik

107、e GDPR or CCPA,demonstratedthrough audit results and user feedback.4.Data provenance process using data lineage and metadata.Requirement:Ensure that datasets used to train an ML model,whether open-source or closed-source,have the data card and that the source of the data,data sensitivity,compliance

108、regime,and datacustodian of the dataset are accessible directly or indirectly through the data card.Method:Conduct a review of data cards and datasets.Verify that each datasets,especially those thathave been used to train or fine-tune models,has a data card.Make sure the data card has the details of

109、the dataset lineage and ownership.Maintain and manage data cards.The collection and content in data cards will change as data stewards,ownership,and datasets change with time.Expected Results:The base model,irrespective of its source,has complete metadata of the modelapplications.5.Obtain and mainta

110、in Data Usage Agreement between AI application developers and base modelprovidersRequirement:Data Usage Agreement between AI application developers and base model providers.This includes the following sub-test or validationSub Requirement:Identification of Parties and Scope13Method:Review the agreem

111、ent to verify that all parties are correctly identified and the scope of theagreement,including the specific base models or datasets,is clearly defined.Expected Result:The agreement accurately identifies all parties and outlines the scope,with lessambiguity about the models or datasets involved.Sub

112、Requirement:Usage Rights and LimitationsMethod:Conduct a detailed analysis of the agreement to ensure that the usage rights and limitations,including any restrictions on modifications and redistribution,are explicitly stated and understood.Expected Result:Clear understanding and documentation of the

113、 usage rights,ensuring that thelicense type(exclusive or non-exclusive)and any limitations are well-defined and adhered to.Sub Requirement:Data Handling and ComplianceMethod:Review processes for handling user prompts,fine turing training data and vector databasecontent.Check for compliance with data

114、 protection laws and procedures for data anonymization,dataresidency,and security.Expected Result:Data handling methods comply fully with the agreement and legal standards,withsecure and compliant data management practices in place.Sub Requirement:Intellectual Property RightsMethod:Verify that the a

115、greement clearly outlines the intellectual property rights concerning thebase models,the fine-tuned models,the input data,fine-tuning data,and output data.Check forcompliance in practice.Also,look at any indemnification clauses that the model provider entitles itsuser.Establish legal risks by review

116、ing the providers indemnification clauses.Expected Result:Intellectual property rights are respected,and clear guidelines are provided forusing,modifying,and redistributing model outputs.Any indemnification clauses and conditions areunderstood based on the companys needs.Sub Requirement:Confidential

117、ity and Non-DisclosureMethod:Evaluate the enforcement of confidentiality and non-disclosure provisions,particularlyregarding sensitive data.14Expected Result:Strong adherence to confidentiality obligations,with all sensitive informationprotected according to the organizations policies on the treatme

118、nt of confidential companyinformation and material nonpublic information.Sub Requirement:Liability and WarrantiesMethod:Review the organizations approach to addressing liabilities and warranties related to AIsystems.Evaluate the extent to which the organization makes good faith efforts to understand

119、 andimplement relevant terms,respond to potential model failures or data breaches,and adhere toapplicable standards and regulations in this area.Expected Result:The organization demonstrates a commitment to properly managing liabilities andhonoring warranties to the best of its abilities,with polici

120、es and processes in place that aim to handleany issues that may arise.Efforts are made to comply with relevant industry standards,best practices,and legal requirements regarding the allocation of responsibility and remediation of failures orbreaches,recognizing that perfect execution may not always

121、be possible.Sub Requirement:Termination and Renewal TermsMethod:Review the agreement for clarity on termination and renewal conditions,including noticeperiods and termination procedures.Expected Result:Well-defined terms for termination and renewal,with a straightforward process foreither party to f

122、ollow.Sub Requirement:Dispute ResolutionMethod:Examine the dispute resolution clause and assess the preparedness to engage in the outlinedprocess in case of disagreements or breaches.Expected Result:Effective mechanisms in place for dispute resolution,aligning with the agreementsterms.Sub Requiremen

123、t:Governing LawMethod:Ensure that the governing law and jurisdiction are clearly stated and understood,and checkfor any potential conflicts with local laws where the AI applications are developed or used.Expected Result:Clear understanding and compliance with the specified governing law,with nolegal

124、 conflicts.15Sub Requirement:SignaturesMethod:Verify that authorized representatives of both parties sign the agreement.Expected Result:Legally binding agreement with signatures from all relevant parties,ensuring itsenforceability.5.1.3 Base Model Inference API Security TestingThis section outlines

125、specific testing specifications for comprehensively evaluating how the clientapplication integrates with a third-party model inference API.These tests are critical when theapplication interacts with external APIs,requiring a different approach than traditional API testing.This section focuses on tes

126、ting from the client applications perspective,which is distinct from thetesting done by the API provider,as outlined in Section 5.4.1.This distinction is crucial because weare dealing with a client application that will consume third-party inference APIs.A unique testingmethodology tailored for this

127、 use case is necessary to ensure secure integration.To ensure a comprehensive and structured approach to testing the security of a client applicationintegrating with a third-party model inference API,we listed the following testing specifications.1.Authentication and Authorization:Requirement:All re

128、quests to the API must be authenticated and authorized to ensure that the clienthas both permission and appropriate access to the requested resource.Method:Simulate various authentication scenarios to test protocol implementation and key/tokenmanagementExpected Results:Clients must include a valid a

129、uthentication token in the request headers,typicallyas a Bearer token.The API should respond with appropriate status codes:200 OK if the request is successful.401 Unauthorized if the token is missing,invalid,or expired.403 Forbidden if the authenticated client lacks permissions for the requested act

130、ion.Provide clear and concise error messages to indicate the specific authorization issue,such as lack ofaccess or an expired token.2.Data Encryption:16Requirements:Encryption must be applied across all states:in-transit,at rest,and in-use.Data in-Transit Sub-Requirements:Use strong encryption proto

131、cols such as TLS 1.2 or later fordata transmitted over networks to ensure secure confidentiality and integrity.Implement perfectforward secrecy to safeguard past encrypted communications from decryption,even if long-term keysare compromised.Prior to transmission,sensitive data must be encrypted,ensu

132、ring authorizeddecryption upon arrival at the destination.Methods:Conduct thorough vulnerability assessments and penetration tests on the Base modelinference API endpoint to gauge its resilience against potential attacks.Ensure that its cryptographicconfigurations and encryption protocols are robust

133、 and impenetrable.Continuously monitor networktraffic to and from the endpoint,verifying the enforcement of stringent encryption standards and theadoption of secure protocols,such as the latest TLS version,with priority given to maintaining perfectforward secrecy.Expected Results:All transmitted dat

134、a are securely encrypted using up-to-date and secure protocolsadhering to current cryptographic standards.Encryption keys should dynamically change per sessionto prevent decryption of past sessions if future keys are compromised.Data at-Rest Sub Requirements:Implement a multi-layered security approa

135、ch to de-identifysensitive data,including tokenization,anonymization,and pseudonymization.Strong encryptionstandards like AES-256 or equivalent should be employed to securely store sensitive data,where de-identifying sensitive data or personal information is not feasible.Encryption keys should be st

136、oredseparately from encrypted data to enhance security.Stringent access controls and effective keymanagement policies must be implemented.Methods:Validate the compliance requirements based on defined testing procedures.For tokenization,assess token generation security and randomness.Assess token dic

137、tionary accesssecurity.Verify logging/auditing of all tokenization operations.Evaluate encryption and accesscontrols for token-data mapping.Test for data leakage risks in tokenized data usage,especially relatedto frequency attacks.For anonymization,validate irreversible anonymization and the inabili

138、ty to re-identify data.Check theutility of anonymized data for intended purposes.Conduct risk analysis for potential re-identificationon stylistically representative data.Review anonymization techniques used and their effectiveness.For pseudonymization,ensure pseudonym uniqueness,security,and separa

139、tion from source data.Analyze re-identification risks considering data correlations.Validate access controls,audit trails,andauthorize access.17Expected Results:Data should be adequately protected in various contexts without compromisingitsutilityforlegitimate business processes.The risk of individu

140、als being re-identified from anonymized datashould ideally fall between 0.04%and 0.1%or lower,depending on the data sensitivity.Onlyauthorized individuals are expected to be able to access and link tokens or pseudonyms to originaldata,with a full audit trail.Re-identification of individuals from ano

141、nymized or pseudonymized data must be practicallyimpossible,thus minimizing privacy risks.Data in-Use Sub-Requirements:Implement encryption for sensitive data being processed inmemory.Applications must use secure coding practices to prevent memory dumps and side-channelattacks.Least privilege princi

142、ples should be adopted to limit access to sensitive data during processing.Confidential Compute hardware or services and Confidential GPUs,which provide hardware root-of-trust as defined by the Confidential Compute Consortium,should be considered.Methods:Validate memory encryption effectiveness by a

143、ssessing applications and systems toconfirm they encrypt sensitive data effectively while in memory.Evaluate how applications handle sensitive data in memory,focusing on preventing leaks throughmemory dumps and protection against side-channel attacks.Test for vulnerabilities to side-channel attacks,

144、examining how data is processed and stored inmemory to identify potential leakage points.Review user and process access rights to ensure they are minimal and necessary,in line with theprinciple of least privilege,to reduce the risk of unauthorized access to sensitive data in memory.Trusted Execution

145、 Environments(TEEs)provide a secure execution environment,isolating sensitivedata in a protected CPU enclave during processing.Attestation is a critical part of confidentialcomputing,allowing incremental trust establishment from the root-of-Trust(RoT)to actual trust abouta system.Expected Results:En

146、sure sensitive data remains encrypted,or pseudonymised,or anonymised andsecure when being processed.Only necessary users and processes should have access to sensitive datain use,and such access must be strictly controlled and logged.Confidential computing and TEEvalidation results are expected to sh

147、ow that the environment is secured and has passed integrity tests.3.Input Moderation and Data Sanitization:18Requirement:Robust validation and sanitization of data received from the API.Method:Evaluate the applications ability to handle and sanitize various potential attack vectors,including inputs

148、that fall outside the usual use case parameters.Perform fuzzing testing,which shouldcover all functional points of the API interface,including various HTTP methods(such as GET,POST,PUT,DELETE,etc.).Perform penetration and security vulnerability testing,such as SQLinjection,cross-site scripting(XSS),

149、command injection,overflow errors,etc.Expected Result:The application effectively filters and sanitizes input,preventing injection,irrelevant inputs and other data manipulation attacks.All prompts identified as malicious are loggedand sent for analysis.4.Error Handling and Logging Security:Requireme

150、nt:Secure error handling and logging that does not expose sensitive information.Method:Trigger error conditions and analyze logs for information leakage.When possible,automatically remove sensitive information before log storage.Expected Result:Errors are handled securely without leaking sensitive d

151、ata,and logs are maintainedsecurely.5.Rate Limiting and Resource Management:Requirement:Adherence to the APIs rate limits and efficient resource management.Method:Test the application under varying load conditions to evaluate compliance with rate limitsand resource usage.Expected Result:The applicat

152、ion respects rate limits and efficiently manages API resources.6.Secret Management for API Keys and Credentials:Requirement:Securely manage API keys and credentials by storing them in a secure vault usingsecret management approaches.API keys and secrets must be rotated at regular intervals notexceed

153、ing 180 days or immediately upon indication of potential compromise.The rotation processesmust support secure key revocation and provisioning to minimize attack windows.Method:Implement a secure vault to store and retrieve API keys and credentials,ensuring they arenot exposed or leaked.Observe and v

154、alidate the secure key management process,including but notlimited to key generation,key rotation,disabling older keys,key destruction,and handling key19materials securely.Interview responsible personnel and review training materials to confirmawareness and understanding of secure API key rotation a

155、nd Secrets Management processes acrossrelevant teams.Perform sample tests by attempting to access resources using old/revoked API keysand Secrets and verifying that access is appropriately denied.Expected Result:API keys and credentials are stored securely,reducing the risk of unauthorizedaccess and

156、 data breaches.7.Dependency and Library Security:Requirement:Up-to-date and secure libraries and dependencies for API communication.Method:Perform vulnerability scans to check for outdated and deprecated components.Expected Result:All components are current and free of known vulnerabilities.8.Compli

157、ance with API Security Policies:Requirement:Full compliance with the API providers security policies.Method:Review and test the applications data handling and integration against the APIs securityprotocols.Expected Result:The application aligns with all specified security guidelines and protocols fo

158、r theAPI.9.Monitoring and Incident Response:Requirement:Effective monitoring for unusual API activity and a robust incident response plan.Method:Determine normal API behavior baseline for the use case for both int API input and output,including relevance,the typical request rates,response sizes,and

159、patterns.This baseline helps detectanomalies,especially those triggered by moderation or use case filtering APIs.Keep detailed logs anduse automated tools to analyze them for suspicious activities.Expected Result:Prompt detection of anomalies and effective execution of the incident responseplan.10.P

160、rivacy and Data Protection Compliance:Requirement:Adherence to data protection laws and privacy-by-design principles.20Method:Audit data handling practices and privacy measures in the application.Expected Result:The application complies with relevant data protection regulations and effectivelyprotec

161、ts user privacy.11.Regular Security Audits:Requirement:Ongoing security assessments focusing on API interactions.Method:Conduct periodic security audits to identify and address vulnerabilities.Expected Result:Continuous identification and timely remediation of security issues related to APIinteracti

162、ons.5.2 Embedding and Vector DatabaseFor the Embedding and Vector Database component of an AI Application,the testing specificationcan be structured as follows:5.2.1 Data Cleaning and Anonymization TestingTo ensure the integrity and privacy compliance of data used for creating embeddings by verifyin

163、g itscleanliness and effective anonymization.Requirement:Ensure that the data used for creating embeddings is cleaned and anonymizedeffectively based on use cases,especially for public-facing applications.This can be done usingtechniques like tokenization.Method:Implement tests to assess the thoroug

164、hness of data cleaning processes,ensuring thatirrelevant,redundant,or erroneous data is identified and corrected or removed.Additionally,test theanonymization processes to confirm that personal or sensitive information is effectively obscured orremoved,in compliance with privacy standards like GDPR.

165、This might involve reviewinganonymization algorithms,techniques,and their effectiveness in various scenarios.Expected Results:Data used in the embedding process is clean,relevant,and free from errors.Anonymization processes are effective in protecting personal and sensitive information,rendering itu

166、nidentifiable.215.2.2 Vector Database Security TestingEnhance the security of the vector database by implementing and verifying advanced encryption,RBAC for data access,robust key management,comprehensive IAM policies,and other criticalsecurity measures.The following are the testing specifications:1

167、.Advanced Encryption Techniques:Requirement:Evaluate the utilization of advanced encryption techniques,including end-to-endencryption,to secure data at rest and in transit.Consider the use of always encrypted databases andconfidential computing to secure data.Method:Conduct thorough assessments of c

168、ryptographic protocols and encryption standards in use,analyzing their effectiveness in protecting data.Expected Result:Enhanced data security by adopting advanced encryption methods andsafeguarding data during transmission,use,and storage.2.Key Management Lifecycle Testing:Requirement:Examine the e

169、ntire lifecycle of encryption keys,from creation to retirement,to ensureadherence to secure key management practices.Method:Test the processes for key issuance,renewal,revocation,and destruction,assessing theirrobustness and compliance with key management standards.Expected Result:A secure key manag

170、ement lifecycle that effectively protects encryption keys,reducing the risk of unauthorized access.3.Fine-Grained IAM Policy Implementation:Requirement:Implement and test fine-grained Identity and Access Management(IAM)policies thatspecify precise permissions for different user roles and scenarios.M

171、ethod:Conduct scenario-based testing to validate that each user role can only access data andperform actions according to their defined permissions.Expected Result:Granular control over access rights,ensuring that users can only performauthorized actions,enhancing data security.4.Regular Security an

172、d Compliance Audits:Requirement:Conduct frequent and comprehensive security audits that go beyond standard checks,including assessments for compliance with international standards and industry-specific regulations.22Method:Perform in-depth audits,vulnerability assessments,and compliance checks to en

173、surealignment with relevant security standards and regulations.Expected Result:Continuous improvement in security posture,with adherence to industry bestpractices and compliance requirements.5.Zero Trust Architecture(ZTA)Evaluation:Requirement:Assess the implementation of a Zero-trust security model

174、,where trust is neverimplicitly granted and must be continually validated.Method:Evaluate the deployment of the vector database within a Zero-trust environment,validatingthat access controls are consistently applied based on identity and context.Expected Result:Enhanced security through the Zero-tru

175、st model,reducing the attack surface andensuring strict access control.6.Real-Time Monitoring and Anomaly Detection:Requirement:Implement and evaluate real-time monitoring systems and anomaly detectionalgorithms to identify and respond to unusual access patterns or potential security threats in real

176、-time.Method:Test the effectiveness of real-time monitoring tools and anomaly detection algorithms bysimulating security incidents and monitoring their detection and response.Expected Result:Early detection and rapid response to security threats,minimizing potential damageand data breaches.7.Disaste

177、r Recovery and Backup Testing:Requirement:Ensure the presence of robust disaster recovery and data backup processes.Method:Test the disaster recovery and backup systems for their ability to quickly and accuratelyrestore data in the event of a breach or data loss incident.Expected Result:Reliable and

178、 efficient disaster recovery and data backup processes,minimizing dataloss and downtime in case of incidents.8.Role-based Access Control Techniques:Requirement:Evaluate whether data access in Vector databases aligns with the Role-based accesscontrols.23Method:Conduct thorough assessments of RBAC of

179、data access.Access data using different rolesand ensure only the right data is accessible to the right role.Flag any anomalous data access.Expected Result:Make sure data for Role is not seen by the other when the other role should nothave access to the data5.3 Prompt and Knowledge Retrieval with RAG

180、The testing specification for the Prompt and Knowledge Retrieval with RAG(Retrieval-AugmentedGeneration)phase of AI applications encompasses the following components:5.3.1 Prompt Construction Testing1.To verify that the prompts created for the RAG model effectively convey the intended query orcomman

181、d.Requirement:Ensure that prompts constructed for the RAG model are effective and accuratelyrepresent the intended query or command.Method:Test the prompt construction process for clarity,relevance,and completeness.This involvesevaluating a variety of prompts to ensure they effectively communicate t

182、he intended request to theRAG model and that the models responses align with the prompts intent.These tests might includeuser scenario simulations and automated testing for prompt variability.Demonstrating this requirement may be resource-intensive or require expertise depending upon thedownstream A

183、I application.Thus,organizations may need a range of approaches to demonstrate thisrequirement.For example,there are public repositories7 8and third-party companies that can helpwith requirement demonstration.Expected Results:Prompts are well-constructed,unambiguous,and effectively guide the RAG mod

184、elto provide relevant and accurate responses.2.To verify the output for the RAG model is relevant for the use case and prompt providedRequirement:Ensure the results of the RAG model are precise and relevant7https:/ different prompts for different use cases and see if the output aligns with the expec

185、tedoutput in terms of clarity and relevance.Demonstrating this requirement may be resource-intensive or require expertise depending upon thedownstream AI application.Thus,organizations may need a range of approaches to demonstrate thisrequirement.For example,there are public resources and third-part

186、y companies that can help withrequirement demonstration.Expected Results:GenerativeAI output can provide results not relevant to the use case.MAking surethe output aligns helps ensure usability,fairness and relevance of the output.3.Prompt Injection TestingRequirement:Ensure the model does not perfo

187、rm unintended actions in response to various craftedinputs(both query and injected context).Method:Test the models response to a wide range of deliberately crafted,potentially maliciousinputs.This involves simulating scenarios that might exploit vulnerabilities in input handling.The testshould inclu

188、de both direct and indirect prompt injection as documented by OWASP Top 10 for LLMApplications.Expected Results:The model consistently handles crafted inputs safely,without executingunintended actions or displaying vulnerable behavior.4.Sensitive Information Disclosure TestingRequirement:Prevent ina

189、dvertent sharing of confidential data through the models outputs.Method:Assess the models outputs for instances where sensitive or confidential information mightbe disclosed.This includes testing with scenarios that might trigger such disclosures via promptengineering,jailbreak,and various tactics a

190、nd techniques.Review academic literature and publiclyavailable independent test results assessing information leakage.If no independent testing is publiclyavailable,organizations should consider if using a different system is an option.Expected Results:The model consistently avoids revealing sensiti

191、ve or confidential information inits outputs.5.Prevent Domain Constrained chatbot from Boundary EvasionRequirement:Implement a multi-layered approach to ensure the chatbot remains within its domain.This approach requires robust fail-safe mechanisms,refined guardrails,output filtering,andspecialized

192、algorithms.25Method:Test the chatbot by providing intentional queries unrelated to its designated domain,assessing its ability to recognize and manage such situations.Simulate real-world data scenarios withanomalies and noise,ensuring the chatbot provides accurate and reliable information.Conduct A/

193、Btesting to compare the chatbots performance with a control group,offering valuable insights into itseffectiveness within the specific domain.Expected results:The chatbot demonstrated an exceptional ability to recognize queries outside itsdomain,politely acknowledging their irrelevance or guiding th

194、e conversation back on track.Thesystem reliably extracted accurate domain-specific data in simulated real-world scenarios,remainingunaffected by irrelevant or noisy information.The chatbots performance exceeded expectationsduring A/B testing,especially regarding response quality,user satisfaction,an

195、d relevance within itsdesignated domain.It diligently adhered to the implemented fail-safe mechanisms,guardrails,andspecialized algorithms,ensuring a robust and secure user experience.Users reported high satisfactionlevels with the chatbots accurate and helpful responses.5.3.2 Prompt Templates Testi

196、ngPrompt templates are predefined structures or guidelines used to generate prompts that facilitatespecific types of responses from the model.These templates are designed to streamline the interactionwith the model by providing a consistent and optimized way to phrase queries or commands,ensuringtha

197、t the model understands the intent of the user as accurately as possible.The design of prompttemplates can significantly impact the effectiveness and efficiency of the models responses,makingthem crucial for applications that require reliable and contextually appropriate outputs.The following are th

198、e testing needed for prompt templates.1.Access Control Testing with TemplatesRequirement:Ensure that the systems use of prompt templates adheres to the overall access controlpolicies,preventing the templates from being exploited to circumvent security mechanisms or accesscontrols.Method:Conduct syst

199、em-level testing to evaluate how prompt templates are accessed by differentroles/users and used within the context of the systems security and access control framework.Thisinvolves verifying that the system checks user permissions before allowing access to specifictemplates or template functionaliti

200、es,especially those that could trigger sensitive operations or accessto privileged information.The testing should simulate various user roles trying to use templates.It26should investigate if they can use the templates in ways that should be restricted,seeing if the systemcorrectly enforces access c

201、ontrols before processing the templates.Expected Results:The system ensures that all interactions with prompt templates are subject to theappropriate access controls.Users are only able to utilize templates in a manner consistent with theirpermissions,with no ability to use templates to bypass syste

202、m-level access restrictions.Attempts byusers to access or use templates beyond their authorization level are denied,demonstrating thesystems effective enforcement of access controls in relation to prompt templates.2.Template Robustness and Clarity TestingRequirement:Ensure that prompt templates are

203、robust against misinterpretation and misuse,whichcould lead to unintended or inappropriate outputs.The templates should guide users clearly,reducingthe risk of inputs that exploit ambiguities or lead to undesirable system responses.Method:Conduct thorough reviews and user testing(covering all releva

204、nt user roles)of the templatesto assess their clarity and the potential for misinterpretation.This includes evaluating the templateswith various users,including those with intentions to test the boundaries of the templateseffectiveness.The goal is to identify and correct any ambiguities or weaknesse

205、s that users couldexploit(intentionally or unintentionally)to generate responses that are unintended,inappropriate,oroutside the scope of the templates intended use.Testing should also assess the templates guidance oninput formatting and content expectations to ensure users understand how to provide

206、 inputs that leadto the desired type of response.Expected Results:The templates effectively guide users in providing inputs consistent with thetemplates intended use,with minimal risk of misinterpretation or misuse.The templates design andinstructions clearly mitigate against the potential for adver

207、sarial manipulation,ensuring that thesystems responses remain within the expected and appropriate range.User inputs and system outputsare strongly aligned,reflecting the templates effectiveness in guiding user interaction with the systemin a secure and intended manner.3.Contextual Access Control and

208、 Response Filtering for RAG ImplementationsRequirements:Implement dynamic access controls.The application must evaluate user requests based on context,including time,location,device type,and network security posture.The application must adjust userpermissions dynamically based on context,such as res

209、tricting access to sensitive data outside ofworking hours and limiting access to specific functionalities on unsecured networks.27Utilize Attribute-Based Access Control(ABAC).The application must use ABAC to manage useraccess based on various attributes,such as user role and data classification.The

210、application mustintegrate ABAC with Enterprise Identity Providers and external APIs to retrieve user attributes inreal-time.Ensure data integration and access verification.The application must securely integrate with externalsystems,verify API keys,and use scoped access tokens to limit access to aut

211、horized data.Theapplication must compare access permissions retrieved from integrated platforms with the userspermissions to ensure consistent access control.Implement contextual response filtering.The application must implement logic to filter search resultsbased on user context and permissions.The

212、 application must dynamically modify responses toexclude unauthorized data based on the users role or context.Methods:Code Review:Review application code to ensure the presence of logic for dynamic accesscontrol,ABAC implementation,and data access verification.Dynamic Analysis:Security testing tools

213、 must be used to dynamically analyze the applicationsbehavior during runtime.User requests with different contexts should be simulated to verify whetheraccess controls and response filtering function as expected.Penetration Testing:Penetration testing must be conducted to attempt unauthorized access

214、 to sensitivedata through various techniques to validate if the implemented access controls prevent unauthorizedaccess.Expected Results:Ensure sensitive information is consistently protected from unauthorized accessand leaks.Users should be able to access only the data necessary for their specific c

215、ontext and role,enhancing security while maintaining operational efficiency.The system must adapt to various usercontexts,dynamically applying the appropriate access controls and filters.The system must complywith relevant data protection laws and standards,minimizing legal and financial risks.5.3.3

216、 External API Integration Testing(Function Calling,Plug-in)External API Integration refers to the process of connecting an LLM application with external APIs toexpand its capabilities and access data or services from other systems.This allows LLMs to performtasks that go beyond their inherent knowle

217、dge and language-processing abilities28To ascertain the reliability and security of the integration between external APIs and the RAG model,ensuring seamless connectivity,accurate data exchange,and robust security measures.We need toperform the following tests.Requirement:Ensure reliable and secure

218、integration of external APIs with the RAG and LLMmodels,including management of secrets used to access the APIs.Method:Conduct testing on API connectivity,data exchange,error handling,and security.Thisincludes testing for correct function calling,data transmission accuracy,robust error and exception

219、handling,and compliance with security protocols(such as authentication and data encryption).Expected Results:External APIs integrate securely with the RAG and LLM models,exhibit reliableand secure data exchange,and handle errors effectively without compromising system performance orsecurity.Refer to

220、 sections 5.4.1 and 5.8.4 for API security,both as a client or a provider of APIs.5.3.4 Retrieval from Vector Database TestingTo guarantee that the RAG system retrieves information from the vector database accurately,efficiently,and relevantly,ensuring prompt and informed responses.Requirement:Ensur

221、e accurate and efficient retrieval of information from the vector database.Method:Test the retrieval process for relevance,accuracy,and speed.This involves querying thevector database with various inputs and evaluating the relevance and correctness of the retrievedinformation.Organizations can also

222、assess additional performance metrics,such as response time.Expected Results:The RAG system retrieves relevant and accurate information from the vectordatabase efficiently,contributing to precise and informed responses to prompts.5.4 Prompt Execution/InferenceThe testing specifications for the Promp

223、t Execution/Inference phase in AI applications,mainlyfocusing on LLM APIs and caching and validation mechanisms,can be structured as follows:295.4 1 LLM Application APIs TestingIf you have LLM application provider API to a third party,you need to conduct testing based on thefollowing testing specifi

224、cations.1.Broken Access Control Mitigation:Requirement for Authentication:Correct implementation of authentication protocols like OAuth2.0,SAML 2.0 and OpenID Connect,and secure handling of API keys and tokens.Use token-basedauthentication mechanisms such as JSON Web Tokens(JWT)to pass authenticatio

225、n information in astateless environment securely.Method:Simulate various authentication scenarios to test protocol implementation and key/tokenmanagement.If JWT Tokens are used,validate the tokens integrity by verifying the signature,checking the issuer,and ensuring the audience matches the intended

226、 recipient.Expected Result:Successful authentication processes and secure,leak-proof handling of sensitivecredentials.Requirement for Authorization:Implement comprehensive access controls to manage and restrictuser actions based on their roles and privileges.These include measures to prevent the ele

227、vation ofprivileges and enforce policy-based access.The authorization matrix must be documented in a structured and machine-readable format whilebeing easily understandable by humans for updates.It should also be designed with a hierarchicalapproach to defining various combinations of authorizations

228、,which should remain applicable acrossdifferent technological platforms and architectural frameworks of the application.Method:Validate Role-based Access Control(RBAC)or Attribute-based Access Control(ABAC)systems with correctly assigned and enforced permissions.Organizations must create an extensiv

229、e setof integration tests to verify the integrity and applicability of the authorization matrix for the testedapplication.These tests should utilize the formalized matrix directly as their input.Any test failureinstances must highlight the breached authorization combination(s).Expected Results:Contr

230、olled access that ensures only authorized API users/clients can access ormodify data based on the permitted scopes,effectively preventing unauthorized breaches.2.Protection Against Cryptographic Failures:Requirement:Employ advanced encryption for all sensitive data in transit and at rest,including t

231、heuse of industry-standard encryption protocols and regular updates to encryption keys.30Method:Utilize established cryptographic standards and robust key management practices.Expected Results:Strong encryption of data,significantly reducing the risk of unauthorized dataaccess and breaches.3.Injecti

232、on Flaw Prevention:Requirement:Protect the API from SQL,NoSQL,and command injection attacks by validating allinput data and using safe methods for database access.Please note that the injection flaw here is notprompt injection.The prompt injection is discussed in 5.3.1.Method:Implement prepared stat

233、ements,stored procedures,and thorough input validation.Expected Results:Effective mitigation of injection vulnerabilities,ensuring data integrity andsecurity.4.Insecure Design Countermeasures:Requirement:Develop the API with a security-first mindset,incorporating security measures into thedesign,con

234、ducting regular threat modeling and risk assessments.Method:Apply secure by design principles,conduct threat modeling,and integrate securitycheckpoints throughout the design and development process.Expected Results:A resilient API architecture minimizing security risks and vulnerabilities from thede

235、sign phase.5.Security Misconfiguration Management:Requirement:Systematically configure and regularly audit all security settings,keeping all systemsand software up to date with the latest security patches.Method:Use automated tools for configuration management and conduct regular security audits.Exp

236、ected Results:A well-configured API environment,minimizing vulnerability risks due tomisconfigurations.6.Handling Vulnerable and Outdated Components:Requirement:Continuously monitor and update all third-party libraries,APIs,frameworks,anddependencies to protect against vulnerable components.Method:R

237、egularly patch and update components,using vulnerability scanning tools.Expected Results:Reduced risk of security breaches due to vulnerabilities in third-party components.317.Robust Identification and Authentication:Requirement:Implement strong authentication systems,including multi-factor authenti

238、cation andsecure password policies,resistant to attacks such as credential stuffing and brute force.Method:Deploy multi-factor authentication,enforce secure password practices,and monitor forunusual authentication attempts.Expected Results:Enhanced protection against unauthorized access.8.Software a

239、nd Data Integrity Assurance:Requirement:Regularly verify the integrity of software and data processed by the API,protectingagainst unauthorized code changes and data tampering.Method:Conduct software integrity checks and data validation processes.Expected Results:Assured integrity and Trustworthy of

240、 software and data.9.Effective Security Logging and Monitoring:Requirement:Implement robust logging and monitoring systems that can detect,alert,and respondto suspicious activities or security breaches in real-time.Method:Establish comprehensive logging and continuous monitoring for unusual patterns

241、 or securityincidents.Expected Results:Early detection and prompt response to potential security issues.10.Server-Side Request Forgery(SSRF)Defense:Requirement:Guard against SSRF attacks by rigorously validating all user-supplied input,especiallyURLs or data used in server-side requests.Method:Imple

242、ment strict input validation and sanitization procedures,focusing on preventing SSRFvulnerabilities.Expected Results:Effective mitigation of SSRF risks,protecting the API from unauthorized internalnetwork access.325.4.2 Caching and Validation TestingTo assess the efficiency of caching mechanisms in

243、enhancing response times and the thoroughness ofvalidation processes in ensuring the accuracy and appropriateness of responses from LLMs.Requirement:Validate the effectiveness of caching mechanisms in improving response time and therobustness of validation processes to ensure response accuracy.Metho

244、d:Test the caching system by assessing its impact on response times for repeated queries.Thisincludes evaluating cache hit rates,data integrity in the cache,and the efficiency of cache updates.Forvalidation testing,implement checks to ensure that responses from the LLM are accurate,relevant,and free

245、 from errors or inappropriate content.This can involve automated validation checks andmanual review processes.Expected Results:The caching mechanism significantly improves response times for frequentlymade queries without compromising data integrity.Validation processes effectively ensure theaccurac

246、y and appropriateness of LLM responses,minimizing errors and inappropriate content.5.5 Agentic BehaviorsAn AI agent is a complex software system that autonomously performs tasks based on predefinedobjectives or responses to specific inputs.Central to its architecture are distinct components thatincl

247、ude a prompt mechanism,which activates the agent through instructions or questions;a memorymodule,dedicated to storing details from past conversations to inform contextually relevant responses;and a separate knowledge base,enriched with real-world,up-to-date information that the agent uses tounderst

248、and and interact with the world accurately.Additionally,a strategic planning and reflectionmodule encompasses algorithms for decision-making,enabling the agent to evaluate options,predictoutcomes,and execute actions accordingly via a set of tools.Despite the rapid evolution of AI agent technology,a

249、universal standard for their developmentremains undefined,fostering a landscape of continuous innovation.Within this evolving domain,theimportance of security cannot be overstated.As AI agents grow more sophisticated and increasinglyintegrated into diverse aspects of daily life,ensuring their resili

250、ence against threats and vulnerabilitiesis paramount,underscoring the necessity for robust security measures in the development of AI agentsto maintain trust and integrity in their operations.The testing specifications for Agentic Behaviors in AI applications can be detailed as follows,covering vari

251、ous aspects such as Prompt,Memory,Knowledge,Planning,Action,and Tools:335.5.1 Prompt Response Testing1.To confirm that the AI agent effectively and accurately interprets prompts and provides coherent,relevant,and contextually appropriate responsesRequirement:Ensure that the AI agent accurately inter

252、prets and responds to prompts.Method:Test the AI agents ability to understand and respond to a wide range of prompts,assessingthe clarity,relevance,and appropriateness of the responses.This involves evaluating the systemsnatural language understanding and generation capabilities.Please refer to sect

253、ions 5.3.1 and 5.3.2 fordetails.Expected Results:The AI agent consistently interprets prompts correctly and provides coherent,relevant,and contextually appropriate responses.2.To confirm that the AI agent can be effectively controlled and does not take autonomous actionsthat are disallowed.Requireme

254、nt:Ensure that the AI agent is not taking autonomous action,which may be disallowed.Italso asks for human approval when taking any action that could cause security concerns.Method:AI agents often have high privileges.Test the AI agents ability to access and takeautonomous actions that may be disallo

255、wed.Make sure the agent is not accessing locations,files ortaking actions that may be adversarial or used by adversaries.Also make sure AI agents ask forhuman approval before they take action and if a human disallows particular actions the agent does nottake that action.Expected Results:The AI agent

256、 constantly asks for human approval before taking any action andworks as expected.5.5.2 Memory Utilization TestingTo verify the AIs proficiency in using its memory for responding to prompts and executing tasks,ensuring accurate recall and application of previously acquired information.Requirement:Va

257、lidate the AI agents ability to effectively use its memory in responding to promptsand carrying out tasks.34Method:Test the AIs memory recall and utilization by assessing how it incorporates previouslylearned or provided information in its responses and actions.This can include testing for consisten

258、cyand accuracy in referencing past interactions or data.Expected Results:The AI demonstrates an effective use of memory,accurately recalling andutilizing relevant past information in its responses and decisions.5.5.3 Knowledge Application TestingTo ascertain the AIs capability to effectively utilize

259、 its knowledge base(in most cases,knowledgebases are composed of vector databases,graph databases,and even SQL/NOSQL databases)inproviding informed,accurate,and comprehensive responses and actions.Requirement:Ensure the AI can effectively apply its knowledge base in responses and actions.Method:Eval

260、uate the AIs use of its knowledge base by presenting scenarios or queries that requiredrawing on its stored information.The assessment should focus on the knowledges relevance,accuracy,and depth.Expected Results:The AI effectively applies its knowledge base,providing accurate and in-depthresponses a

261、nd actions informed by its accumulated information.5.5.4 Planning Capability TestingTo evaluate the AIs proficiency in planning and executing complex tasks,focusing on its strategicthinking and problem-solving abilities.Requirement:Test the AIs ability to plan and execute complex tasks.Method:Assess

262、 the AIs planning capabilities by presenting tasks or scenarios requiring actions ordecision-making steps.This involves evaluating the AIs strategic thinking and problem-solvingabilities.Expected Results:The AI demonstrates robust planning capabilities,formulating and executingeffective strategies o

263、r action plans for a variety of scenarios.355.5.5.Action Execution TestingTo ensure the AIs competency in executing actions effectively and appropriately,with an emphasison accuracy,timeliness,and suitability in various scenarios.Requirement:Validate the AIs ability to execute actions effectively an

264、d appropriately.Method:Test the AIs execution of actions in simulated environments or through predefined tasks.The focus should be on the accuracy,timeliness,and appropriateness of the actions taken by the AI.Expected Results:The AI consistently executes actions correctly,efficiently,and appropriate

265、ly inresponse to given tasks or prompts.5.5.6 Tools Utilization TestingTo confirm the AIs effectiveness in integrating and utilizing available tools,thereby enhancing itsperformance and capabilities in task execution and prompt responses.Requirement:Ensure that the AI effectively utilizes available

266、tools to enhance its capabilities.Method:Evaluate the AIs integration and use of various tools(such as databases,software libraries,or hardware devices)in performing tasks or responding to prompts.This includes testing the AIsability to leverage these tools to improve its performance or capabilities

267、.Expected Results:The AI successfully integrates and utilizes various tools,demonstrating enhancedperformance and capabilities in its responses and actions.5.5.7 Excessive Agency TestingTo critically assess and regulate the range of actions the Agent executes,ensuring they are balancedand do not lea

268、d to unintended or excessive outcomes.Requirement:Analyze and limit the extent of actions undertaken by the Agent to prevent unintendedconsequences.Method:36Scenario-based testing:Develop a wide range of test scenarios that cover various decision-makingsituations,including edge cases and potential e

269、thical dilemmas.Evaluate the AI agents responses andbehaviors in each scenario to ensure alignment with human values and intended objectives.Adversarial testing:Employ techniques such as fuzzing,input manipulation,and deliberate attemptsto break the system to identify vulnerabilities,unintended cons

270、equences,and potential failure modesin the AI agents decision-making processes.Simulation testing:Create detailed simulations of real-world environments to test the AI agentsdecision-making capabilities under realistic conditions.Monitor the agents performance,adaptability,and adherence to predefine

271、d rules and constraints.Access control testing:Implement and thoroughly test access control mechanisms to ensure that onlyauthorized users can interact with or modify the AI agents decision-making processes.This includestesting for proper authentication,authorization,and auditing capabilities to pre

272、vent unauthorizedaccess or tampering.It is crucial to give AI agents only limited access to systems and data,based onthe principle of least privilege.This means granting the agent the minimum level of access required toperform its intended functions and no more.By restricting the agents access to se

273、nsitive informationand critical systems,we can mitigate the potential risks associated with a compromised ormalfunctioning AI agent.This limited access approach should be rigorously tested to ensure that theagent cannot exceed its intended permissions or gain unauthorized access to protected resourc

274、es.Regular audits and reviews should be conducted to verify that access controls remain effective andproperly scoped as the AI agents capabilities and deployment environment evolve over time.Human-in-the-loop testing:Involve human experts in the testing process to provide oversight,guidance,and feed

275、back on the AI agents decisions.This collaboration helps ensure that the agentsactions align with human judgment and can be adjusted as needed.Continuous monitoring and evaluation:Implement mechanisms for ongoing monitoring andevaluation of the AI agents decision-making processes post-deployment.Reg

276、ularly assess the agentsperformance against established metrics,benchmarks,and human feedback to identify any deviationsor areas for improvement.Expected Results:The Agent demonstrates balanced and controlled agency,avoiding excessive orunintended actions.375.6 Fine TuningThe testing specifications

277、for Fine Tuning in AI applications,focusing on Data Privacy Check,BaseModel Selection,Model Deployment,and Training Data Poisoning Testing,can be structured asfollows:5.6.1 Data Privacy Check TestingTo guarantee that the data employed for fine-tuning the AI model strictly complies with privacy andda

278、ta protection regulations,ensuring ethical sourcing and proper anonymization.Requirement:Ensure that the data used for fine-tuning respects privacy and complies with relevantdata protection regulations.Method:Conduct a thorough review of the data collection,processing,and storage practices in the co

279、ntext offine-tuning.This includes verifying the adherence to privacy laws(like GDPR or HIPAA),ensuringdata anonymization where required,and checking for proper consent mechanisms and purpose-binding where personal data is used.Check to see if Differential privacy(DP)is used for training data privacy

280、:DP is an approach forproviding privacy while sharing information about a group of individuals,by describing the patternswithin the group while withholding information about specific individuals.It is done by makingarbitrary small changes to individual data that do not change the statistics of inter

281、est.Thus the datacannot be used to infer much about any individual.If DP is used,please use NITSs Guidelines forEvaluating Differential Privacy Guarantees to evaluate,see the link below:https:/nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-226.ipd.pdfExpected Results:The data used in the

282、fine-tuning process is fully compliant with privacyregulations,properly anonymized or pseudonymised,and ethically sourced,with all necessaryconsents obtained.385.6.2 Base Model Selection Testing for Fine TuningTo ascertain that the chosen base model aligns optimally with the specific application and

283、 fine-tuningrequirements,as detailed in section 5.1 of the document,ensuring its performance and adaptability aresuited for the intended purpose.Requirement:Confirm that the selected base model is the most suitable for the specific applicationand fine-tuning process.Please also refer to section 5.1

284、in this document.Method:Evaluate the base models performance,suitability for the target domain,and its ability tointegrate new data effectively.This can include benchmarking the model against specific performancemetrics and assessing its adaptability to the new data introduced during fine-tuning.Exp

285、ected Results:The selected base model demonstrates high compatibility with the fine-tuningobjectives,showing significant performance improvements post-tuning and suitability for the specificapplication domain.5.6.3 Base Model Storage Testing for Fine TuningTo ascertain that the fine tuned model is c

286、orrectly stored in the model registry.The fine-tuned modelmodel card is appropriately updated based on the fine tuning procedures.Requirement:Confirm that any fine tuned model is correctly stored with the proper access.Modelsare appropriately stored with the right model cardsMethod:Evaluate the fine

287、 tuned models access based on the data a model is fine tuned on.Make sureusers without permissions for a particular sensitivity of a model are not able to access the model afterits fine tuned with a higher sensitivity data.Check that the model card has the right data for the modeland is kept up to d

288、ate.Expected Results:The selected base model demonstrates high compatibility with the fine-tuningobjectives,showing significant performance improvements post-tuning and suitability for the specificapplication domain.5.6.4 Training Data Poisoning TestingTo ensure the integrity of the training data,de

289、tect and prevent tampering,biases,or corruption,thereby maintaining the models unbiased nature.39Requirement:Detects and prevents tampering or biases in training data.Method:Examine the integrity of the training data,looking for signs of tampering,insertion of biases,or other forms of corruption.Exp

290、ected Results:Training data is free from tampering and biases,ensuring the integrity andunbiased nature of the model.5.6.5 Model Deployment Testing Post Fine-TuningTo verify that the fine-tuned model operates efficiently,securely,and scales effectively in aproduction environment,maintaining high per

291、formance and robustness against security threats.Requirement:Ensure that the fine-tuned model performs effectively and securely in a productionenvironment,and does not expose confidential,sensitive,or proprietary data.Method:Test the deployed model for performance,scalability,and security post fine-

292、tuning,andproperly control input requests that would lure the model to expose confidential,sensitive,orproprietary data.This involves assessing the models response accuracy,latency,handling of high-load scenarios,and resilience to security threats in a real-world environment.Expected Results:The fin

293、e-tuned model maintains high performance and accuracy in production,scales effectively under varying loads,and exhibits robust security against potential threats.5.7 Response HandlingThe testing specifications for Response Handling in AI applications,focusing on Grounding or FactCheck,Relevance Chec

294、k,Toxicity Check,and Ethical Check,are as follows:5.7.1 Grounding or Fact Check Testing1.Fact Check TestingRequirement:Ensure that the AI Application s responses are factually accurate and grounded inreality.40Method:Implement tests to verify the factual accuracy of responses.This involves cross-ref

295、erencingAI responses with reliable data sources or established facts,particularly for responses that involveclaims of factual information.Expected Results:The AI consistently provides responses that are factually accurate and verifiable,demonstrating a strong grounding in reality.2.Feedback Loop Mec

296、hanisms Testing:Requirement:Establish and test feedback systems for users or other systems to report issues with theAI-generated content,facilitating continuous improvement.Method:Evaluate the effectiveness of feedback mechanisms in collecting user or system-reported issues.Test the process for anal

297、yzing and acting upon feedback to drive improvements.Assess the responsiveness of the AI Application to feedback and its ability to enhance contentgeneration iteratively.Expected Results:Confirmation that feedback mechanisms effectively collect and analyze user or system-reported issues.Assurance th

298、at the AI Application is responsive to feedback and demonstrates continuousimprovement in generated content.Identification and resolution of issues related to feedback handling and improvement processes.5.7.2 Relevance Check TestingTo confirm that the AIs responses are consistently pertinent and con

299、textually appropriate to the givenprompts or queries.Requirement:Validate that the AIs responses are relevant to the given prompts or queries.Method:Assess the relevance of the AIs responses by comparing them against the context andcontent of the prompts.This includes evaluating a variety of prompts

300、 and ensuring that the AIsresponses are consistently on-topic and appropriate to the query at hand.41Expected Results:Responses from the AI are consistently relevant to the prompts,demonstrating anunderstanding of the context and the specific requirements of the query.5.7.3 Toxicity Check TestingTo

301、guarantee that the AIs responses are free from toxic,offensive,or inappropriate content,upholding a high standard of conversational quality and appropriateness.Requirement:Ensure that the AIs responses do not contain toxic,offensive,or inappropriate content.Method:Conduct tests to identify and measu

302、re the presence of toxic or inappropriate language in theAIs responses.This can involve automated scanning using predefined toxicity markers,as well asmanual review by human evaluators.Depending upon the specific downstream AI application,demonstrating this requirement may beresource-intensive or re

303、quire expertise.Thus,organizations may need a range of approaches todemonstrate this requirement.For example,there are public resources and third-party companies thatcan help with requirement demonstration.What is toxic,offensive or inappropriate is highly context-dependent and will vary depending u

304、pon the specific downstream AI application and the operationalenvironment.Evaluators should consider and account for this context during testing.Expected Results:The AI Application consistently avoids generating toxic or inappropriate content,maintaining a high standard of conversational quality and

305、 appropriateness.5.7.4 Ethical Check TestingTo ensure that the AIs responses are ethically sound,free from harmful biases or stereotypes,and donot endorse unethical practices,aligning with established ethical guidelines.Requirement:Verify that the AIs responses adhere to ethical guidelines and do no

306、t promote harmfulbiases or unethical views.Method:Evaluate the AIs responses for ethical integrity,checking for biases,stereotypes,orpromotion of unethical practices.This might involve the use of ethical guidelines or frameworks asbenchmarks for assessment.Depending upon the specific downstream AI a

307、pplication,demonstrating this requirement may beresource-intensive or require expertise.Thus,organizations may need a range of approaches to42demonstrate this requirement.For example,there are public resources and third-party companies thatcan help with requirement demonstration.What is unethical is

308、 highly context-dependent and will varydepending upon the specific downstream AI application and the operational environment.Evaluatorsshould consider and account for this context during testing.Expected Results:The AI consistently provides responses that are free from harmful biases andstereotypes,

309、align with ethical standards,and do not promote unethical practices.5.7.5 Insecure Output Handling TestingRequirement:Ensure secure handling of model outputs to prevent exploitation.Method:Validate the mechanisms and processes involved in handling model outputs,checking forvulnerabilities that might

310、 lead to exploitation.Expected Results:Output handling processes are secure,effectively preventing any form ofexploitation.5.7.6 Back Door Attack TestingRequirement:Test the AI systems resilience against back door attacks,which involve maliciouslytrained models that behave normally in typical situat

311、ions but exhibit targeted misclassifications orbehaviors under specific trigger conditions.Method:Implement tests that attempt to introduce back door triggers into the AI system during training or fine-tuning.Evaluate the models behavior and outputs under these trigger conditions to detect any targe

312、tedmisclassifications or deviations from expected performance.Assess the effectiveness of defensive measures and monitoring systems designed to detect andmitigate back door attacks.Expected Results:The AI system demonstrates robust resilience against back door attacks,maintaining expectedperformance

313、 and outputs even in the presence of potential triggers.43Defensive measures and monitoring systems effectively detect and flag any attempts to introduce backdoors or suspicious model behaviors under specific conditions.The system is able to withstand or recover from back door attacks without compro

314、mising overallfunctionality,security,or integrity.5.7.7 Privacy and Copyright Compliance CheckRequirement:Ensure that the AI systems responses and outputs comply with relevant privacyregulations and copyright laws,respecting user privacy and intellectual property rights.Method:Evaluate the AI system

315、s handling of user data and personal information,verifying compliance withapplicable privacy regulations such as GDPR,CCPA,or other region-specific laws.Test the AI systems ability to protect user privacy by anonymizing or protecting sensitive informationin its responses and outputs.Assess the AI sy

316、stems respect for intellectual property rights by testing its ability to attribute contentappropriately,avoid plagiarism,and obtain necessary permissions for using copyrighted material.Utilize the Coalition for Content Provenance and Authenticity(C2PA)standard to verify theprovenance of data used in

317、 the AI system,ensuring compliance with copyright requirements andfacilitating proper attribution.Expected Results:The AI system consistently demonstrates compliance with relevant privacy regulations,properlyhandling and protecting user data and personal information in its responses and outputs.The

318、system effectively anonymizes or safeguards sensitive user information,ensuring privacy ismaintained throughout interactions.The AI system respects intellectual property rights by attributing content correctly,avoidingplagiarism,and obtaining required permissions when using copyrighted material in i

319、ts outputs.The systems responses and outputs are free from privacy violations and copyright infringements,reducing legal risks for the organization deploying the AI application.44Audits and assessments confirm the AI systems adherence to privacy and copyright requirements,providing assurance to stak

320、eholders and regulatory bodies.The AI system demonstrates the ability to adapt to updates in privacy regulations and intellectualproperty laws,ensuring ongoing compliance.The C2PA standard is successfully implemented to verify the provenance of data used in the AIsystem,enabling proper attribution a

321、nd compliance with copyright requirements.5.7.8 Graceful Handling of Unknown or Unsupported QueriesRequirement:Ensure that the AI system handles unknown,unsupported,or irrelevant queriesgracefully,providing appropriate feedback to the user.Method:Test the AI systems response to queries that are outs

322、ide its knowledge domain,unsupported,orirrelevant to the intended use case.Evaluate the systems ability to provide informative and user-friendly feedback,guiding the usertowards more appropriate queries or resources.Expected Results:The AI system gracefully handles unknown,unsupported,or irrelevant

323、queries,avoiding confusion ormisleading responses.The system provides clear and informative feedback to the user,suggesting alternative queries,offering guidance,or redirecting them to relevant resources when appropriate.5.8 AI Application Runtime SecurityThe following are the testing specifications

324、 for AI Application Runtime Security.455.8.1 Data Protection TestingTo protect sensitive data and uphold privacy standards,rigorous measures must be in place to ensuredata integrity and confidentiality.Requirement:Ensure data integrity and confidentiality.Method:Implement tests for encryption effica

325、cy,access control robustness,and continuous monitoringsystems.When emergent privacy preserving technology such as Confidential Computing or other PrivacyEnhancing Technologies(PETs)like Fully Homomorphic Encryption(FHE)are used,it is crucial tovalidate that the PET technology is correctly implemente

326、d and functioning as intended.Propervalidation of the PET implementation helps ensure the confidentiality and integrity of the data beingprocessed and the effectiveness of the privacy-preserving techniques.Without thorough validation,there is a risk that the PET solution may not be providing the exp

327、ected level of protection,potentiallyexposing sensitive data or computations to unauthorized access or manipulation.Expected Results:Data is fully encrypted at rest and in transit,access controls are effective inpreventing unauthorized access,and monitoring systems promptly detect and report any dat

328、a breachesor leaks.5.8.2 Model Security TestingProtect the fine tuned AI model from adversarial attacks and unauthorized replication with thefollowing Testing Specifications:1.Model Watermarking:Requirement:Implement watermarking techniques in the AI model to embed a unique identifierwithin the mode

329、l.This identifier should help identify the ownership and origin of the model whenreplicated.Method:Test the effectiveness of the watermarking process by attempting to replicate the model andverifying if the embedded identifier can be extracted.Additionally,Evaluate model performancedegradations(if a

330、ny)when watermarking is integrated.46Expected Result:The successful identification of model ownership and origin through the watermarkdeterred unauthorized replication.Model performance degradations,if any,do not violate orcompromise intended use or or safety or security outcomes(say healthcare deci

331、sion etc).2.Access Control and Authentication:Requirement:Enforce strict access control mechanisms and authentication protocols for accessingthe model.Method:Test user authentication processes,role-based access controls,and monitor access logs forunauthorized access attempts.Expected Result:Robust a

332、ccess control,ensuring that only authorized users can access the model,with unauthorized attempts being promptly detected and prevented.3.API Security and Rate Limiting:Requirement:Strengthen the security of APIs used to interact with the model.Method:Conduct comprehensive tests to verify the securi

333、ty of API endpoints including rate limitingto prevent mass downloading or scraping of model data.Expected Result:Secure APIs with effective rate limiting to protect against data misuse andunauthorized access.4.Code/parameter Obfuscation and Encryption:Requirement:Employ code/parameter obfuscation and encryption techniques to make the model lessintelligible and harder to replicate.Method:Test the r