《CNCF: 2024云原生人工智能白皮书(英文版)(31页).pdf》由会员分享,可在线阅读,更多相关《CNCF: 2024云原生人工智能白皮书(英文版)(31页).pdf(31页珍藏版)》请在三个皮匠报告上搜索。
1、CNCF AI Working GroupCLOUD NATIVE ARTIFICIAL INTELLIGENCEAuthorsAdel ZaaloukAlex JonesAndrey VelichkevichBoris KurktchievCassandra ChinCathy ZhangClaudia MisaleHuamin ChenJoel RobertsKai-Hsun ChenMalini BhandaruMichael YaoNikhita RaghunathPeter PanRajas KakodkarRasik PandeyRicardo AravenaRonald Pett
2、yRyan TaylorSaad SheikhShawn WilsonTom ThorleyVictor LuPUBLISHEDMARCH 20,2024(1st edition)WHITEPAPER|CLOUD NATIVE ARTIFICIAL INTELLIGENCE2EXECUTIVE SUMMARYCloud Native(CN)and Artificial Intelligence(AI)are the most critical technology trends today.Cloud Native1 technology provides a scalable and rel
3、iable platform for running applications.Given recent advances in AI and Machine Learning(ML),it is steadily rising as a dominant cloud workload.While CN technologies readily support certain aspects of AI/ML workloads,challenges and gaps remain,presenting opportunities to innovate and better accommod
4、ate.This paper presents a brief overview of the state-of-the-art AI/ML techniques,followed by what CN technologies offer,covering the next challenges and gaps before discussing evolving solutions.The paper will equip engineers and business personnel with the knowledge to understand the changing Clou
5、d Native Artificial Intelligence(CNAI)ecosystem and its opportunities.We suggest a reading path depending on the readers background and interest.Exposure to microservices2 and CN technologies3 such as Kubernetes(K8s)is assumed.For those without experience in engineering AI systems,we recommend readi
6、ng from start to finish.For those further along in their AI/ML adoption or delivery journey,per their user persona4 we suggest diving into the sections pertinent to the challenges they are grappling with or are interested in solving.We also share where society needs to invest in this context.WHITEPA
7、PER|CLOUD NATIVE ARTIFICIAL INTELLIGENCE3TABLE OF CONTENTExecutive Summary.02Table of Content.03Introduction to Cloud Native Artificial Intelligence(CNAI).04 Emergence of Cloud Native(CN).04 Evolution of Artificial Intelligence(AI).05 Merging of Cloud Native and Artificial Intelligence.06 What is Cl
8、oud Native Artificial Intelligence?.07 Why Cloud Native Artificial Intelligence?.09 Using AI to Improve Cloud Native Systems.09Challenges For Cloud Native Artificial Intelligence.10 Data Preparation.10 Model Training.12 Model Serving.13 User Experience.15 Cross-Cutting Concerns.16Path Forward with C
9、loud Native Artificial Intelligence.19 Recommendations.19 Evolving Solutions for AI/ML.20 Opportunities.21Artificial Intelligence for Cloud Native.24Conclusion .25Appendix.25 References.25 Glossary.25End Notes.28WHITEPAPER|CLOUD NATIVE ARTIFICIAL INTELLIGENCE4INTRODUCTION TO CLOUD NATIVE ARTIFICIAL
10、INTELLIGENCE(CNAI)Before we get into CNAI,the coming together of Cloud Native and AI technologies,let us examine briefly the evolution of each.The Emergence of Cloud NativeWidely known and used since 2013,5 the term Cloud Native(CN)saw an increase in popularity with the rise of container technology
11、from LXC6 to Docker7 to Kubernetes(K8s)8 Today,Cloud Native is more broadly an aspirational target of well-balanced systems built using the microservice design pattern that promotes modular design and development with a high degree of re-usability,which also lends itself to deployability,scalability
12、,and resilience.Kubernetes has evolved to become the de facto cloud operating system,with private,public,and hybrid cloud offerings.It implements a distributed orchestrator that handles network,storage,and compute resources of multiple types.Further,K8s offers an interface that enables DevOps10 best
13、 practices such as GitOps.11 Every Cloud Service Provider(CSP)has some flavor of Kubernetes as a service,facilitating access to infrastructure and a slew of support services to run various workloads,including AI/ML.The Cloud Native Computing Foundation defines9 Cloud Native as:Cloud Native technolog
14、ies empower organizations to build and run scalable applications in modern,dynamic environments such as public,private,and hybrid clouds.Containers,service meshes,microservices,immutable infrastructure,and declarative APIs exemplify this approach.These techniques enable loosely coupled systems that
15、are resilient,manageable,and observable.Combined with robust automation,they allow engineers to make high-impact changes frequently and predictably with minimal toil.The Cloud Native Computing Foundation seeks to drive the adoption of this paradigm by fostering and sustaining an ecosystem of open so
16、urce,vendor-neutral projects.We democratize state-of-the-art patterns to make these innovations accessible to everyone.Cloud Native Artificial Intelligence is an evolving extension of Cloud Native.Cloud Native Artificial Intelligence(CNAI)refers to approaches and patterns for building and deploying
17、AI applications and workloads using the principles of Cloud Native.Enabling repeatable and scalable AI-focused workflows allows AI practitioners to focus on their domain.WHITEPAPER|CLOUD NATIVE ARTIFICIAL INTELLIGENCE5Evolution of Artificial IntelligenceArtificial Intelligence,first introduced as a
18、term in 1956,12 is the ability of machines to simulate human intelligence.Over the decades,it has been used in applications such as speech recognition,machine translation,image processing,game playing,and even excelling as a Jeopardy player.13 But,AI has exploded in mindshare more recently thanks to
19、 innovations in artificial neural networks and deep learning,mainly applied to natural language understanding.There are two primary classifications of AI:discriminative and generative.Discriminative AI seeks to learn decision boundaries or classifications,with the knowledge captured as a“model,”whic
20、h is used to predict new data.For example,classifying email as spam or not,distinguishing between images of cats and dogs,and much more.Discriminative AI is typically used for tasks where the desired output is known(e.g.,via Supervised Learning,a form of machine learning).AI excels in sequence predi
21、ction,for example,guessing with a high probability what we will type next by analyzing large bodies of existing text,including our personal writing styles.Generative AI learns latent structures or representations within data.It enables synthesizing new data using these structures or representations,
22、such as creating stories,music,and visual art from word prompts.Generative AI is used for tasks where the desired output is unknown,or the“correct”output is ill-defined.With Generative AI,AI has transcended into what humans consider creative,original,and sublime.Let us take a closer look at some of
23、AIs spectacular breakthroughs.Convolutional Neural Networks14(CNNs)were first developed in the 1980s but were only widely used in the early 2000s.In recent years,CNNs have become increasingly popular thanks to their ability to learn from large datasets of images and perform well on various image pro
24、cessing tasks,such as object detection,image classification,and segmentation.Transformers were developed by researchers from the University of Toronto and Google in 2017.Transformers use a specialized mechanism called scaled dot-product attention,which imbues them with a memory-like structure.15 Tra
25、nsformer-based models are very effective for natural language processing tasks,such as answering questions,summarizing text,and translation.Consequently,they are vital in most Large Language Models(LLM).The most well-known LLM is GPT,the model that powers the popular ChatGPT service.16 WHITEPAPER|CL
26、OUD NATIVE ARTIFICIAL INTELLIGENCE6LLMs are trained on massive datasets.They take sequences of prompts,that can be long,to generate context-sensitive responses in addition to being able to be fine-tuned for specialized domains with additional data,be it current affairs,medicine,law,or others.Novel t
27、echniques for fine-tuning,such as Reinforcement Learning from Human Feedback(RLHF)and Direct Preference Optimization(DPO),have been developed to make LLMs even more compelling.Research and innovation have enabled end-user interactions that are faster,more creative,and more accurate than ever before.
28、Just as significant as the innovations in data science and software is the evolution of infrastructure to power model inference(the process of computing results from AI models)and model training(the process of building an AI model from data).With AI accelerator technology,AI practitioners can iterat
29、e faster to deliver higher-quality models in days and weeks versus months.Further,several traditional techniques employed by data scientists and statisticians are being re-evaluated to take advantage of the capabilities of CN systems.Merging of Cloud Native and Artificial IntelligenceAs mentioned in
30、 the previous section,AI is the broader concept that aims to create systems that can perform tasks akin to humans.Machine learning is a way to learn from and make informed predictions and decisions based on data.It can be thought of as yet another form of automation that involves using algorithms to
31、 learn and improve over time without explicit programming.Finally,Data Science,as a multidisciplinary field,melds techniques from statistics,mathematics,and computer science to enact a wide range of activities,from data analysis and interpretation to the application of machine learning algorithms.Th
32、inking about it broadly,we could divide applications for AI,ML,and data science into two broad categories:namely Predictive AI and Generative AI.Predictive AI aims at predicting and analyzing existing patterns or outcomes(e.g.,classification,clustering,regression,object detection,etc.).In contrast,g
33、enerative AI aims at generating new and original content(e.g.,LLMs,RAG17,etc.).As such,the algorithms and techniques underpinning predictive and generative AI can vary widely.Figure 1 Cloud Native AIWHITEPAPER|CLOUD NATIVE ARTIFICIAL INTELLIGENCE7Below is a sample set of areas where predictive and g
34、enerative AI have distinct needs across computing,networking,and storage:Challenges/NeedGenerative AI Predictive AIComputational PowerExtremely high.Requires specialized hardware.Moderate to high.General-purpose hardware can suffice.Data Volume and DiversityMassive,diverse datasets for training.Spec
35、ific historical data for prediction.Model Training and Fine-tuningComplex,iterative training with specialized compute.Moderate training.Scalability and ElasticityHighly scalable and elastic infrastructure(variable and intensive computational demands)Scalability is necessary but lower elasticity dema
36、nds.Batch processing or event-driven tasks.Storage and ThroughputHigh-performance storage with excellent throughput.Diverse data types.Requires high throughput and low-latency access to data.Efficient storage with moderate throughput.It focuses more on data analysis and less on data generation;data
37、is mostly structured.NetworkingHigh bandwidth and low latency for data transfer and model synchronization(e.g.,during distributed training).Consistent and reliable connectivity for data access.In the coming sections,we will explore how to meet the needs that arise from either form,the challenges tha
38、t come with it,and potential recommendations to employ when faced with such challenges.What is Cloud Native Artificial Intelligence?Cloud Native Artificial Intelligence allows the construction of practical systems to deploy,run,and scale AI workloads.CNAI solutions address challenges AI application
39、scientists,developers,and deployers face in developing,deploying,running,scaling,and monitoring AI workloads on cloud infrastructure.By leveraging the underlying cloud infrastructures computing(e.g.,CPUs and GPUs),network,and storage capabilities,as well as providing isolation and controlled sharing
40、 mechanisms,it accelerates AI application performance and reduces costs.Figure 2(below)maps these enabling mechanisms between tooling and techniques.WHITEPAPER|CLOUD NATIVE ARTIFICIAL INTELLIGENCE8Figure 2 Enabling Tools and Techniques18 Running AI On Cloud Native InfrastructureThe value of Cloud Na
41、tive for AI is highlighted by articles in the media published by cloud service providers and/or AI companies.19 20 The emergence of AI-related offerings by cloud providers and emerging start-ups in this space are crucial indicators of how Cloud Native principles can shape the systems necessary for t
42、he AI evolution.Cloud Native Artificial Intelligence is an evolving extension of Cloud Native.Kubernetes is an orchestration platform that can be used to deploy and manage containers,which are lightweight,portable,self-contained software units.AI models can be packaged into containers and then deplo
43、yed to K8s clusters.Containerization is especially crucial for AI models because different models typically require different and often conflicting dependencies.Isolating these dependencies within containers allows for far greater flexibility in model deployments.CN tooling allows for the efficient
44、and scalable deployment of AI models,with ongoing efforts to tailor these for AI workloads specifically.OPENAI Scaling Kubernetes to 7,500 nodesHUGGING FACE Hugging Face Collaborates with Microsoft to launch Hugging Face Model Catalog on AzureWHITEPAPER|CLOUD NATIVE ARTIFICIAL INTELLIGENCE9The Kuber
45、netes Scheduler21 continues to evolve,22 23 particularly to better integrate and support sharing Graphics Processing Units(GPUs)that have become highly popular in speeding AI workloads.Beyond supporting applications sharing a GPU and handling multi-tenancy,efforts are underway to support leveraging
46、remote pools of resources outside of Kubernetes.High-quality data is needed to train and test AI models to obtain superior inference.Cloud Native infrastructure can access data through various methods,such as data lakes and warehouses.Many cloud providers offer block,object,and file storage systems
47、that are perfect for providing low-cost,scalable storage.For example,the size of models can run into gigabytes.During the training phase,pulling the models checkpoints each time can cause a severe load on networking and storage bandwidth.Treating models as containerized artifacts opens the door for
48、hosting them in OCI24 registries and enables caching.It further allows applying software supply chain best practices to models such as artifact signing,validation,attestation,and data provenance.Additionally,containerizing models/artifacts facilitate bundling in WebAssembly(WASM)binaries.WASM is a p
49、latform-independent,efficient CN approach to inference.Why Cloud Native Artificial Intelligence?With its elastic,always-on infrastructure,the cloud has allowed enterprises,startups,and developers to prototype quickly,offer new services,scale solutions,and much more.It also does so cost-effectively t
50、hrough resource sharing.The average user no longer has to worry about ordering hardware or dealing with logistics like space,power,network connectivity,cooling,software licensing,and installation.AI has similar concerns rapid prototyping,accessing storage,networking,and computing resources to tackle
51、 small and large-scale training and inference tasks.Using AI to Improve Cloud Native SystemsWhether packaged as observability tooling or leveraging LLM capabilities for natural language processing(NLP)of logs,AI-powered solutions/projects are entering the hands of operators and end-users to enhance
52、their productivity and make their lives easier.One such open source Cloud Native Computing Foundation(CNCF)project is K8sGPT,25 which leverages the pattern recognition and language capabilities of LLM,such as Bedrock,Cohere,and others,to aid K8s operators in their daily work.More significantly thoug
53、h,the symbiosis of CN and AI opens up the ecosystem to new and unforeseen opportunities.For example,we expect a rise in less technical users able to operate and manage complex systems.WHITEPAPER|CLOUD NATIVE ARTIFICIAL INTELLIGENCE10CHALLENGES FOR CLOUD NATIVE ARTIFICIAL INTELLIGENCEIts important to
54、 note that CNAI challenges will vary between the different personas.26 And,while Cloud Natives flexible,scalable platform is a promising fit for AI workloads,AIs scale and latency needs pose challenges and expose gaps in CN technologies while also presenting opportunities.We tease these out in the c
55、ontext of an end-to-end ML pipeline.27 also referred to in the literature as MLOps.28 Issues with the traditional trade-offs of time and space,parallelism,and synchronization all surface,exposing ease-of-use gaps.To summarize,the ML Lifecycle looks as follows:The data volumes involved in training,si
56、milarity search,and model size,particularly with LLMs,each drive memory and performance considerations.While CN handles access control and scheduling for CPUs,GPU allocation with adequate sharing is still evolving.The ML training phase is all about search,requiring tracking the performance of interm
57、ediate models to determine which to keep and how to tune model parameters further to obtain even greater accuracy.Security is more critical given the sensitivity of the handled data and the models intrinsic value.Observability is vital to detect model drift,usage load,and more.Let us dive a little d
58、eeper into the challenges in each pipeline stage.The reader is encouraged to consider additional challenges related to their domain and add to the conversation.29 Figure 3 ML LifecycleThe typical ML pipeline is comprised of:Data Preparation(collection,cleaning/pre-processing,feature engineering)Mode
59、l Training(model selection,architecture,hyperparameter tuning)CI/CD,Model Registry(storage)Model Serving Observability(usage load,model drift,security)Data PreparationAs the first phase in an AI/ML pipeline,data preparation can present various challenges.These can be broadly grouped into three main
60、categories:managing large data sizes,ensuring data synchronization during development and deployment,and adhering to data governance policies.WHITEPAPER|CLOUD NATIVE ARTIFICIAL INTELLIGENCE11Data SizeThe demand for data to build better AI/ML models is increasing faster than Moores Law,doubling every
61、 18 months.30 Whether its data management/handling,data processing,or data analysis,there is a rapid escalation in data demands for building AI/ML models.Therefore,distributed Cloud Native computing and efficient data movement and storage become essential to bridge the gap between these computationa
62、l demands and hardware capabilities.Data SynchronizationData may need to be sourced from multiple disparate locations in different formats;the developer and production environments,more often than not,are different,and all this is in addition to handling the increased complexity arising from distrib
63、uted computing,such as partitioning and synchronization.Let us take a closer look at the latter.In data processing systems like Spark,the industry-standard interface,SQL,plays a crucial role in providing users with a familiar uniform experience,whether they are prototyping locally or running large w
64、orkloads in a distributed manner.However,ML workloads dont have an industry-standard interface.Consequently,data scientists develop their ML Python scripts with small datasets locally,and then distributed systems engineers rewrite these scripts for distributed execution.If the distributed ML workloa
65、ds do not function as expected,data scientists might need to debug the issues using their local Python scripts.This process is inefficient and often ineffective.This is true despite the availability of better observability tools and the reproducibility afforded by container technology.Potentially vi
66、able solutions exist for resolving this inconsistency between local development and production environments.The first is using an industry-standard interface to support the end-to-end ML lifecycle.For example,users can leverage APIs of native ML frameworks like PyTorch or TensorFlow to create traini
67、ng code and validate it by running it locally in a Python runtime.Then,users can easily reuse the same code and leverage the Python SDK from Kubeflow to run this code locally in a distributed fashion via Kind/Minikube or just as easily scale their training code by deploying it to a remote,large-scal
68、e Kubernetes cluster using the same Python SDK.Another option is to use a general-purpose distributed computing engine such as Ray,whose computational abstractions also enable users to run the same Ray scripts seamlessly in local and production environments.Data volume is a cross-cutting issue.It al
69、so manifests in the training stage.Data GovernanceData governance is crucial to building trust and ensuring responsible AI development.One should consider three critical pillars regarding data governance.1.Privacy and Security:It is essential to navigate the complex landscape of data privacy regulat
70、ions such as GDPR31 and CCPA32.Robust security measures should be implemented to safeguard sensitive data used in AI models.Encryption,access controls,and regular vulnerability assessments should be used to protect valuable information.2.Ownership and Lineage:It is imperative to clearly define who o
71、wns and has access to the data throughout the AI lifecycle,from collection to use.Data lineage tracking tools should be utilized to understand how data flows through the system,ensuring transparency and accountability.Doing so helps to prevent unauthorized access and misuse of sensitive information.
72、WHITEPAPER|CLOUD NATIVE ARTIFICIAL INTELLIGENCE123.Mitigating Bias:AI models are only as good as the data they are trained on.Hence,it is essential to actively monitor and address potential biases in the data and algorithms.This includes using diverse datasets,employing fairness metrics,and continuo
73、usly evaluating the model to ensure it delivers fair and ethical outcomes,including capturing its limitations.Model Cards33 are evolving to capture these.Data privacy and security is a cross-cutting issue that requires consideration at every stage.Model TrainingModel training data volumes have risen
74、 exponentially,resulting in a need for distributed processing and accelerators to achieve even more parallelism.Further training is an iterative multi-step process,which makes scaling a complex multi-component coordinated task.We review these aspects in greater detail in this section.Rising Processi
75、ng DemandsLLMs are rapidly pushing the boundaries to meet the growing AI/ML training and inference computing demands,and accelerators are becoming popular.These range from GPUs from multiple vendors with different capabilities to Googles tensor processing units(TPUs),Intels Gaudi,and even field-prog
76、rammable gate arrays(FPGAs).These varied compute resources need virtualization support,drivers,the ability to configure and share them,and CN scheduler enhancements.Further,these accelerators limited availability and cost have prompted the exploration of multi-cloud resource banding,and even sky34 c
77、omputing.Using CN technology for AI can be complex regarding GPU virtualization and dynamic allocation.Technologies,such as vGPUs,MIG,MPS(see glossary),and Dynamic Resource Allocation(DRA),35 enable multiple users to share a single GPU while providing isolation and sharing between containers in a po
78、d.They can increase GPU utilization,which in turn reduces costs,in addition to allowing multiple workloads to benefit simultaneously from them.However,implementation requires careful orchestration and management,especially when allocating and deallocating resources dynamically.Close collaboration be
79、tween the AI and CN engineering teams is necessary to ensure smooth and efficient integration.Cost EfficiencyThe elasticity and scalability inherent in Cloud Native environments allow organizations to provision and scale resources dynamically based on fluctuating demands.This aspect also applies to
80、AI tasks.However,resource proper sizing and reactive scheduling to meet varying workload demands are even more compelling in the context of accelerators such as GPUs,which are expensive and limited in supply.It drives the need to be able to fractionalize GPUs to utilize them better.Reducing the carb
81、on footprint during model serving can be achieved using an autoscaling serving framework,which dynamically adjusts resources based on demand.36 KServe,37 an LF AI&Data Foundation project,provides such functionality.Sustainability 38 can be significantly improved by various means,such as using smalle
82、r,more specialized models,using a mixture of experts,and techniques such as compression and distillation.Distributing ML serving into geographical regions powered by renewable or cleaner energy sources can significantly reduce carbon footprint.39Responsible development of ML models can include metad
83、ata on carbon footprints to aid in tracking and reporting the impact of model emissions on the environment.Additional tooling,such as mlco240 and codecarbon41 exists,with limitations,to help predict the carbon footprint of new neural networks before physical training.WHITEPAPER|CLOUD NATIVE ARTIFICI
84、AL INTELLIGENCE13ScalabilityAI/ML workflows are complex and characterized by diverse components that run in a distributed environment.In the context of training,this complexity is particularly exacerbated by the data volumes being handled and the need to support multiple rounds of training until mod
85、el convergence.Coordinating the scaling of various microservices,with each encapsulating specific AI functionalities,demands intricate orchestration to ensure seamless communication and synchronization.Furthermore,the heterogeneity of AI models and frameworks complicates standardization,making creat
86、ing generic scaling solutions applicable across various applications challenging.Orchestration/SchedulingAs alluded to earlier,Cloud Native tools and projects simplify the orchestration and scheduling of AI workloads by leveraging the inherent features of containerization,microservices,and scalable
87、cloud infrastructure.Complex AI workflows can be decomposed into modular components,making it easier to manage and scale specific functions independently.However,as mentioned earlier,GPUs are a precious and in-demand resource,and the ability to more efficiently manage their sharing and scheduling fo
88、r GPU-based AI workloads is critical to the success of AI development teams.Well-tested tools for addressing advanced scheduling needs like bin packing,placement,resource contention,and pre-emption will be essential and foundational for cloud native AI to thrive.Better scheduling support is evolving
89、 in Kubernetes through efforts such as Yunikorn,42 Volcano,43 and Kueue,44 the latter two addressing batch scheduling,which is particularly valuable for efficient AI/ML training.Training jobs benefit from gang(or group)scheduling,45 as the container replicas belonging to the job need an all-or-nothi
90、ng placement policy to function correctly,and those jobs are not easily scaled up or down.Gang scheduling support is an area of opportunity.Custom DependenciesAI applications often rely on specific frameworks and versions of libraries,and these dependencies may not be readily available or compatible
91、 with standard container images.Since many AI workloads benefit from GPU acceleration,having the necessary GPU drivers and libraries to support running workloads on GPUs can be challenging,especially when dealing with different vendors and GPU architectures.For example,when running distributed train
92、ing on NVIDIA devices,one can use NVIDIA Collective Communications Library(NCCL),to take advantage of optimized multi-GPU and multi-node communication primitives.Different versions of the library might lead to different performance.Reproducible builds,a good build hygiene practice for all software,r
93、equire using versioned dependencies to avoid runtime incompatibilities and performance surprises.Model ServingModel serving differs chiefly from data processing and training because of load variability and often latency requirements.Further,there are considerations of service resiliency in addition
94、to sharing infrastructure to reduce costs.Also,AI model characteristics are distinct,varying significantly across classical ML,Deep Learning(DL),Generative AI(GAI)LLMs,and,more recently,the multi-modal approaches(e.g.,text to video).Different workloads necessitate varied support from ML infrastructu
95、re.For example,before the emergence of LLMs,model serving typically required only a single GPU.Some users opted for CPU-based inference if the workloads were not latency-sensitive.However,when serving LLMs,the performance bottleneck shifts from being compute-bound to memory-bound due to the autoregr
96、essive nature of the Transformer decoder.46WHITEPAPER|CLOUD NATIVE ARTIFICIAL INTELLIGENCE14 This section explores how CN supports these facets and what challenges remain.Microservice Architecture and Developer ExperienceCN is based on microservice architecture.However,this may pose a challenge for
97、AI,dealing with each stage in the ML pipeline as a separate microservice.Many components may make maintaining and synchronizing their outputs and hand-offs challenging.Even if users only want to play with these solutions on their laptops,they might still need to create tens of Pods.The complexity ma
98、kes the infrastructure lack the flexibility to adapt to versatile ML workloads.Second,the microservice-based ML infrastructure leads to a fragmented user experience.For example,in their daily workflows,AI Practitioners may need to build container images,write custom resource YAML files,use workflow
99、orchestrators,and so on instead of focusing solely on their ML Python scripts.This complexity also manifests as a steeper learning curve,requiring users to learn many systems outside their expertise and/or interest.Third,the cost increases significantly when integrating each stage from different sys
100、tems in the ML model lifecycle.The Samsara engineering blog47 mentions that its ML production pipelines were hosted across several microservices with separate data processing,model inference,and business logic steps.Split infrastructure involved complex management to synchronize resources,slowing th
101、e speed of development and model releases.Then,using Ray,Samsara built a unified ML platform that enhanced their production ML pipeline performance,delivering nearly a 50%reduction in total yearly ML inferencing costs for the company,stemming chiefly from resource sharing and eliminating serializati
102、on and deserialization across stages.These issues highlight the need for a unified ML infrastructure based on a general-purpose distributed computation engine like Ray.Ray can supplement the existing Cloud Native ecosystem,focusing on computation,allowing the Cloud Native ecosystem to concentrate on
103、 deployment and delivery.The Ray/KubeRay community has collaborated extensively with multiple Cloud Native communities,such as Kubeflow,48 Kueue,49 Google GKE,50 and OpenShift.51 Model PlacementUsers ideally like to deploy multiple,possibly unrelated,models for inference in a single cluster while al
104、so seeking to share the inference framework to reduce costs and obtain model isolation.Further,for resiliency,they want replicas in different failure zones.Kubernetes provides affinity and anti-affinity mechanisms to schedule workloads in different topology domains(e.g.,zone,node),52 but usability i
105、mprovements can help users take advantage of these features.Resource AllocationModel serving requires handling,chiefly,the model parameters.The number of parameters and the representation size indicate the memory needed.Unless dealing with a trillion parameter LLM,these typically require only a port
106、ion of a GPU.This highlights the need to be able to fractionalize expensive accelerators like GPUs.The DRA project,53 which is still in alpha,seeks to make GPU scheduling more flexible.Another consideration is response latency,which depends significantly on the use case.For instance,the response lat
107、ency desired to detect objects on the road in an autonomous driving context is several orders lower than tolerable while creating an image or writing a poem.Additional serving instances may need to be launched for low-latency applications under high-load conditions.These could land on a CPU,GPU,or o
108、ther computing resource if the desired latency can be honored.Support for such cascading opportunistic scheduling on available resources is still evolving in Kubernetes.WHITEPAPER|CLOUD NATIVE ARTIFICIAL INTELLIGENCE15Further,event-driven hosting is ideal for not wasting resources and keeping costs
109、down.The Kubernetes Event Driven Autoscaling(KEDA)54 project is well-suited here,provided the model loading latency is tolerable to still deliver on the end-to-end service latency.An opportunity here is to provide better support for model sharing by delivering models in an Open Container Initiative5
110、5(OCI)format,an immutable file system that lends itself to sharing.Another solution is to use AI for CN,in particular,to predict use and proactively float or shut down serving instances to handle the expected load.User ExperienceThe hallmark of CN,aka containers,allows portability and reproducibilit
111、y,while Kubernetes APIs and operators,like Kubeflow,simplify the deployment of the AI workloads,making them“write once and run(virtually)anywhere in an easily scalable fashion.Once users transition from traditional batch systems on bare metal or virtualized environments to containers and Kubernetes,
112、they appreciate the benefits of cloud technologies despite their initial adoption challenges.The learning curve,however,can be steep.Lets consider AI training workloads.Configuring the runtime environment can be time-consuming,particularly when highly customizable libraries are used.The user has the
113、 option to use default settings for a plethora of environment variables,but these may yield inferior performance.Once optimized on a given Kubernetes platform for a particular training workload,there are no guarantees it will perform likewise on another platform or training task or container bundle
114、with different libraries included.This affects workload portability and ease of use.The previous paragraph looked at just one stage in an AI pipeline,typically multi-stage,spanning data preparation,training,tuning,serving,and fine-tuning.How can one provide a seamless user experience for AI practiti
115、oners who arent necessarily savvy with systems or cloud concepts and provide them with a streamlined product experience that eliminates friction in AI development?Giving AI practitioners user-friendly and well known SDKs written in Python that abstract away the complex details of Kubernetes can help
116、 increase the adoption of Cloud Native AI tools.Users would like to build ML models using PyTorch and TensorFlow and then quickly and easily deploy them to Kubernetes infrastructure by using simple Python SDKs without worrying about details such as packaging,building Docker images,creating Kubernete
117、s custom resources(e.g.,PyTorchJob,TFJob),and scaling those models using complex cloud native tools.A strong product development focus will be required to invent an open source product experience for the MLOps lifecycle,which is much more user friendly.Integrating tools like JupyterLab,which contain
118、s space for an IDE-like experience with useful APIs that may exist in AI/ML tools available today(ex.,Kubeflow Katib API),would allow ML practitioners to more quickly iterate on their AI development with fewer context switches across multiple user interfaces.JupyterLabs extensible nature gives ML pr
119、actitioners a workspace to build,deploy,and monitor AI/ML workloads within a familiar tool without learning new tools and interfaces.It is even possible to use JupyterLab to schedule workflows of code developed in individual AI/ML Notebooks using GUI workflow building tools like Elyra56 coupled with
120、 Kubeflow Pipelines.Big Data,inside and outside the enterprise,is a mainstay of AI.It is essential to consider how to bridge the gap between the Big Data and ML ecosystems.For example,modern Generative AI models require large amounts of data for training.Still,the tools for loading large amounts of
121、data from formats like Iceberg into training frameworks like PyTorch require enhancement,with tools like TorchArrow57 and PyIceberg58 demonstrating early promise.Tools used for large-scale data preparation,like Spark,arent well connected to the tools in the ML ecosystem.Extra overhead is required to
122、 prepare data,build features,store features to disk,and then read those features back into memory for use in training workloads.Solutions like RayData59 or a data caching microservice built upon Arrow Flight RPC may significantly improve the Input/Output overhead involved with the first phases of tr
123、aining workloads.WHITEPAPER|CLOUD NATIVE ARTIFICIAL INTELLIGENCE16ML tools are complex,and users typically need help to deploy them on Kubernetes.It is nontrivial to identify and deploy appropriate drivers for GPUs and make them compatible with a users AI/ML workloads.The upgrade path for existing M
124、L workloads should be simplified and improved,similar to other Kubernetes control plane components.Users should get clear guidelines on how to keep their AI workloads resilient to Kubernetes upgrades and cluster downtime.Another aspect that affects the ease of use is multi-tenancy,using quotas and n
125、amespaces.Non-admin users need help to figure out the system resources available to them.Typically,administrators provide tools(e.g.,Grafana dashboards)for observability;when these are lacking,non-expert/non-admin users are left in the lurch.Finally,debugging is challenging,made more so in distribut
126、ed environments and even more so when the processing pipeline comprises multiple complex services.Hardware and software failure might be more or less explicit and easy to identify to a cloud user,but an AI practitioner may need help to see the complete picture of failure.For example,NCCL termination
127、 errors can be vague with any of a multitude of possible causes,each requiring investigation.The user may need to parlay the error message to an administrator for further assistance.Cross-Cutting ConcernsIn the previous sections,we addressed challenges specific to a stage in the AI pipeline.But othe
128、rs are common to all stages and all software applications,spanning reference implementations,observability,security,and more.For instance,right-sizing resources are valid for processing data,training,or serving.It has resource utilization,cost,and sustainability ramifications.Let us dive a little de
129、eper into them.Reference ImplementationNeither cloud nor AI are easy studies,and getting them to work together after making choices from many tools and projects is non-trivial.Adoption needs to be improved by requiring a reference implementation that meets a majority of simple use cases.Kind for Kub
130、ernetes did wonders to help developers get started on their laptops.Jupyter Notebook did likewise for the budding AI/ML developer.We need something similar for an AI/ML pipeline that runs in the cloud.Right-sizing Resource ProvisioningAI/ML workloads are resource intensive,especially with LLMs with
131、their billions or trillions of parameters.As discussed earlier,accelerators like GPUs are expensive and in short supply,and it is essential to use the proper size allocation to save resources and control costs.We need to be able to not only timeslice GPUs but also slice or partition them into fracti
132、onal sections and allocate them judiciously as required by different workloads.In conjunction with the above back-end effort,there is a need for front-end support to request GPU sub-units and configure them while launching workloads.To address this need,Kubernetes introduced a new API,Dynamic Resour
133、ce Allocation(DRA),60 61 as alpha in v1.26.The API provides more flexibility to manage specialized hardware resources,in particular:Network-attached resources Arbitrary parameters for resource requests Arbitrary,resource-specific setup and cleanup actions Custom matching resource requests with avail
134、able resources,including handling optional requests.WHITEPAPER|CLOUD NATIVE ARTIFICIAL INTELLIGENCE17 The DRA API offers several advantages compared to existing approaches:-Custom hardware can be added by developing and deploying DRA drivers without needing to modify the core Kubernetes codebase-Ven
135、dors can define resource parameters-Resources can be shared between containers and podsCost ControlAI/ML can quickly become a budget black hole.Automating resource allocation and scaling processes to optimize AI cloud costs is essential.Microservices can be scaled individually as needed.Further,it l
136、ends itself well to using the Kubernetes auto-scaling feature that will further help right sizing the number of active instances and thus the infrastructure costs.Last,Spot Instances can be leveraged with policies that capture balancing risk with meeting Service Level Agreements(SLAs).ObservabilityO
137、bservability is valuable across the AI/ML pipeline.CN offers tools like OpenTelemetry62 and Prometheus63 that can monitor load,number of accesses,response latency,and more.It is vital to monitor model performance and health in production environments.It is crucial to keep track of model drift to ens
138、ure the accuracy and reliability of your AI system.For example,facial recognition systems may experience degradation as more people wore masks during the COVID-19 pandemic.Similarly,a housing price predictor model may diverge from reality due to external factors such as natural disasters or changes
139、in interest rates.Therefore,monitoring your AI models continuously is essential to detect any performance issues and make necessary adjustments.Infrastructure monitoring is essential,especially with long running workloads.As AI training workloads run,anomalies in GPUs and networking may happen at ti
140、mes.Examples are errors in the GPU memory or unreachable nodes,which may result in the job crashing.However,issues that are not immediately identifiable may arise:for instance,training performance may start to degrade without any apparent hardware fault being reported.In these cases,only deep diagno
141、stics could identify the issues.Current metrics do not expose results from deep diagnostics.Therefore,providing tools to detect,avoid,and handle infrastructure issues before,during,and after running AI training jobs becomes crucial.Disaster Recovery and Business ContinuityAll production services mus
142、t be resilient,with backups.AI services are no different.Failed or slow to respond services can cause reputational damage and loss of revenue.Developing a comprehensive disaster recovery plan is essential,which may include data backup,running instances in multiple availability zones,and running mult
143、iple instances.Policies can help with these.Security and Compliance AuditsAll outward facing services,particularly Model Serving instances,need firewall protection,access control,and more.And like any other service,your AI/ML workloads must follow security best practices.These include penetration te
144、sting,vulnerability scanning,and compliance checks of the workload domain,such as health care,finance,etc.WHITEPAPER|CLOUD NATIVE ARTIFICIAL INTELLIGENCE18Tools like Grype64 and Trivy65 can scan containerized workloads for vulnerabilities.Kyverno66 and policy enforcement services can ensure containe
145、rized workloads are running at the lowest privilege necessary with minor capabilities needed.An additional layer of security is possible using confidential computing67 or Trusted Execution Environments(TEE).These hardware-supported environments provide encrypted memory,data integrity protection,and
146、testability.TEEs protect the data and workload from other infrastructure users while in use.AMD,Intel,NVIDIA,and IBM have TEE offerings,and they are becoming available in public clouds.Protecting sensitive data such as health care and financial information and ML models are prime use cases.Sustainab
147、ilityAI/ML model training has always been resource intensive,especially with Large Language Models like GPT-3.Training emissions are comparable to multiple transcontinental flights,while inference emissions add up due to high query volumes.68 The industrys trend towards oversized models for market d
148、ominance leads to inefficiencies,contributing to energy and resource consumption.69 More transparency and standardization in reporting the environmental impacts of a model are challenges.Recently,there have been efforts to increase transparency with LLama,70 while some insights are becoming availabl
149、e concerning water usage for cooling servers running LLMs,like ChatGPT.ChatGPTs carbon footprint is significant,given its millions of users.The drive for sustainability presents opportunities for innovation.DeepMinds BCOOLER and smaller,more efficient models like DistilBERT and FlexGen show promise
150、in reducing AI/ML energy consumption.71 Adopting best practices like efficient ML architectures,optimized processors,and locating cloud computing infrastructure in energy-efficient locations can curb the carbon footprint of ML training.Google has been successful in controlling the energy consumption
151、 of its machine learning systems.Education for KidsToday,technology education mainly focuses on traditional programming languages without AI or computer assistance.Schools typically dont use modern IDEs that support refactoring,templating,or API assistance and will have students code on a contained
152、website for ease of setup.They also dont teach the use of AI coding assistance technologies like Githubs Copilot,even though this will become the standard mode of development in the future.Most students arent even aware this technology exists.Schools actively dissuade students from using AI technolo
153、gies like ChatGPT and Copilot due to concerns about cheating.This prevents students from learning how to use AI technologies to augment their work and excel effectively.Because schools paint AI technology in a negative light,studious students get scared off from using it,and the students looking for
154、 a way to avoid doing their homework are more likely to use AI.The challenges mentioned above provided us insight into areas of concern when it comes to implementing CNAI systems.Fortunately,CN tooling is facing many challenges head-on.We next consider opportunities that stem from these challenges.W
155、HITEPAPER|CLOUD NATIVE ARTIFICIAL INTELLIGENCE19PATH FORWARD WITH CLOUD NATIVE ARTIFICIAL INTELLIGENCEThis section provides a forward looking approach to taking the initiative to implement CNAI.We begin with recommendations(or actions),then enumerate existing yet evolving solutions(i.e.,CNAI softwar
156、e),and finally consider opportunities for further development.RecommendationsFlexibilitySometimes,the variety of options regarding AI can become overwhelming.Fortunately,thanks to many,popular tools and techniques remain valid in this new world.From REST interfaces for interface to cloud based resou
157、rces and services,CN technologies work well today and will continue to work well as new offerings evolve.SustainabilityImproving the accountability of AI workload environmental impact is crucial for ecological sustainability,particularly in the cloud native landscape.This can be achieved by supporti
158、ng projects,methodologies,and taxonomy that help clarify,classify,and catalyze AI workload on ecological sustainability.Additionally,integrating cloud native technologies to optimize AI workload scheduling,autoscaling,and tuning is necessary.Furthermore,advocating for adopting standardized methodolo
159、gies in environmental impact assessments is vital.It is also important to promote the development and use of energy-efficient AI models and foster transparency in model development and usage,primarily through cloud native stacks such as Kubeflow.Finally,emphasizing the importance of purposeful and e
160、fficient AI usage will help minimize unnecessary computational loads.Custom Platform DependenciesWe recommend ensuring the Cloud Native environment has the required GPU drivers and supports GPU acceleration for AI workloads.This is crucial as AI applications often depend on specific frameworks and l
161、ibrary versions that may not be easily accessible or compatible with standard container images.This will help with the challenge of having various vendors and GPU architectures.Reference ImplementationGiven the number and complexity of the tools involved in AI development,it may be advisable to cons
162、ider the value of a Cloud Native,OpenTofu-based reference implementation of a user-friendly combination of various tools that can provide a product-like experience for any team around the world to get started doing AI/ML in the Cloud quickly.Combining the best available open source tools for data pr
163、eparation,feature store,training,tuning,model registry,and serving can help teams get started doing machine learning quickly and scale up their work efficiently using the power of the Cloud.Consider the value/power of combining a sophisticated set of technologies into a functional and scalable distr
164、ibution to serve such a purpose.(e.g.JupyterLab,Kubeflow,PyTorch,Spark/Ray/Trino,Iceberg,Feast,MLFlow,Yunikorn,EKS/GKE,S3/GCS,etc.).Such a reference implementation may be extremely valuable for advancing open and responsible AIML development powered by Cloud-based technologies.WHITEPAPER|CLOUD NATIV
165、E ARTIFICIAL INTELLIGENCE20Industry Acceptance of TerminologyAs AI becomes ubiquitous,it becomes increasingly complex in some dimensions but simpler in others.For example,terminology evolves,providing businesses with more effortless conversations about AI(e.g.,terms such as“repurpose”to reuse existi
166、ng content).This also applies to more technical terms,such as RAG,Reason,and Refinement.Evolving Solutions for AI/MLThe following are just a few examples of specific tools or technologies that have become options to enable AI,including CNAI.Orchestration-KubeflowKubeflow is an example of a CNAI tool
167、 supporting ML Operations(MLOps).Using technologies such as Kubernetes,stateless architectures,and distributed systems,Kubeflow helps AI/ML communities adopt Cloud Native tools more efficiently.The successful adoption of Kubeflow highlights the successful integration of Cloud Native technologies for
168、 AI/ML/DL.Kubeflow has been highly progressive in its ability to apply machine learning concepts to elastic substrates provided by Kubernetes,with many other projects following suit.72 Kubeflow follows Kubernetes best practices and applies them to the AI/ML space,such as declarative APIs,composabili
169、ty,and portability.Kubeflow implements individual microservices for every stage of the ML lifecycle.For example,Kubeflow Training Operator is used for distributed training,Katib is used for hyperparameter tuning fine-tuning,and Kubeflow KServe is used for model serving.That allows users to integrate
170、 individual Kubeflow components into their ML infrastructure or use Kubeflow as an end-to-end ML platform.Context-Vector DatabasesLLMs are trained with vast volumes of,typically,publicly available data at a point in time.We interact with them via prompts.But to make the responses more valuable witho
171、ut the user having to enter longer or multiple prompts and possibly retrieve more domain-specific responses,it is helpful to“enrich”the prompt.This is where vector databases come in.They are giant,indexed stores of vectors,a mathematical representation of data in numerical form.Embeddings are a spec
172、ific vector representation of each additional piece of data,often proprietary,domain specific,or newer,that aims to capture relationships and similarities(context)between the data they represent.The user-provided LLM prompt is transformed using the same embedding used by the vector database,and the
173、resulting vector is then used to find similar vectors in the database.They are then merged to provide additional context before feeding into the LLM to generate a response.Multi-modal GenAI systems would handle prompts that might be text,images,audio,or other,with the embedding ability to handle div
174、erse input.Vector databases can be purpose-built or traditional databases with extensions to handle vectors more specifically.Instances may vary in their choice of indexing scheme,distance metric used to compute similarity,and whether and what data compression technique they employ.Some offerings in
175、clude Redis,73 Milvus,74 Faiss,75 and Weaviate.76Observability-OpenLLMetryOpenLLMetry77 is a project that builds on top of OpenTelemetry78 to enable thorough and vendor-neutral instrumentation for LLM Observability.Because Generative AI is not debuggable in the traditional sense(i.e.,you cant“just s
176、tep through the code”),developers must turn towards Observability tools and practices to improve their use of Generative AI over time.This data is also often the source of evaluations and fine-tuning workflows.WHITEPAPER|CLOUD NATIVE ARTIFICIAL INTELLIGENCE21OpportunitiesCNCF Project LandscapeSevera
177、l Linux Foundation(LF)groups,including CNCF,LF AI79&Data,along with partners such as the AI Alliance,80 and more,provide a hub for AI projects that both AI and cloud engineers can use.Existing tools,such as the Cloud Native Landscape,81 give a birds eye view into the CN ecosystem.The following figur
178、e lists established and evolving projects grouped by their functional area.Figure 4 ML Tool to Task Mind MapWHITEPAPER|CLOUD NATIVE ARTIFICIAL INTELLIGENCE22CNAI for Kids and StudentsKids already use AI assistive technologies like ChatGPT daily and have no idea how they work.The underpinnings of mod
179、ern AI,like discriminative and generative AI algorithms,are a black box that kids and even technology savvy parents dont understand,so it is difficult to take an interest in it.Rather than just taking LLMs like ChatGPT for granted,students education should include the basics of neural networks and m
180、achine learning algorithms to explain how AI technologies work and how to use them better in their future careers.The Cloud Native community and successful programs like CNCF Kids Day82 at KubeCon provide educational opportunities on Cloud Native and AI technologies.Introducing kids to AI technologi
181、es early will also prevent the diversity,equity,and inclusion issues plaguing computer science.AI is an equalizing technology because people of every race,sexual orientation,and socioeconomic status can experience AI/ML daily and help improve this technology with the proper training and education.Th
182、e AI/ML revolution is analogous to the dot-com era,where web technology became ubiquitous,and even ordinary workers embraced this technology to improve their business.As AI/ML technology becomes ubiquitous in society,we must ensure that students keep pace with the advances in AI and Cloud Native tec
183、hnologies.ParticipationAs AI grows,more opportunities for education and involvement happen.There is room for AI specialists(e.g.,Ph.D.in ML to Data Scientists)and AI generalists(e.g.,operators and end-users).Educational programs such as MOOCs83 and certifications have emerged to focus on AI tooling
184、and techniques on all fronts.Professional societies(e.g.,ACM84 and IEEE85)and meetups provide chances to meet in person to learn and discuss challenges.Industry groups such as the CNCF,86 along with Linux Foundation AI,AI Alliance,87 and others,provide the ability to coordinate projects and protocol
185、s at scale.Trust and Safety/Safety By DesignAs we build AI and Cloud Native technology,there is a significant risk of unintended consequences and negative impacts.These can be due to unintentional design issues causing adverse impacts on vulnerable groups,for example,recommending algorithms that ina
186、dvertently promote hate-based,violent,extremist material.They can also be due to individuals or groups malicious use of systems and/or tools to harm deliberately,such as using Generative AI tools to create misinformation and disinformation campaigns or individuals purposely fine-turning LLMs to prod
187、uce child sexual abuse material.88 AI and Cloud Native technology are also at the core of the tooling used by Trust and Safety:“The field and practices employed by digital services to manage content and conduct scans for risks to users and others,mitigate online or other forms of technology-facilita
188、ted abuse,advocate for user rights,and protect brand safety.”89 Systems have been built to deliver every part of the Trust and Safety cycle90 including identifying and assessing potentially violent behavior,triaging and prioritizing cases,making and recording enforcement decisions,selecting and appl
189、ying interventions,and gathering threat intelligence.Apart from being central to the safety and health of the internet,these systems can have significant negative impacts if designed without due consideration.Responsible technology is about reducing harm from technology,diversifying the tech pipelin
190、e,and ensuring that technology aligns with the public interest.It explores and actively considers techs values,unintended consequences,and negative impacts to manage and mitigate risk and harm.As we build AI and Cloud Native technology,we must consider these potential ethical and human rights impact
191、s,WHITEPAPER|CLOUD NATIVE ARTIFICIAL INTELLIGENCE23optimizing freedom of expression,the right to privacy,the right to life,liberty,and the security of person,91 and other fundamental universal human rights.The World Economic Forum states:“Safety by Design puts user safety and rights at the center of
192、 the design and development of online products and services”.92 This proactive and preventative approach focuses on embedding safety into the culture and leadership of an organization.It emphasizes accountability and aims to foster more positive,civil,and rewarding online experiences for everyone.Th
193、ere is a growing field of experts to help with these development best practices,such as the Global Internet Forum to Counter Terrorism(GIFCT),93 The Tech Coalition,94 and the Internet Society.95 All Tech is Human curated list of experts in this sector and can provide links to critical resources.96 T
194、he AI Alliance 97 initiative(IBM,Meta,and 50+institutions)focuses on advancing open innovation and science in AI to propose alternatives to closed AI systems and advance the field of responsible AI(ethics,trust,safety).OpenAI,98 the organization behind ChatGPT,was initially founded as a non-profit f
195、ocusing on guaranteeing safety and fairness in AI.The Emergence of a New Engineering DisciplineIn the last two decades,we have seen how the tech industry has been creating and changing engineering job roles rapidly,depending on their responsibilities.We have witnessed the rise of roles such as DevOp
196、s Engineer,SRE Engineer,and Infrastructure Engineer.We foresee the MLDevOps or AI engineer becoming the glue between Data Science,Infrastructure and Development in the next few months or years.Its important to know that this industry area is developing,and the role titles can fluctuate;only time wil
197、l tell.Different terms may also become a reality.In the future,that role will need to focus more on AI tooling,infra,and deploying AI chains and agents.WHITEPAPER|CLOUD NATIVE ARTIFICIAL INTELLIGENCE24ARTIFICIAL INTELLIGENCE FOR CLOUD NATIVEThis paper has focused mainly on Cloud Native supporting AI
198、 development and usage.But AI can enhance Cloud Native in many ways from anticipating load and better resource scheduling,particularly with multiple optimization criteria involved,such as power conservation,increased resource utilization,reducing latency,honoring priorities,enhancing security,unders
199、tanding logs and traces,and much more.Natural Language Interface for Cluster ControlAt Cloud Native AI+HPC Day in Chicago in 202399 Kubernetes Controllers with a natural language interface were demonstrated to tackle cluster-related tasks.It used an LLM in that back-end that comprehended user reques
200、ts and translated them to Kubernetes API calls.It further supported launching chaos tests to ascertain service resiliency,scan for CVEs,and more.It is a precursor to more intuitive orchestration and management of Kubernetes clusters and,in time,lowers the learning curve for administrators and site r
201、eliability engineers.SecurityMachine learning can analyze massive datasets to rapidly identify patterns and predict potential threats or weaknesses in the system.Integrating AI in red teaming100 accelerates identifying security gaps and allows organizations to strengthen their defenses against emerg
202、ing cyber threats.ML models that detect anomalous network behavior can just as easily be used in clusters to protect workloads or across a fleet of clusters for edge deployments.Smarter Orchestration/Scheduling AI can analyze historical cluster usage over the day/week/month to identify workload patt
203、erns and resource availability,to understand when and how to deploy workloads,whether to scale them horizontally or vertically,when to consolidate workloads on a few nodes to put others into quiescence for power savings or even drop them from the cluster to reduce costs.ML-driven models can optimize
204、 task sequencing,automate decision-making processes,and enhance the overall efficiency of workload management.A natural language interface facilitates the whole orchestration and scheduling process.These enhancements would make it easier for organizations to manage and schedule complex workflows in
205、dynamic cloud environments.Processor power models are being built to help plan and optimize for reduced power consumption.AI Integration Efforts in Flight and Under Exploration Fine-tuned custom LLMs to analyze logs.MLOps pipeline to capture and maintain data provenance.AI semantic conventions to CN
206、CF projects like OpenTelemetry.101 AI-powered development environments(IDEs)are used to develop and deploy AI applications.We expect to report on advances in this space in the not-too-distant future.WHITEPAPER|CLOUD NATIVE ARTIFICIAL INTELLIGENCE25CONCLUSIONAPPENDIXCombining Artificial Intelligence(
207、AI)and Cloud Native(CN)technologies offers an excellent opportunity for organizations to develop unprecedented capabilities.With the scalability,resilience,and ease of use of Cloud Native infrastructure,AI models can be trained and deployed more efficiently and at a grander scale.This white paper de
208、lves into the intersection of these two areas,discussing the current state of play,the challenges,the opportunities,and potential solutions for organizations to take advantage of this potent combination.While several challenges remain,including managing resource demands for complex AI workloads,ensu
209、ring reproducibility and interpretability of AI models,and simplifying user experience for non-technical practitioners,the Cloud Native ecosystem is continually evolving to address these concerns.Projects like Kubeflow,Ray,and KubeRay pave the way for a more unified and user-friendly experience for
210、running AI workloads in the cloud.Additionally,ongoing research into GPU scheduling,vector databases,and sustainability offers promising solutions for overcoming limitations.As AI and Cloud Native technologies mature,organizations embracing this synergy will be well-positioned to unlock significant
211、competitive advantages.The possibilities are endless,from automating complex tasks and analyzing vast datasets to generating creative content and personalizing user experiences.By investing in the right talent,tools,and infrastructure,organizations can leverage the power of AI and Cloud Native techn
212、ologies to drive innovation,optimize operations,and deliver exceptional customer experiences.This paper brought to you by the CNCF AI Working Group.References1.Glossary2.ReferencesGlossaryAI PractitionersIn the context of this paper,it refers to(not limited to)ML Engineers,Data Scientists,Data Engin
213、eers,roles whose primary responsibilities include manipulating relevant data,creating,and optimizing machine learning models.DevelopersIn the context of this paper,it refers to(not limited to),Software Engineers,Frontend Engineers,Backend Engineers,Full Stack Engineers,Software Architects,and Softwa
214、re Testers.The roles whose primary responsibility include writing and testing software including user interfaces,microservices,and backend software.WHITEPAPER|CLOUD NATIVE ARTIFICIAL INTELLIGENCE26DeployersIn the context of this paper,it refers to(not limited to),DevOps Engineers,Site Reliability En
215、gineers,Infrastructure Engineers,Infrastructure Architects,Application Administrators,Cluster Administrators.The roles whose primary responsibility include deploying software and cloud infrastructure to multiple environments including development,staging and production.DRADRA stands for Dynamic Reso
216、urce Allocation.It is an API abstraction of general resource claim and provisioning for Pods,allowing 3rd party vendors to provide HW/SW resources on demand without having to rewrite the Kubernetes core API.LLM“LLM”stands for“Large Language Model.”Large language models are artificial intelligence mo
217、dels trained on vast amounts of text data to understand and generate human-like text.LLMs are a subset of machine learning models specifically designed for natural language processing(NLP)tasks.LLMOpsLLMOps,which stands for Large Language Model Operations,encompasses the operational aspects tailored
218、 specifically for Large Language Models(LLMs).In essence,LLMOps is the adaptation of MLOps principles and tools to the unique requirements of LLM-powered applications,encompassing their entire lifecycle from development to deployment and maintenance.MIGMulti-Instance GPU technology is an innovation
219、that allows a single physical GPU(Graphics Processing Unit)to be partitioned into multiple more minor instances,each operating as an independent GPU with its own resources and capabilities.This technology enhances GPU utilization and flexibility in data center and cloud computing environments.MLOpsM
220、LOps,short for machine learning operations,refers to the practices,methodologies,and tools used to streamline and automate machine learning models deployment,monitoring,and management in production environments.MLOps aims to bridge the gap between machine learning development and operations,ensuring
221、 that ML models are deployed efficiently,reliably,and at scale.It involves a combination of software engineering principles,DevOps practices,and specialized tools to automate the end-to-end ML lifecycle,including data preparation,model training,model deployment,monitoring,and maintenance.MLOps helps
222、 organizations accelerate their ML projects,improve model performance,and maintain consistency and reliability across the ML pipeline.MPSMPS stands for Multi-Process Service in the context of GPU computing.MPS technology allows multiple GPU-accelerated applications or processes to share a single phy
223、sical GPU while maintaining isolation and efficient resource utilization.RAGWHITEPAPER|CLOUD NATIVE ARTIFICIAL INTELLIGENCE27In the context of AI,RAG stands for“Retrieval-Augmented Generation.”Its a model architecture combining retrieval-based and generative models to produce text.RAGs generation pr
224、ocess is augmented with a retrieval mechanism that helps the model access relevant information from an extensive database or knowledge base.This retrieval component allows the model to incorporate external knowledge into the generation process,improving the quality and relevance of the generated tex
225、t.vGPUvGPU,or Virtual Graphics Processing Unit,technology enables multiple virtual machines(VMs)to share a single physical GPU(Graphics Processing Unit).This technology efficiently utilizes GPU resources in virtualized environments such as cloud computing,data centers,and virtual desktop infrastruct
226、ure(VDI).WHITEPAPER|CLOUD NATIVE ARTIFICIAL INTELLIGENCE28REFERENCES1 https:/ https:/en.wikipedia.org/wiki/Microservices3 https:/cf.io/guide4 https:/ First release of Docker March 20,2013.6 https:/en.wikipedia.org/wiki/LXC7 https:/en.wikipedia.org/wiki/Docker_(software)8 https:/ https:/ as of Jul 18
227、,20210 https:/en.wikipedia.org/wiki/DevOps11 https:/ https:/ai100.stanford.edu/2016-report/appendix-i-short-history-ai13 https:/youtu.be/P18EdAKuC1U?si=Dd74AdpbF3EgzVmn14 https:/www.cs.toronto.edu/hinton/absps/NatureDeepReview.pdf15 https:/arxiv.org/abs/2008.0221716 https:/ https:/en.wikipedia.org/w
228、iki/Prompt_ engineering#Retrieval-augmented_generation18 https:/ https:/ https:/huggingface.co/blog/hugging-face-endpoints-on-azure21 https:/kubernetes.io/docs/concepts/scheduling-eviction/kube-scheduler/22 https:/ https:/kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/24 https:/opencontainers.
229、org/25 https:/k8sgpt.ai/26 https:/ https:/ https:/ https:/cloud- https:/ https:/gdpr-info.eu/32 https:/oag.ca.gov/privacy/ccpa33 https:/iapp.org/news/a/5-things-to-know-about-ai-model-cards/34 https:/arxiv.org/abs/2205.0714735 https:/kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-a
230、llocation/36 Open Source for Sustainability37 https:/ 2112.06905 GLaM:Efficient Scaling of Language Models with Mixture-of-Experts39 A carbon-aware workload dispatcher in multi-cluster Kubernetes environments for Cloud WHITEPAPER|CLOUD NATIVE ARTIFICIAL INTELLIGENCE29 Native Sustainability Week 2023
231、|IBM Research40 https:/mlco2.github.io/impact/41 https:/codecarbon.io/42 https:/yunikorn.apache.org/43 https:/volcano.sh/44 https:/kueue.sigs.k8s.io/45 https:/en.wikipedia.org/wiki/Gang_scheduling46 https:/arxiv.org/abs/1706.0376247 https:/ https:/ https:/kueue.sigs.k8s.io/docs/tasks/run_rayjobs/50
232、https:/ https:/ https:/kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/53 https:/kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/54 https:/keda.sh/55 https:/en.wikipedia.org/wiki/Open_Container_Initiative56 https:/ https:/pytorch.org/torcharrow/beta/index.
233、html58 https:/py.iceberg.apache.org/59 https:/docs.ray.io/en/latest/data/data.html60 https:/kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/61 https:/ https:/opentelemetry.io/63 https:/cf.io/projects/prometheus/64 https:/ https:/ https:/ https:/en.wikipedia.org/wiki/Confi
234、dential_computing68 https:/ https:/ https:/arxiv.org/pdf/2302.13971.pdf71 https:/ https:/landscape.lfai.foundation/73 Redis74 Vector database-Milvus75 facebookresearch/faiss:A library for efficient similarity search and clustering of dense vectors.()76 _77 https:/ https:/opentelemetry.io/79 https:/l
235、faidata.foundation/80 https:/thealliance.ai/81 https:/cf.io/82 https:/events.linuxfoundation.org/kubecon-cloudnativecon-europe/program/kids-day/#kids-day83 https:/www.mooc.org/WHITEPAPER|CLOUD NATIVE ARTIFICIAL INTELLIGENCE3084 https:/www.acm.org/85 https:/www.ieee.org/86 https:/cf.io/87 https:/thea
236、lliance.ai/88 CSAM-child sexual abuse material89 https:/dtspartnership.org/wp-content/uploads/2023/01/DTSP_Trust-Safety-Glossary13023.pdf90 https:/ https:/www.ohchr.org/documents/publications/guidingprinciplesbusinesshr_en.pdf92 https:/www.weforum.org/projects/safety-by-design-sbd/93 https:/www.gifct.org94 https:/www.technologycoalition.org/95 https:/www.internetsociety.org/96 https:/alltechishuman.org/responsible-tech-organizations97 https:/thealliance.ai/news98 https:/ https:/ https:/en.wikipedia.org/wiki/Red_team101 https:/ YOU!