《From the Cluster to the Cloud-Lateral Movements in Kubernetes.pdf》由会员分享,可在线阅读,更多相关《From the Cluster to the Cloud-Lateral Movements in Kubernetes.pdf(86页珍藏版)》请在三个皮匠报告上搜索。
1、Yossi Weizman&Ram Pliskin,MicrosoftFrom the Cluster to the Cloud:Lateral Movements in KubernetesAgendaIdentity types in KubernetesInner-cluster lateral movementCluster-to-cloud lateral movement:Azure,AWS and GCPDetections&mitigationsKey takeawaysIdentity types in KubernetesThree main areas:How users
2、(or applications)from outside the cluster authenticate with the cluster.How workloads in the cluster authenticate within the cluster.How workloads in the cluster authenticate with resources in the cloud outside the cluster.Identity types in KubernetesThree main areas:How users(or applications)from o
3、utside the cluster authenticate with the cluster.How workloads in the cluster authenticate within the cluster.(Inner-cluster lateral movement)How workloads in the cluster authenticate with resources in the cloud outside the cluster.(Cluster-to-cloud lateral movement)Inner-cluster lateral movementInn
4、er-cluster lateral movementLets assume a pod is compromisedAPI ServerK8s control planeetcdscheduler Controller managerNode 1Pod APod BNode 2Pod CPod DNode 3Pod EPod FAPI ServerK8s control planeetcdscheduler Controller managerNode 1Pod APod BNode 2Pod DNode 3Pod EPod FPod CAPI ServerK8s control plane
5、etcdscheduler Controller managerNode 1Pod APod BNode 2Pod CPod DNode 3Pod EPod FAPI ServerK8s control planeetcdscheduler Controller managerNode 1Pod APod BNode 2Pod DNode 3Pod EPod FPod CAPI ServerK8s control planeetcdscheduler Controller managerNode 1Pod APod BNode 2Pod CPod DNode 3Pod EPod FAPI Se
6、rverK8s control planeetcdscheduler Controller managerNode 1Pod APod BNode 2Pod CPod DNode 3Pod EPod FInner-cluster lateral movementHow can attackers leverage a compromised pod for cluster takeover?API ServerK8s control planeetcdscheduler Controller managerNode 1Pod APod BNode 2Pod CPod DNode 3Pod EP
7、od FPods A service account tokenKubeletscredentialsInner-cluster lateral movementHow can attackers leverage a compromised pod for cluster takeover?Good news it becomes more difficult:Read secrets permission isnt enough Newer versions of K8s dont create long-lived SA tokens as secret objects Node tak
8、eover cluster takeover Node authorizer+NodeRestriction admission controller limit the permissions of the nodes identitiesInner-cluster lateral movementHow can attackers leverage a compromised pod for cluster takeover?Good news it becomes more difficult:Read secrets permission isnt enough Newer versi
9、ons of K8s dont create long-lived SA tokens as secret objects Node takeover cluster takeover Node authorizer+NodeRestriction admission controller limit the permissions of the nodes identitiesHowever,common misconfigurations still allow it.Inner-cluster lateral movementHow can attackers leverage a co
10、mpromised pod for cluster takeover?Good news it becomes more difficult:Read secrets permission isnt enough Newer versions of K8s dont create long-lived SA tokens as secret objects Node takeover cluster takeover Node authorizer+NodeRestriction admission controller limit the permissions of the nodes i
11、dentitiesHowever,common misconfigurations still allow it.Lets see an exampleInner-cluster lateral movementExample:Self-update permissionsWeve observed cases in which applications had permissions to update themselves.This allows the applications to change their own configuration.For example:update th
12、eir configuration to be privileged,change their service account,schedule to specific node(s).Effectively,this can lead to cluster takeoverAPI ServerK8s control planeetcdscheduler Controller managerNode 1Pod APod BNode 2Pod CPod DNode 3Pod EPod FChange the application configurationAPI ServerK8s contr
13、ol planeetcdscheduler Controller managerNode 1Pod BNode 2Pod CPod DNode 3Pod EPod FPodAPI ServerK8s control planeetcdscheduler Controller managerNode 1Pod BNode 2Pod CPod DNode 3Pod EPod FPodPodAPI ServerK8s control planeetcdscheduler Controller managerNode 1PodPod BNode 2Pod CPod DNode 3Pod EPod FP
14、odPodAPI ServerK8s control planeetcdscheduler Controller managerNode 1Pod BNode 2Pod CPod DNode 3Pod EPod FPod Pod Pod Inner-cluster lateral movementOther sensitive permissions(partial list):PermissionAllows toCreate podcontrollersUse a privileged service account in a new podUpdate controllerChange
15、the configuration to use a privileged service accountCreate secrets&getlist secretCreate a new long-lived token for SA and read its valueCreate serviceaccounts/tokenCreate a short-lived token for SACluster-to-cloud lateral movementCluster-to-cloud lateral movementHow tightly coupled are Kubernetes c
16、lusters with their Cloud hosting environment?1.Managed clusters use cloud services for their ongoing operation(e.g.,VMs allocation)2.Workloads in Kubernetes may need access to cloud resources(e.g.,cloud storage,cloud secret store,etc.)Cluster-to-cloud lateral movementHow do workloads in Kubernetes a
17、uthenticate with the cloud?Well go over the following authentication methods:1.Storing cloud identity credential on the node2.Direct access to IMDS3.Indirect access to IMDS4.Using OIDC(identity federation)Cluster-to-cloud lateral movementStoring cloud identity credential on the node Used to be the d
18、efault authentication method in AKS in the past.In this method,each Kubernetes node stores a file with service principal(SPN)credentials.SPNs are Azure application identities.By default,this SPN has Contributor role for the node resource group.Users can bring their own SPN or grant more permissions
19、to the SPN if their applications need access to more cloud resources.For example:add permissions to a cloud storage.Cluster-to-cloud lateral movementStoring cloud identity credential on the node Used to be the default authentication method in AKS in the past.In this method,each Kubernetes node store
20、s a file with service principal(SPN)credentials.SPNs are Azure application identities.By default,this SPN has Contributor role for the node resource group.Users can bring their own SPN or grant more permissions to the SPN if their applications need access to more cloud resources.For example:add perm
21、issions to a cloud storage.Access to the nodes File System meant elevation to Contributor Role on the K8s resource groupAPI ServerK8s control planeetcdscheduler Controller managerNode 1Pod APod BNode 2Pod CPod DNode 3Pod EPod FKubernetes Service AccountAzure Service principalAPI ServerK8s control pl
22、aneetcdscheduler Controller managerNode 1Pod APod BNode 2Pod CPod DNode 3Pod EPod FKubernetes Service AccountAzure Service principalAPI ServerK8s control planeetcdscheduler Controller managerNode 1Pod APod BNode 2Pod CPod DNode 3Pod EPod FKubernetes Service AccountAzure Service principal“Pod create”
23、operationIn the pod config:mount the SPN into the containerAPI ServerK8s control planeetcdscheduler Controller managerNode 1Pod APod BNode 2Pod CPod DNode 3Pod EPod FKubernetes Service AccountAzure Service principalBackdoorpodAPI ServerK8s control planeetcdscheduler Controller managerNode 1Pod APod
24、BNode 2Pod CPod DNode 3Pod EPod FKubernetes Service AccountAzure Service principalBackdoorpodCluster-to-cloud lateral movementDirect access to IMDSCluster-to-cloud lateral movementDirect access to IMDS The metadata service is a special endpoint that is accessible to VMs,allowing them to retrieve inf
25、ormation about the VM.Implemented by all major cloud providers.Metadata service allows retrieving tokens for the cloud identity that is attached to the VM:Azure:169.254.169.254/metadata/identity/oauth2 AWS:169.254.169.254/latest/meta-data/iam/security-credentials GCP:metadata.google.internal/compute
26、Metadata/v1/instance/service-accounts Querying the metadata service doesnt require any authentication.Cluster-to-cloud lateral movementDirect access to IMDS In managed K8s clusters,the nodes are VMs which can access to their metadata service.By default,pods can access to the metadata service of thei
27、r nodes.Thus,pods can acquire tokens of cloud identities attached to the nodes.The permissions of the identities depend on the cloud provider and the specific environment.Cluster-to-cloud lateral movementDirect access to IMDS-AKS AKS uses several managed identities to operate the cluster.Users can c
28、hange the default permissions of those identities,or alternatively attach additional managed identities to the nodes.Cluster-to-cloud lateral movementDirect access to IMDS-AKSCluster-to-cloud lateral movementDirect access to IMDS-EKS EKS uses EC2 Roles for the Kubernetes nodes.By default,the EC2 Rol
29、e has the following policies:AmazonEC2ContainerRegistryReadOnly-Pull permissions to the container registry AmazonEKSWorkerNodePolicy-Read permissions of the compute environment(EC2,VPC,etc.)AmazonEKS_CNI_Policy Attach network interfaces and IPs to VMs Users can add more policies,if their containers
30、require access to cloud resources.Cluster-to-cloud lateral movementDirect access to IMDS-GKE GKE uses IAM service accounts to authenticate with the cloud.By default,all the VMs in a project,including the Kubernetes nodes,share a default SA.This SA has Editor role for the project.While the access sco
31、pe limits the permissions,they are still powerful by default:Cluster-to-cloud lateral movementDirect access to IMDSHow does lateral movement from the cluster to the cloud would look like?API ServerK8s control planeetcdscheduler Controller managerNode 1Pod APod BNode 2Pod CPod DNode 3Pod EPod FAPI Se
32、rverK8s control planeetcdscheduler Controller managerNode 1Pod APod BAPI ServerK8s control planeetcdscheduler Controller managerNode 1Pod APod BMetadataserviceAPI ServerK8s control planeetcdscheduler Controller managerNode 1Pod APod BMetadataserviceGet tokenAPI ServerK8s control planeetcdscheduler C
33、ontroller managerNode 1Pod APod BMetadataserviceReturns cloud identity tokenAPI ServerK8s control planeetcdscheduler Controller managerNode 1Pod APod BMetadataserviceList storage accounts storage bucketsAPI ServerK8s control planeetcdscheduler Controller managerNode 1Pod APod BMetadataserviceList se
34、crets from secret store(Key Vault,KMS,etc)API ServerK8s control planeetcdscheduler Controller managerNode 1Pod APod BMetadataserviceGet credentials of Kubernetes clustersCluster-to-cloud lateral movementDirect access to IMDSThe problem Pods can freely access to their nodes cloud identities.All pods
35、share the same cloud identities(the nodes identities).What we want:Allocate a specific identity to each pod(that needs access to cloud resources)with the minimal needed permissions.Make sure pods can only acquire tokens for their own identities.Cluster-to-cloud lateral movementIndirect access to IMD
36、SCluster-to-cloud lateral movementIndirect access to IMDS Users allocate different identities to the various applications in the cluster.When pods query IMDS,the traffic is intercepted and redirected to a local server in the cluster.The local server is K8s-aware,thus can identify the querying pod.Th
37、e local server queries IMDS on behalf of the pod and request the pod-specific identity.This concept was implemented in Azure by AAD Pod Identity in deprecation.API ServerK8s control planeetcdscheduler Controller managerNode 1Pod AMetadataservice32Local server(NMI)31.Requests to the metadata server a
38、re intercepted and sent to the NMI pod(by modifying the IPTables of the node).2.NMI recognize the querying pod and retrieves its token from the metadata server.3.NMI send back the token to the podPod B1API ServerK8s control planeetcdscheduler Controller managerNode 1Pod AMetadataservice32Local serve
39、r(NMI)3Pod B1Limitations1.Works only for Linux containers(uses IPTables).2.Not supported by all Kubernetes network configuration.Cluster-to-cloud lateral movementUsing OIDC(identity federation)Implemented by all major cloud providers:Azure:AAD workload identity AWS:IAM roles for service accounts(IRS
40、A)GCP:Workload identity The Kubernetes cluster is used as an OIDC identity provider(IdP).Trust relation is created such as the cloud identity platform(AAD,AWS IAM,GCP IAM)trusts the service accounts issued by the K8s cluster.This trust relation allows applications in the cluster to exchange a K8s se
41、rvice account token with a cloud identity token.Cluster-to-cloud lateral movementUsing OIDC(identity federation)General flowKubeletPodCloud identity serviceClustersOIDC endpointProjects a signed service account token to the podSends the token to the cloud identity service and requests to exchange it
42、 with a cloud identity tokenValidates the service account tokenReturns a cloud identity tokenCluster-to-cloud lateral movementUsing OIDC(identity federation)Example:AzureBinding of a Kubernetes service account to an AAD application(federation)Cluster-to-cloud lateral movementUsing OIDC(identity fede
43、ration)GCP In GCP,theres a unified identity pool for the entire project.Meaning,theres a single binding of a K8s service account(namespace+SA name)to a cloud identity.Cluster ANamespace:monitoringService account:sa-1Cluster BNamespace:monitoringService account:sa-1MyCloudAppgcloudgcloud iamiam servi
44、ceservice-accounts addaccounts add-iamiam-policypolicy-binding binding MyCloudAppMyCloudAppMyMy-GCPGCP-ProjectP -role roles/role roles/iam.workloadIdentityUseriam.workloadIdentityUser -member member serviceAccount:serviceAccount:MyMy-GCPGCP-ProjectProject.svc.id.goog.svc.id.goog monitoringmonitoring
45、/sasa-1 1 Cluster ANamespace:monitoringService account:sa-1Cluster BNamespace:monitoringService account:sa-1MyCloudAppCluster CCluster-to-cloud lateral movementUsing OIDC(identity federation)GCPCluster-to-cloud lateral movementUsing OIDC(identity federation)GCPCluster ANamespace:monitoringService ac
46、count:sa-1Cluster BNamespace:monitoringService account:sa-1MyCloudAppCluster CNamespace:MonitoringService account:sa-1Create new namespace:monitoringCreate new Service Account:sa-1Cluster-to-cloud lateral movementUsing OIDC(identity federation)GCPCluster ANamespace:monitoringService account:sa-1Clus
47、ter BNamespace:monitoringService account:sa-1MyCloudAppCreate new namespace:monitoringCreate new Service Account:sa-1Cluster CNamespace:MonitoringService account:sa-1The security boundary is the project,not the cluster!Cluster-to-cloud lateral movementWhat have we seen so far?Inner-cluster lateral m
48、ovement Example:Self-update permissions that lead to cluster-takeover Additional permissions can lean to cluster takeover(partial list)Cluster-to-cloud lateral movement:Storing cloud identity credential on the node Direct access to IMDS Indirect access to IMDS Using OIDC(identity federation)Detectio
49、ns&mitigationsDetections&mitigationsDetections&mitigationsCloud provider control planeMonitor suspicious activity in the cluster using K8s audit log(kube-audit).Examples:1.Deployment of abnormal images2.Pods with suspicious configurations(sensitive volume mounts,privileged etc.)3.Reconnaissance acti
50、vity(for example:SelfSubjectRulesReview API call).4.Sensitive API calls,such as“get secret”Monitor suspicious activity of cloud identities used by K8s workloadsnodes.Examples:1.Abnormal behavior of cloud identities.Usually,the cloud identities used by the workloads have a consistent behavior.2.Suspi
51、cious access to sensitive cloud services(e.g.storage,secret store etc.)Kubernetes control planeDetections&mitigationsCloud provider control planeMonitor suspicious activity in the cluster using K8s audit log(kube-audit).Examples:1.Deployment of abnormal images2.Pods with suspicious configurations(se
52、nsitive volume mounts,privileged etc.)3.Reconnaissance activity(for example:SelfSubjectRulesReview API call).4.Sensitive API calls,such as“get secret”Monitor suspicious activity of cloud identities used by K8s workloadsnodes.Examples:1.Abnormal behavior of cloud identities.Usually,the cloud identiti
53、es used by the workloads have a consistent behavior.2.Suspicious access to sensitive cloud services(e.g.storage,secret store etc.)Azure:Activity LogAWS:CloudTrailGCP:Cloud Audit LogsKubernetes control planeDetections&mitigations In December,a new version of the Threat Matrix for Kubernetes was relea
54、sed(v3).The new version now includes mitigation techniques.http:/aka.ms/KubernetesThreatMatrixDetections&mitigationsDetections&mitigationsDetections&mitigationsDetections&mitigationsDetections&mitigationsDetections&mitigationsDetections&mitigationsDetections&mitigationsDetections&mitigationsDetectio
55、ns&mitigationsDetections&mitigationsDetections&mitigationsThreat Matrix for Kubernetes1.Threat Matrix for Kubernetes v1 April 2020.a.MITRE ATT&CK framework became the standard for EDR evaluationsb.OS focused(missing coverage of Cloud-ish TTPs)Threat Matrix for Kubernetes1.Threat Matrix for Kubernete
56、s v1 April 2020.a.MITRE ATT&CK framework became the standard for EDR evaluationsb.OS focused(missing coverage of Cloud-ish TTPs)2.Consensually acceptance by the industry a.Customers b.Competitors c.MITRE (+Microsofts matrix v2)April 2021 Threat Matrix for Kubernetes1.Threat Matrix for Kubernetes v1
57、April 2020.a.MITRE ATT&CK framework became the standard for EDR evaluationsb.OS focused(missing coverage of Cloud-ish TTPs)2.Consensually acceptance by the industry a.Customers b.Competitors c.MITRE (+Microsofts matrix v2)April 2021 3.v3 December 2022a.New attack TTPsb.Introducing Mitigation Key tak
58、eaways1.Implement a holistic strategy for K8s security by considering both the cluster and the cloud levels.2.Identities are a key aspect of K8s security:monitor their activity using auditing tools.3.Adhere to the least-privilege principle.4.Use mitigation measures to prevent potential attacks.Thank You!Yossi Weizmanyossi-weizmanRam Pliskinrampliskin