应用场景

Kubernetes 审计（Auditing） 功能提供了与安全相关的、按时间顺序排列的记录集， 记录每个用户、使用 Kubernetes API 的应用以及控制面自身引发的活动。通过APIServer审计日志可以追踪用户对集群的操作行为。更进一步的，通过使用日志中心可以将集群的APIServer审计日志持久化到云日志服务，从而长时间追踪APIServer请求。

前提条件

已创建专有版集群，具体操作请参见 用户指南 > 集群管理 > 新建集群 。若已有集群，无需重复操作。

拥有集群master节点的root或sudo权限，能够远程登录节点执行命令。

操作步骤

以下步骤需要在集群所有master节点执行（可在集群菜单 节点管理 > 节点 查看master节点，一般有3个或5个）

增加apiserver审计策略文件

在集群master节点，新增 /etc/kubernetes/audit-policy.yaml 文件

apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
#不要为RequestReceived阶段中的所有请求生成审核事件。
omitStages:
- "RequestReceived"
rules:
# 以下请求被手动确定为高容量和低风险，因此请取消这些请求。
- level: None
  users: ["system:kube-proxy"]
  verbs: ["watch"]
  resources:
    - group: "" # core
      resources: ["endpoints", "services"]
- level: None
  users: ["system:unsecured"]
  namespaces: ["kube-system"]
  verbs: ["get"]
  resources:
    - group: "" # core
      resources: ["configmaps"]
- level: None
  users: ["kubelet"] # legacy kubelet identity
  verbs: ["get"]
  resources:
    - group: "" # core
      resources: ["nodes"]
- level: None
  userGroups: ["system:nodes"]
  verbs: ["get"]
  resources:
    - group: "" # core
      resources: ["nodes"]
- level: None
  users:
    - system:kube-controller-manager
    - system:kube-scheduler
    - system:serviceaccount:kube-system:endpoint-controller
  verbs: ["get", "update"]
  namespaces: ["kube-system"]
  resources:
    - group: "" # core
      resources: ["endpoints"]
- level: None
  users: ["system:apiserver"]
  verbs: ["get"]
  resources:
    - group: "" # core
      resources: ["namespaces"]
#不要记录这些只读URL。
- level: None
  nonResourceURLs:
    - /healthz*
    - /version
    - /swagger*
#不要记录事件请求。
- level: None
  resources:
    - group: "" # core
      resources: ["events"]
# 机密、配置映射和令牌审查可以包含敏感和二进制数据，
# 因此，只能在元数据级别进行日志记录。
- level: Metadata
  resources:
    - group: "" # core
      resources: ["secrets", "configmaps"]
    - group: authentication.k8s.io
      resources: ["tokenreviews"]

- level: Request
  verbs: ["get", "list", "watch"]
  resources:
    - group: "" # core
    - group: "admissionregistration.k8s.io"
    - group: "apps"
    - group: "authentication.k8s.io"
    - group: "authorization.k8s.io"
    - group: "autoscaling"
    - group: "batch"
    - group: "certificates.k8s.io"
    - group: "extensions"
    - group: "networking.k8s.io"
    - group: "policy"
    - group: "rbac.authorization.k8s.io"
    - group: "settings.k8s.io"
    - group: "storage.k8s.io"
# 已知API的默认级别。
- level: RequestResponse
  resources:
    - group: "" # core
    - group: "admissionregistration.k8s.io"
    - group: "apps"
    - group: "authentication.k8s.io"
    - group: "authorization.k8s.io"
    - group: "autoscaling"
    - group: "batch"
    - group: "certificates.k8s.io"
    - group: "extensions"
    - group: "networking.k8s.io"
    - group: "policy"
    - group: "rbac.authorization.k8s.io"
    - group: "settings.k8s.io"
    - group: "storage.k8s.io"
# 所有其他请求的默认级别。
- level: Metadata

修改apiserver的pod的yaml文件

修改 /etc/kubernetes/manifests/kube-apiserver.yaml 文件，操作前可对原文件备份，当有问题时可以回退，增加以下yaml 由 "# 增加的部分start" 及 "# 增加的部分end" 括住的内容（共三处）。

apiVersion: v1
kind: Pod
metadata:
annotations:
kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: xx.xx.xx.xx
creationTimestamp: null
labels:
component: kube-apiserver
tier: control-plane
name: kube-apiserver
namespace: kube-system
spec:
containers:
- command:
- kube-apiserver
- --advertise-address=xx.xx.xx.xx
- --allow-privileged=true
- --authorization-mode=Node,RBAC
- --client-ca-file=/etc/kubernetes/pki/ca.crt
- --enable-admission-plugins=NodeRestriction
- --enable-bootstrap-token-auth=true
- --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
- --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
- --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
- --etcd-servers=https://127.0.0.1:2379
- --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
- --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
- --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
- --requestheader-allowed-names=front-proxy-client
- --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
- --requestheader-extra-headers-prefix=X-Remote-Extra-
- --requestheader-group-headers=X-Remote-Group
- --requestheader-username-headers=X-Remote-User
- --secure-port=6443
- --service-account-issuer=https://kubernetes.default.svc.cluster.local
- --service-account-key-file=/etc/kubernetes/pki/sa.pub
- --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
- --service-cluster-ip-range=10.96.0.0/12
- --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
- --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
# 增加的部分start
- --audit-log-path=/var/log/kubernetes/audit/audit.log
- --audit-policy-file=/etc/kubernetes/audit-policy.yaml
- --audit-log-maxage=7
- --audit-log-maxsize=100
- --audit-log-maxbackup=1
# 增加的部分end
image: xx.xx.xx.xx
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 8
httpGet:
host: xx.xx.xx.xx
path: /livez
port: 6443
scheme: HTTPS
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 15
name: kube-apiserver
readinessProbe:
failureThreshold: 3
httpGet:
host: xx.xx.xx.xx
path: /readyz
port: 6443
scheme: HTTPS
periodSeconds: 1
timeoutSeconds: 15
resources:
requests:
cpu: 250m
startupProbe:
failureThreshold: 24
httpGet:
host: xx.xx.xx.xx
path: /livez
port: 6443
scheme: HTTPS
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 15
volumeMounts:
- mountPath: /etc/ssl/certs
name: ca-certs
readOnly: true
- mountPath: /etc/pki
name: etc-pki
readOnly: true
- mountPath: /etc/kubernetes/pki
name: k8s-certs
readOnly: true
# 增加的部分start
- mountPath: /etc/kubernetes/audit-policy.yaml
name: audit
readOnly: true
- mountPath: /var/log/kubernetes/audit/
name: audit-log
readOnly: false
# 增加的部分end
hostNetwork: true
priorityClassName: system-node-critical
securityContext:
seccompProfile:
type: RuntimeDefault
volumes:
- hostPath:
path: /etc/ssl/certs
type: DirectoryOrCreate
name: ca-certs
- hostPath:
path: /etc/pki
type: DirectoryOrCreate
name: etc-pki
- hostPath:
path: /etc/kubernetes/pki
type: DirectoryOrCreate
name: k8s-certs
# 增加的部分start
- hostPath:
path: /etc/kubernetes/audit-policy.yaml
type: File
name: audit
- hostPath:
path: /var/log/kubernetes/audit/
type: DirectoryOrCreate
name: audit-log
# 增加的部分end
status: {}

修改后重启apiserver

执行命令sudo systemctl restart kubelet，注意此操作会重启节点上的k8s控制面组件，业务容器不会被重启，若业务依赖k8s APIServer则有影响，否则不受影响，请根据业务实际情况选择合适的时间执行。

检查apiserver的运行状态

kubectl get po -nkube-system | grep kube-apiserver ，如果当前节点apiserver pod的运行状态为 1/1 Running 则正常，否则需要将 /etc/kubernetes/manifests/kube-apiserver.yaml文件还原，再次执行sudo systemctl restart kubelet回退修改。

查看审计日志

检查节点/var/log/kubernetes/audit/audit.log 文件是否能看到apiserver的审计日志。

参考文档

Kubernetes 官方文档：https://kubernetes.io/zh-cn/docs/tasks/debug/debug-cluster/audit/