Kubernetes Audit Logs Integration
Sematext offers a Kubernetes Audit logs receiver endpoint. Everything you need to do is to configure the Kubernetes API Server to send logs to it.
Sematext Kubernetes Audit Logs Quick Start¶
Kubernetes audit logs are detailed descriptions of each call made to the Kubernetes API Server. They provide a chronological sequence of activities that lead to the state of the system at a specific moment. They are extremely useful for security and compliance purposes, telling you exactly who did what, when, and how. You can configure Kubernetes Audit Logs by using any of the two options below.
We recommend you use the Dynamic Backend because it does not require you to install any agents. You only need to configure the Kubernetes API Server to send audit logs to the URL we provide you with.
The Log Backend is a viable option if you already have Logagent running in your Kubernetes cluster.
Kubernetes Audit Dynamic Backend¶
Configuring the Dynamic Backend simpler than a Log Backend. You need to editing the configuration of your Kubernetes API Server and add three new configuration parameters.
Kops¶
If you use Kops for cluster management you run kops edit cluster <cluster> to open the configuration. You only need to set one field to enable the dynamic audit configuration:
spec: kubeAPIServer: auditDynamicConfiguration: true
By enabling this feature you are allowing for auditsinks to be registered with the API server. You can read more about it here.
Next, you have to add an auditsink resource. Create a file named auditsink.yaml.
apiVersion: auditregistration.k8s.io/v1alpha1 kind: AuditSink metadata: name: k8sauditsink policy: level: Metadata stages: - ResponseComplete webhook: throttle: qps: 10 burst: 15 clientConfig: url: "https://logsene-k8s-audit-receiver.sematext.com/<LOGS_TOKEN>/" # For EU Region # url: "https://logsene-k8s-audit-receiver.eu.sematext.com/<LOGS_TOKEN>/"
Now apply the auditsink.yaml to the Kubernetes cluster.
kubectl apply -f auditsink.yaml
That’s it. You’re now shipping Kubernetes Audit logs to Sematext Logs for safekeeping.
Kubeadm / Minikube¶
With Kubeadm or Minikube the Kubernetes API Server configuration will be in the /etc/kubernetes/manifests/kube-apiserver.yaml file, on the master node. Edit the file and add these three new configuration values.
... spec: containers: - command: - kube-apiserver ... - --audit-dynamic-configuration - --feature-gates=DynamicAuditing=true - --runtime-config=auditregistration.k8s.io/v1alpha1=true
Save the changes and exit the file. This will trigger a restart of the Kubernetes API server.
Next, you have to add an auditsink resource. Create a file named auditsink.yaml.
apiVersion: auditregistration.k8s.io/v1alpha1 kind: AuditSink metadata: name: k8sauditsink2 policy: level: Metadata stages: - ResponseComplete webhook: throttle: qps: 10 burst: 15 clientConfig: url: "https://logsene-k8s-audit-receiver.sematext.com/<LOGS_TOKEN>/" # or for the EU Region # url: "https://logsene-k8s-audit-receiver.eu.sematext.com/<LOGS_TOKEN>/"
Now apply the auditsink.yaml to the Kubernetes cluster.
kubectl apply -f auditsink.yaml
That’s it. You’re now shipping Kubernetes Audit logs to Sematext Logs for safekeeping.
If you are not using the Dynamic Backend, then you can set up the Log Backend.
Kubernetes Audit Log Backend¶
You need to enable audit logs only on the master nodes. First of all you need to create a policy to specify what will be recorded. A good example of the audit-policy.yaml file is the audit profile below from the official Kubernetes docs.
apiVersion: audit.k8s.io/v1 # This is required. kind: Policy # Don't generate audit events for all requests in RequestReceived stage. omitStages: - "RequestReceived" rules: # Log pod changes at RequestResponse level - level: RequestResponse resources: - group: "" # Resource "pods" doesn't match requests to any subresource of pods, # which is consistent with the RBAC policy. resources: ["pods"] # Log "pods/log", "pods/status" at Metadata level - level: Metadata resources: - group: "" resources: ["pods/log", "pods/status"] # Don't log requests to a configmap called "controller-leader" - level: None resources: - group: "" resources: ["configmaps"] resourceNames: ["controller-leader"] # Don't log watch requests by the "system:kube-proxy" on endpoints or services - level: None users: ["system:kube-proxy"] verbs: ["watch"] resources: - group: "" # core API group resources: ["endpoints", "services"] # Don't log authenticated requests to certain non-resource URL paths. - level: None userGroups: ["system:authenticated"] nonResourceURLs: - "/api*" # Wildcard matching. - "/version" # Log the request body of configmap changes in kube-system. - level: Request resources: - group: "" # core API group resources: ["configmaps"] # This rule only applies to resources in the "kube-system" namespace. # The empty string "" can be used to select non-namespaced resources. namespaces: ["kube-system"] # Log configmap and secret changes in all other namespaces at the Metadata level. - level: Metadata resources: - group: "" # core API group resources: ["secrets", "configmaps"] # Log all other resources in core and extensions at the Request level. - level: Request resources: - group: "" # core API group - group: "extensions" # Version of group should NOT be included. # A catch-all rule to log all other requests at the Metadata level. - level: Metadata # Long-running requests like watches that fall under this rule will not # generate an audit event in RequestReceived. omitStages: - "RequestReceived"
No matter which tool you use:
- Kops
- Minikube
- Kubeadm
- EKS
- GKE
- etc.
You need to add this policy file to your master nodes. To read more about auditing you can check out the official Kubernetes docs.
To enable the audit policy, you need to edit the definition of the Kubernetes API Server.
Kops¶
If you use Kops for cluster management you run kops edit cluster <cluster> to open the configuration. In the auditPolicyFile field you need to specify the absolute path to your policy file.
spec: ... kubeAPIServer: auditPolicyFile: /etc/kubernetes/policies/audit-policy.yaml auditLogPath: - # log to stdout auditLogMaxAge: 10 # num days auditLogMaxBackups: 1 # the num of audit logs to retain auditLogMaxSize: 100 # the max size in MB to retain
Once you’ve configured logging the Audit logs to stdout you can use cluster-level logging to store these logs in a central location.
Install the Sematext Agent Helm chart:
helm install --name st-agent \ --set infraToken=xxxx-xxxx \ --set containerToken=xxxx-xxxx \ --set logsToken=xxxx-xxxx \ --set region=US \ stable/sematext-agent
Kubeadm / Minikube¶
Otherwise, if you’re using Kubeadm or Minikube the Kubernetes API Server configuration will be in the /etc/kubernetes/manifests/kube-apiserver.yaml file, on the master node.
... spec: containers: - command: - kube-apiserver ... - --audit-policy-file=/etc/kubernetes/policies/audit-policy.yaml - --audit-log-path=- # log to stdout - --audit-log-format=json ... volumeMounts: - mountPath: /etc/kubernetes/policies name: policies readOnly: true ... hostNetwork: true volumes: - hostPath: path: /etc/kubernetes/policies type: DirectoryOrCreate name: policies
Apply these changes by restarting the Kubelet.
sudo systemctl restart kubelet
Once you’ve configured logging the Audit logs to stdout you can use cluster-level logging to store these logs in a central location.
Install the Sematext Agent Helm chart:
helm install --name st-agent \ --set infraToken=xxxx-xxxx \ --set containerToken=xxxx-xxxx \ --set logsToken=xxxx-xxxx \ --set region=US \ stable/sematext-agent
This will start both a log agent, and optionally a monitoring agent if you so wish. The agent will collect logs all logs from stdout, including the audit logs and send them to Sematext Logs.
If you prefer, you can set up Logagent with kubectl as well if you go here.