1. Cluster Setup and Hardening

Security benchmark

• Disable root

• Configure sudo

• Configure the firewall

• Auditing and Logging

CIS Center for Internet Security

• CIS-CAT Lite sh ./Assessor-CLI.sh -i -rd /var/www/html/ -nts -rp index

• CIS-CAT Pro Assessor https://workbench.cisecurity.org/dashboard

• CIS-CAT Pro Dashboard

Kube-bench

• wget https://github.com/aquasecurity/kube-bench/releases/download/v0.4.0/kube-bench_0.4.0_linux_amd64.tar.gz

• tar -xf kube-bench_0.4.0_linux_amd64.tar.gz

• ./kube-bench --config-dir pwd/cfg --config pwd/cfg/config.yaml

Authentication

• Static Password file --basic-auth-file=user-details.csv

• Static Token file --token-auth-file=user-details.csv

• Certificates

• External Identity Services

Service accounts

• Service account -

  • kubectl create serviceaccount dashboard-sa

  • kubectl get serviceaccount

  • kubectl describe serviceaccount dashboard-sa

• Secret -

• Token - kubectl create tocken dashboard-sa

Certificates

Certificates API

• openssl req -new -nodes -newkey rsa:2048 -keyout jane.key -out jane.csr -subj "/CN=jane"

• create csr.yaml from template

• cat jane.csr | base64 -w 0 >> csr.yaml

• kubectl create -f csr.yaml

• kubectl get csr -o yaml

• kubectl certificate approve jane

• kubectl get csr jane -oyaml | grep certificate: | cut -d " " -f 4 | base64 -d > jane.crt

• kubectl get csr myuser -o jsonpath="{.status.certificate}" | base64 -d > jane.crt

• kubectl config set-credentials jane --client-key jane.key --client-certificate jane.crt --embed-certs

• kubectl config set-context jane@kubernetes --user jane --server kubernetes

• kubectl config use-context jane@kubernetes

KubeConfig

• Clusters/Users/Contexts

• kubectl config view --kubeconfig=my-custom-config

• kubectl config use-context prod-user@production

API Groups

/version

/api

/apis

Authorization modes (--authorization-mode=)

• Node - special auth mode used by kubelets

• RBAC

• Webhook - Open Policy Agent

RBAC

• role, role-binding, cluster role, cluster role binding

• kubectl auth can-i create deployments --as dev-user --namespace test

• kubectl auth can-i list --as dev-user --namespace test

• kubectl auth can-i create deployments --as system:serviceaccount:default:default --namespace test

• kubectl api-resources --namespaced=true

Kublet Security

• /var/lib/kubelet/config.yaml

• /etc/kubernetes/kubelet.conf

• ports 10250,10255

Kubectl Proxy and Port Forwarding

• kubectl proxy &

• curl http://localhost:8001 -k

• curl http://localhost:8001/api/v1/namespaces/default/servoces/nginx/proxy/

• kubectl port-forward service/nginx 28080:80 &

• curl http://localhost:28080/

Kubernetes Dashboard

Verify platform binaries before deploying

• sha512sum kubernetes.tar.gz | grep 3062a427a45548bd9c5a8358c740f0a5cfea7b546dca724c71d28768bb36c628280c91263a362afd01c89ef3944f5a768ed44e75d421fe9dc1ec2e8ba26214f3

Kubernetes Software Versions

Cluster Upgrade Process

• kube-apiserver - X-1 or X version

• control-manager, kube-scheduler - X-1 or X versions

• kubelet, kube-proxy - X-2, X-1 or X versions

• kubectl - X-1, X or X+1 versions

1. Control Plane

   1. apt update && apt upgrade -y

   2. apt-cache madison kubeadm | grep 1.26

   3. apt-mark unhold kubeadm

   4. apt install kubeadm=1.26.1-00 -y

   5. apt-mark hold kubeadm

   6. kubeadm upgrade plan

   7. kubeadm upgrade apply v1.26.1 -y

   8. kubectl drain cp --ignore-daemonsets

   9. apt-mark unhold kubelet kubectl

   10. apt install kubelet=1.26.1-00 kubectl=1.26.1-00 -y

   11. apt-mark hold kubectl kubelet

   12. systemctl daemon-reload

   13. systemctl restart kubelet.service

   14. kubectl uncordon cp

2. Additional control plane

   1. apt update && apt upgrade -y

   2. apt-mark unhold kubeadm

   3. apt install kubeadm=1.26.1-00 -y

   4. apt-mark hold kubeadm

   5. kubeadm upgrade node -y

   6. kubectl drain cp2 --ignore-daemonsets (run this on cp)

   7. apt-mark unhold kubelet kubectl

   8. apt install kubelet=1.26.1-00 kubectl=1.26.1-00 -y

   9. apt-mark hold kubectl kubelet

   10. systemctl daemon-reload

   11. systemctl restart kubelet.service

   12. kubectl uncordon cp2 (run thist on cp)

3. Worker node

   1. apt update && apt upgrade -y

   2. apt-mark unhold kubeadm

   3. apt install kubeadm=1.26.1-00 -y

   4. apt-mark hold kubeadm

   5. kubeadm upgrade node -y

   6. kubectl drain worker --ignore-daemonsets (run this on cp)

   7. apt-mark unhold kubelet kubectl

   8. apt install kubelet=1.26.1-00 kubectl=1.26.1-00 -y

   9. apt-mark hold kubectl kubelet

   10. systemctl daemon-reload

   11. systemctl restart kubelet.service

   12. kubectl uncordon worker (run thist on cp)

Network Policy

• Ingress

• Egress

Ingress

• host

• annotation - rewrite

• don't forget to specify

  spec:

  ingressClassName: nginx