클라우드 컴퓨팅/쿠버네티스

Kubernetes Cluster Maintenance

espossible 2021. 10. 16. 17:04

OS Upgrades

노드의 OS Upgrade를 위해서는 노드를 정지해야한다. 이때 도커 컨테이너의 라이브 마이그레이션에 해당하는 기능이 존재하지 않아 메인터넌스 등으로 노드를 정지하는 경우 파드를 정지해야 한다. 따라서 SIGTERM, SIGKILL 신호에 동작하는 애플리케이션을 만들고 terminationGracePeriodSeconds를 적절하게 설정해야한다.

 

즉, 각 파드에 정지 처리가 실행될 때 안전하게 정지하도록 preStop 설정 / gracefulPeriod 설정 / 애플리케이션 측에서 SIGTERM 신호 핸들링 등과 같은 적절한 준비를 해두어야 한다.

 

쿠버네티스 노드는 SchedulingEnabled(Default)와 SchedulingDisabled 중 하나의 상태를 가진다. 노드가 SchedulingDisabled 상태로 변경되도 이미 노드에서 실행되는 파드에는 영향을 주지 않는다.

 

노드의 OS Upgrade 절차는 아래와 같다.

참고) 테스트를 위해 kube-controller-manager의 --pod-eviction-timeout을 사용할 수 있다.

 

첫째, 파드축출(관련 포드들을 gracefully stop 시키고, 다른 노드에 띄운다.)

배출 처리에는 이후의 스케줄링 대상에서 제외하는 처리가 포함되어 있다.

kubectl drain node_name (--force --ignore-daemonsets --delete-local-data)

--force 옵션 : 단일 파드가 있으면 삭제 후에 재성성되지 않아도 되는 경우.

--delete-local-data 옵션 : 로컬 스토리지를 사용하고 있는 경우 데이터가 삭제해도 되는 경우.

--ignore-daemonsets 옵션 : 데몬셋이 관리하는 파드도 축출하고자 하는 경우.

 

둘째, 해당 노드를 스케줄링 대상에서 제외한다. 이미 노드에서 실행되는 파드에는 영향을 주지 않는다.

kubectl cordon node_name

셋째, OS Upgrade를 진행한다.

넷째, 해당 노드를 스케줄링 대상으로 복귀시킨다.

kubectl uncordon node_name

 

Cluster Upgrade

v1.11.3 : major.minor(features, funtionalites).patch(bug fix)

v1.11.0-alpha > v1.11.0-beta > v1.11.0

kube-apiserver, controller-manager, kube-scheduler, kubelet, kube-proxy, kubectl는 같은 버전의 패키지에 포함된다.

 

kubeadm으로 쿠버네티스 클러스터를 구성했다는 가정하에 master/controlplane components의 버전을 업그레이드 하는 과정을 살펴보겠다.

 

Automatic upgrade by providers(AWS EKS, ...)

# 어떤 버전으로 업그레이드 가능한지 볼수 있음
kubectl upgrade plan
# Upgrade
kubectl upgrade apply

 

Upgrade in a hard way(kubeadm)

Master 노드

첫째, master 노드의 파드를 축출한다.

kubectl drain master --ignore-daemonsets

둘째, kubeadm tool을 업그레이드 한다.

# Kubeadm installation
apt install kubeadm=1.xx.0-00 -y

# Check kubeadm version
kubeadm version

셋째, master components를 업그레이드 한다.

# Upgrade
kubeadm upgrade apply v1.xx.0

# Check if it is Upgraded
# not yet upgraded because kubelet is not upgraded
kubectl version --short
kubectl get nodes

넷째, kubelet을 업그레이드 한다.

# upgrade the kubelet on the master node
apt install kubelet=1.xx.0-00

# Recheck if it is Upgraded
kubectl get nodes

다섯째, master node를 스케줄링 대상으로 복귀시킨다.

# Mark the master/controlplane node as "Schedulable"
kubectl uncordon controlplane

 

Work 노드

첫째, work 노드의 파드를 축출한다.

# Drain the worker node of the workloads and mark it UnSchedulable
kubectl drain node-1 --ignore-daemonsets

둘째, kubeadm tool을 업그레이드 한다.

# Kubeadm installation
apt install kubeadm=1.xx.0-00 -y

# Check kubeadm version
kubeadm version

 

셋째, worker 노드를 업그레이드 한다.

kubeadm upgrade node

넷째, local kubelet configuration을 업그레이드 한다.

# upgrade the kubelet on the worker node
apt install kubelet=1.xx.0-00

# Recheck if it is Upgraded
kubectl get nodes

다섯째, worker 노드를 스케줄링 대상으로 복귀시킨다.

# Mark the worker node as "Schedulable"
kubectl uncordon worker_node

 

ETCD

ETCD is a distributed reliable key-value store that is Simple, Secure & Fast.

HA topology

- stacked etcd architecture : each control plane node creates a local etcd member

출처 : https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/ha-topology/

- external etcd architecture : etcd members run on separate hosts

출처 : https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/ha-topology/

# cat /etc/systemd/system/kube-apiserver.service
[Service]
ExecStart=/usr/local/bin/kube-apiserver \\
  --advertise-address=${INTERNAL_IP}
  --allow-privileged=true \\
  --apiserver-count=3 \\
  --etcd-cafile=/var/lib/kubernetes/ca.pem \\
  --etcd-certfile=/var/lib/kubernetes/kubernetes.pem \\
  --etcd-keyfile=/var/lib/kubernetes/kubernetes-key.pem \\
  --etcd-servers=https://10.240.0.10:2379,https://10.240.0.11:2379

ETCD options

--listen-client-urls(2379 포트)

--listen-peer-urls : 다른 노드에서 접근시 사용(2380 포트)

--cert-file=/etc/kubernetes/pki/etc/server.crt(ETCD server certificate)

--peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt(ETCD CA certificate)

ETCD read & write

  • READ : you can easily read data from any node
  • Write : only one of the instances is responsible for processing the writes. the two nodes elect a leader among them. Of the total instances one node becomes the leader and the other node becomes the followers. If the writes came into the leader node, then the leader processes the write. The leader makes sure that the other nodes are sent a copy of the data. A write in only considered complete, if the leader gets consent from other members in the cluster.

ETCD Leader Election(RAFT protocol)

  • Quorum = N/2 + 1 : If it can be written on the majority of the nodes in the cluster, a write will be successful.
  • Fault Tolerance : HA 구성을 위해서는 1 노드가 죽더라도 Quorum 2를 유지할 수 있게 적어도 3개 노드는 필요함(recommended).
  • Even or Odd : with even number of nodes there is possibility of the cluster faliling during a network segmentation. So need to choose the odd number.
[etcd.service]
ExecStart=/usr/local/bin/etcd \\
  --name ${ETCD_NAME}
  --cert-file=/etc/etcd/kubernetes.pem \\
  --key-file=/etc/etcd/kubernetes-key.pem \\
  --peer-cert-file=/etc/etcd/kubernetes.pem \\
  --peer-key-file=/etc/etcd/kubernetes-key.pem \\
  --trusted-ca-file=/etc/etcd/ca.pem \\
  --peer-trusted-ca-file=/etc/etcd/ca.pem \\
  --peer-client-cert-auth \\
  --client-cert-auth \\
  --initial-advertise-peer-urls https://${INTERNAL_IP}:2380 \\
  --listen-peer-urls https://${INTERNAL_IP}:2380 \\
  --listen-client-urls https://${INTERNAL_IP}:2379,https://127.0.0.1:2379 \\
  --advertise-client-urls https://${INTERNAL_IP}:2379 \\
  --initial-cluster-token etcd-cluster-0 \\
  --initial-cluster peer-1=https://${PEER1_IP}:2380,peer-2=https://${PEER2_IP}:2380 \\
  --initial-cluster-state new \\
  --data-dir=/var/lib/etcd

<initial-cluster>

ETCDCTL

  • GET/PUT
export ETCDCTL_API=3
etcdctl put name john
etcdctl get name
etcdctl get / --prefix --keys-only

 

ETCD Backup and Restore configuration

  • Git에 Resource configuration(yaml ...)을 저장
  • ETCD : 스냅샷 변경시 --data-dir 설정을 확인하자.
#etcd.service
ExecStart=/usr/local/bin/etcd \\
  --name ${ETCD_NAME} \\
  --cert-file=/etc/etcd/kubernetes.pem \\
  --key-file=/etc/etcd/kubernetes-key.pem \\
  --peer-cert-file=/etc/etcd/kubernetes.pem \\
  --peer-key-file=/etc/etcd/kubernetes-key.pem \\
  --trusted-ca-file=/etc/etcd/ca.pem \\
  --peer-trusted-ca-file=/etc/etcd/ca.pem \\
  --peer-client-cert-auth \\
  --client-cert-auth \\
  --initial-advertise-peer-urls https://${INTERNAL_IP}:
  --listen-peer-urls https:${INTERNAL_IP}:2380 \\
  --listen-client-urls https:${INTERNAL_IP}:2379,
  --adverties-client-urls https:${INTERNAL_IP}:2379 \\
  --initial-cluster-token etcd-cluster-0 \\
  --initial-cluster controller-0=https://${CONTROLLER0_ \\
  --initial-cluster-state new \\
  --data-dir=/var/lib/etcd

 

ETCD - SAVE

ETCDCTL_API=3 etcdctl snapshot save snapshot.db

# With TLS-Enabled
ETCDCTL_API=3 etcdctl \
		snapshot save snapshot.db \
        --endpoints=https://127.0.0.1:2379 \
        --cacert=/etc/etcd/ca.crt \
        --cert=/etc/etcd/etcd-server.crt \
        --key=/etc/etcd/etcd-server.key

ETCD - STATUS

ETCDCTL_API=3 etcdctl snapshot status snapshot.db

ETCD - RESTORE

# Stop kube-apiserver related to ETCD
service kube-apiserver stop

# Restore
ETCDCTL_API=3 etcdctl snapshot restore snapshot.db --data-dir /var/lib/etcd-from-backup

# Option1. change --data-dir (etcd.service)
ExecStart=/usr/local/bin/etcd \\
  --name ${ETCD_NAME} \\
  --cert-file=/etc/etcd/kubernetes.pem \\
  --key-file=/etc/etcd/kubernetes-key.pem \\
  --peer-cert-file=/etc/etcd/kubernetes.pem \\
  --peer-key-file=/etc/etcd/kubernetes-key.pem \\
  --trusted-ca-file=/etc/etcd/ca.pem \\
  --peer-trusted-ca-file=/etc/etcd/ca.pem \\
  --peer-client-cert-auth \\
  --client-cert-auth \\
  --initial-advertise-peer-urls https://${INTERNAL_IP}:
  --listen-peer-urls https:${INTERNAL_IP}:2380 \\
  --listen-client-urls https:${INTERNAL_IP}:2379,
  --adverties-client-urls https:${INTERNAL_IP}:2379 \\
  --initial-cluster-token etcd-cluster-0 \\
  --initial-cluster controller-0=https://${CONTROLLER0_ \\
  --initial-cluster-state new \\
  --data-dir=/var/lib/etcd-from-backup

# Option2. etcd volume 수정
# /etc/kubernetes/manifests/etcd.yaml(etcd pod - static pod)
volumes:
  - hostPath:
      path: /var/lib/etcd-from-backup
      type: DirectoryOrCreate
    name: etcd-data


# Reload daemon
systemctl daemon-reload

# Restart ETCD
service etcd restart

# Start the kube-apiserver
service kube-apiserver start

 

Note:

If any API servers are running in your cluster, you should not attempt to restore instances of etcd. Instead, follow these steps to restore etcd:

  • stop all API server instances
  • restore state in all etcd instances
  • restart all API server instances

We also recommend restarting any components (e.g. kube-scheduler, kube-controller-manager, kubelet) to ensure that they don't rely on some stale data. Note that in practice, the restore takes a bit of time. During the restoration, critical components will lose leader lock and restart themselves.

 

 

ETCD 디버깅

  • docker ps -a | grep etcd
  • docker logs xxx(위 명령어로 docker container id를 얻는다)
  • If the etcd pod is not getting Ready 1/1, then restart it by kubectl delete pod -n kube-system etcd-controlplane and wait 1 minute.

퀴즈

1. How many nodes can host workloads in this cluster?

>> Inspect the applications and taints set on the nodes.

>> taint 걸려있지 않는 노드의 수

 

2. How many applications are hosted on the cluster?

>> Count the number of deployments

 

3. You are tasked to upgrade the cluster. you can't provision new VMs and User's accessing the applications must not be impacted. What strategy would you use to upgrade the cluster?

>> Upgrade one node at a time while moving the workloads to the other

 

출처

- [강의] udemy - Certified Kubernetes Administrator 

- [책] 쿠버네티스 완벽 가이드(마사야 아오야마 지음, 박상욱 옮김)

- https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/#backing-up-an-etcd-cluster

- https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/recovery.md

- https://www.youtube.com/watch?v=qRPNuT080Hk

- https://v1-19.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/

- https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md

- https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md

- https://kubernetes.io/docs/concepts/overview/kubernetes-api/