Build k8s cluster on Raspberry Pi(Ubuntu) in the hard way

espossible 2022. 3. 28. 18:05

2022. 3. 28. 18:05

쿠버네티스 컨트롤플레인을 수동으로 설치하는 방법을 통해 부트스트랩과정을 이해하고 배워보고자한다.

참고 : https://github.com/kelseyhightower/kubernetes-the-hard-way/tree/master/docs

4개의 라즈베리파이 중 2개를 마스터 노드로 활용하여 컨트롤플레인을 구축할 것이고, 나머지 2대는 워커 노드로서 간단한 웹 페이지를 띄우고 정상 동작하는 지를 확인하려한다.

Infrastructure Setting

Raspberry Pi cluster setup

- 4pc 라즈베리파이3 B+ 모델

- 4pc 16 GB SD card

- 4pc ethernet cables

- 1pc 스위치 허브

- 1pc USB power hub

- 4pc Micro-USB cables

Cluster Static IP

Host Name	IP
k8s-master-1	172.30.1.40
k8s-master-2	172.30.1.41
k8s-worker-1	172.30.1.42
k8s-worker-2	172.30.1.43

1. Infrastructure

우분터 서버 이미지는 기본적으로 ssh를 내장하고 있다.

Default user/pwd : ubuntu/ubuntu

- hostname 변경

sudo hostnamectl set-hostname xxx

- static ip 설정

vi /etc/netplan/xxx.yaml
network:
    ethernets:
        eth0:
            #dhcp4: true
            #optional: true
                addresses: [192.168.10.x/24]
                gateway4: 192.168.10.1
                nameservers:
                        addresses: [8.8.8.8]
    version: 2

sudo netplan apply

- /etc/hosts

172.30.1.40       k8s-master-1
172.30.1.41       k8s-master-2
172.30.1.42       k8s-worker-1
172.30.1.43       k8s-worker-2

- /boot/firmware/nobtcmd.txt

아래 내용 추가

cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1

2. Generating an CA and Certificates for k8s cluster(PKI certificates)

- Certificate Authority

Client Certificates

- Admin Client

- Kubelet Client

- Kube Proxy Client

- Controller Manager Client

- Service Account key pair : Kube controller manager uses a key pair to generate and sign service account tokens

- Scheduler Client

Server Certificates

- API Server

install cfssl on Mac to provision a PKI Infrastructure

- brew install cfssl

# Certificate Authority

참고 : https://kubernetes.io/ko/docs/tasks/administer-cluster/certificates/

Write an ca-config.json

{
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "kubernetes": {
        "usages": ["signing", "key encipherment", "server auth", "client auth"],
        "expiry": "8760h"
      }
    }
  }
}

Create a CSR for your new CA(ca-csr.json)

- CN : the name of the user

- C : country

- L : city

- O : the group that this user will belong to

- OU : organization unit

- ST : state

{
  "CN": "Kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "South Korea",
      "L": "Seoul",
      "O": "Kubernetes",
      "OU": "CA",
      "ST": "Seoul"
    }
  ]
}

Initialize a CA

cfssl gencert -initca ca-csr.json | cfssljson -bare ca
>> ca-key.pem, ca.csr, ca.pem

# Admin Client Certificate

Write admin-csr.json

{
  "CN": "admin",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "South Korea",
      "L": "Seoul",
      "O": "system:masters",
      "OU": "Kubernetes the Hard Way",
      "ST": "Seoul"
    }
  ]
}

Generate an Admin Client Certificate

- client certificate for the kubernetes admin user

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  admin-csr.json | cfssljson -bare admin
  
>> admin-key.pem, admin.csr, admin.pem

# Kubelet Client Certificate

Write k8s-node-x-csr.json

- Kubernetes uses a special-purpose authorization mode called Node Authorizer, that specifically authorizes API request made by Kubelets. In order to be authorized by the Node Authorizer, Kubelets must use a credential that identifies them as being in the system:nodes group, with a username of system:node:<nodeName>

{
    "CN": "system:node:k8s-worker-1",
    "key": {
      "algo": "rsa",
      "size": 2048
    },
    "names": [
      {
        "C": "South Korea",
        "L": "Seoul",
        "O": "system:nodes",
        "OU": "Kubernetes The Hard Way",
        "ST": "Seoul"
      }
    ]
}

Generate an Kubelet Client Certificate for each Worker Node

export WORKER_IP=172.30.1.42
export WORKER_HOST=k8s-worker-1
cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -hostname=${WORKER_IP},${WORKER_HOST} \
  -profile=kubernetes \
  ${WORKER_HOST}-csr.json | cfssljson -bare ${WORKER_HOST}
  
>> k8s-worker-1-key.pem, k8s-worker-1.csr, k8s-worker-1.pem

# Kube Proxy Client Certificate

Write kube-proxy-csr.json

{
    "CN": "system:kube-proxy",
    "key": {
      "algo": "rsa",
      "size": 2048
    },
    "names": [
      {
        "C": "South Korea",
        "L": "Seoul",
        "O": "system:node-proxier",
        "OU": "Kubernetes the Hard Way",
        "ST": "Seoul"
      }
    ]
}

Generate Kube Proxy Client certificate

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  kube-proxy-csr.json | cfssljson -bare kube-proxy
  
>> kube-proxy.csr, kube-proxy.pem, kube-proxy-key.pem

# Controller Manager Client Certificate

Write kube-controller-manager-csr.json

{
    "CN": "system:kube-controller-manager",
    "key": {
      "algo": "rsa",
      "size": 2048
    },
    "names": [
      {
        "C": "South Korea",
        "L": "Seoul",
        "O": "system:kube-controller-manager",
        "OU": "Kubernetes the Hard Way",
        "ST": "Seoul"
      }
    ]
}

Generate Controller Manager Client Certificate

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
  
  >> kube-controller-manager-csr, kube-controller-manager.pem, kube-controller-manager-key.pem

# Service Account Key Pair

Write service-account-csr.json

{
    "CN": "service-accounts",
    "key": {
      "algo": "rsa",
      "size": 2048
    },
    "names": [
      {
        "C": "South Korea",
        "L": "Seoul",
        "O": "Kubernetes",
        "OU": "Kubernetes the Hard Way",
        "ST": "Seoul"
      }
    ]
}

Generate Service Account Key pair

- The Kubernetes Controller Manager leverages a key pair to generate and sign service account tokens(https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/)

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  service-account-csr.json | cfssljson -bare service-account
  
>> service-account.csr, service-account.pem, service-account-key.pem

# Scheduler Client Certificate

Write kube-scheduler-csr.json

{
    "CN": "system:kube-scheduler",
    "key": {
      "algo": "rsa",
      "size": 2048
    },
    "names": [
      {
        "C": "South Korea",
        "L": "Seoul",
        "O": "system:kube-scheduler",
        "OU": "Kubernetes the Hard Way",
        "ST": "Seoul"
      }
    ]
}

Generate Kube Scheduler Client certificate

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  kube-scheduler-csr.json | cfssljson -bare kube-scheduler
  
>> kube-scheduler.csr, kube-scheduler.pem, kube-scheduler-key.pem

# Kubernetes API Server Certificate

Write kubernetes-csr.json

{
    "CN": "kubernetes",
    "key": {
      "algo": "rsa",
      "size": 2048
    },
    "names": [
      {
        "C": "South Korea",
        "L": "Seoul",
        "O": "Kubernetes",
        "OU": "Kubernetes the Hard Way",
        "ST": "Seoul"
      }
    ]
}

Generate Kubernetes API Server certificate

- The Kubernetes API server is automatically assigned the kubernetes internal dns name, which will be linked to the first IP address(10.32.0.1) from the address range(10.32.0.0/24) reserved for internal cluster services during the control plane bootstrapping.

- Master node IP address will be included in the list of subject alternative names for the Kubernetes API Server certificate. This will ensure the certificate can be validated by remote clients.

- LB 사용시에 CERT_HOSTNAME에 LB 호스트명과 IP도 추가한다.

CERT_HOSTNAME=10.32.0.1,172.30.1.40,k8s-master-1,172.30.1.41,k8s-master-2,127.0.0.1,localhost,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.svc.cluster.local

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -hostname=${CERT_HOSTNAME} \
  -profile=kubernetes \
  kubernetes-csr.json | cfssljson -bare kubernetes
  
>> kubernetes.csr, kubernetes.pem, kubernetes-key.pem

3. Generating Kubernetes Configuration Files for Authentication

kubeconfigs enable k8s clients to locate and authenticate to the kubernetes api servers.

Worker nodes need

- ${worker_node}.kubeconfig (for Kubelet)

- kube-proxy.kubeconfig (for Kube-proxy)

# Generate Kubelet-kubeconfig

- When generating kubeconfig files for kubelets the client certificate matching the kubelet's node name must be used. This will ensure kubelets are properly authorized bye the kubernetes Node Authorizer.

- KUBERNETES_ADDRESS 를 마스터 노드의 IP로 설정하였지만 앞단에 LB를 두는 경우 LB의 IP를 지정.

KUBERNETES_ADDRESS=172.30.1.40
INSTANCE=k8s-worker-2

kubectl config set-cluster kubernetes-the-hard-way \
        --certificate-authority=ca.pem \
        --embed-certs=true \
        --server=https://${KUBERNETES_ADDRESS}:6443 \
        --kubeconfig=${INSTANCE}.kubeconfig

kubectl config set-credentials system:node:${INSTANCE} \
    --client-certificate=${INSTANCE}.pem \
    --client-key=${INSTANCE}-key.pem \
    --embed-certs=true \
    --kubeconfig=${INSTANCE}.kubeconfig

kubectl config set-context default \
    --cluster=kubernetes-the-hard-way \
    --user=system:node:${INSTANCE} \
    --kubeconfig=${INSTANCE}.kubeconfig
    
kubectl config use-context default --kubeconfig=${INSTANCE}.kubeconfig

>> k8s-worker-1.kubeconfig, k8s-worker-2.kubeconfig

# Generate kubeproxy-kubeconfig

KUBERNETES_ADDRESS=172.30.1.40

kubectl config set-cluster kubernetes-the-hard-way \
    --certificate-authority=ca.pem \
    --embed-certs=true \
    --server=https://${KUBERNETES_ADDRESS}:6443 \
    --kubeconfig=kube-proxy.kubeconfig

kubectl config set-credentials system:kube-proxy \
--client-certificate=kube-proxy.pem \
--client-key=kube-proxy-key.pem \
--embed-certs=true \
--kubeconfig=kube-proxy.kubeconfig

kubectl config set-context default \
--cluster=kubernetes-the-hard-way \
--user=system:kube-proxy \
--kubeconfig=kube-proxy.kubeconfig

kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

>> kube-proxy.kubeconfig

Master nodes needs

- admin.kubeconfig (for user admin)

- kube-controller-manager.kubeconfig (for kube-controller-manager)

- kube-scheduler.kubeconfig (for kube-scheduler)

# Generate admin.kubeconfig

kubectl config set-cluster kubernetes-the-hard-way \
    --certificate-authority=ca.pem \
    --embed-certs=true \
    --server=https://127.0.0.1:6443 \
    --kubeconfig=admin.kubeconfig

kubectl config set-credentials admin \
    --client-certificate=admin.pem \
    --client-key=admin-key.pem \
    --embed-certs=true \
    --kubeconfig=admin.kubeconfig

kubectl config set-context default \
    --cluster=kubernetes-the-hard-way \
    --user=admin \
    --kubeconfig=admin.kubeconfig

kubectl config use-context default --kubeconfig=admin.kubeconfig

>> admin.kubeconfig

# Generate kube-controller-manager-kubeconfig

kubectl config set-cluster kubernetes-the-hard-way \
    --certificate-authority=ca.pem \
    --embed-certs=true \
    --server=https://127.0.0.1:6443 \
    --kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-credentials system:kube-controller-manager \
    --client-certificate=kube-controller-manager.pem \
    --client-key=kube-controller-manager-key.pem \
    --embed-certs=true \
    --kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-context default \
    --cluster=kubernetes-the-hard-way \
    --user=system:kube-controller-manager \
    --kubeconfig=kube-controller-manager.kubeconfig

kubectl config use-context default --kubeconfig=kube-controller-manager.kubeconfig

>> kube-controller-manager.kubeconfig

# Generate kubescheduler-kubeconfig

kubectl config set-cluster kubernetes-the-hard-way \
    --certificate-authority=ca.pem \
    --embed-certs=true \
    --server=https://127.0.0.1:6443 \
    --kubeconfig=kube-scheduler.kubeconfig

kubectl config set-credentials system:kube-scheduler \
    --client-certificate=kube-scheduler.pem \
    --client-key=kube-scheduler-key.pem \
    --embed-certs=true \
    --kubeconfig=kube-scheduler.kubeconfig

kubectl config set-context default \
    --cluster=kubernetes-the-hard-way \
    --user=system:kube-scheduler \
    --kubeconfig=kube-scheduler.kubeconfig

kubectl config use-context default --kubeconfig=kube-scheduler.kubeconfig

>> kube-scheduler.kubeconfig

4. Generating the Data Encryption Config and Key(Encrypt)

- Kubernetes stores a variety of data including cluster state, application configurations, and secrets. Kubernetes supports the ability to encrypt cluster data at rest.

- Generate an encryption key and an encryption config suitable for encrypting Kubernetes Secrets.

ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)

cat > encryption-config.yaml << EOF
kind: EncryptionConfig
apiVersion: v1
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: ${ENCRYPTION_KEY}
      - identity: {}
EOF

5. Distribute the Client and Server Certificates

Copy certificate to worker nodes

sudo scp ca.pem k8s-worker-1-key.pem k8s-worker-1.pem k8s-worker-1.kubeconfig kube-proxy.kubeconfig pi@172.30.1.42:~/
sudo scp ca.pem k8s-worker-2-key.pem k8s-worker-2.pem k8s-worker-2.kubeconfig kube-proxy.kubeconfig pi@172.30.1.43:~/

Copy certificates to master nodes

sudo scp ca.pem ca-key.pem kubernetes-key.pem kubernetes.pem service-account-key.pem service-account.pem encryption-config.yaml kube-controller-manager.kubeconfig kube-scheduler.kubeconfig admin.kubeconfig encryption-config.yaml pi@172.30.1.40:~/
sudo scp ca.pem ca-key.pem kubernetes-key.pem kubernetes.pem service-account-key.pem service-account.pem encryption-config.yaml kube-controller-manager.kubeconfig kube-scheduler.kubeconfig admin.kubeconfig encryption-config.yaml pi@172.30.1.41:~/

6. Bootstrapping the ETCD cluster

Kubernetes components are stateless and store cluster state in etcd.

# Download and Install the etcd binaries

wget -q --show-progress --https-only --timestamping   "https://github.com/etcd-io/etcd/releases/download/v3.4.15/etcd-v3.4.15-linux-arm64.tar.gz"
tar -xvf etcd-v3.4.15-linux-arm64.tar.gz
sudo mv etcd-v3.4.15-linux-arm64/etcd* /usr/local/bin/

---
wget https://raw.githubusercontent.com/robertojrojas/kubernetes-the-hard-way-raspberry-pi/master/etcd/etcd-3.1.5-arm.tar.gz
tar -xvf etcd-3.1.5-arm.tar.gz

# Certs to their desired location

sudo mkdir -p /etc/etcd /var/lib/etcd
sudo chmod 700 /var/lib/etcd
sudo cp ca.pem kubernetes-key.pem kubernetes.pem /etc/etcd/

# Create the etcd.service systemd unit file

주요 파라미터

- initial-advertise-peer-urls

- listen-peer-urls

- listen-client-urls

- advertise-client-urls

- initial-cluster

ETCD_NAME=k8s-master-1
INTERNAL_IP=172.30.1.40
INITIAL_CLUSTER=k8s-master-1=https://172.30.1.40:2380,k8s-master-2=https://172.30.1.41:2380

cat << EOF | sudo tee /etc/systemd/system/etcd.service
[Unit]
Description=etcd
Documentation=https://github.com/coreos

[Service]
ExecStart=/usr/local/bin/etcd \\
  --name ${ETCD_NAME} \\
  --cert-file=/etc/etcd/kubernetes.pem \\
  --key-file=/etc/etcd/kubernetes-key.pem \\
  --peer-cert-file=/etc/etcd/kubernetes.pem \\
  --peer-key-file=/etc/etcd/kubernetes-key.pem \\
  --trusted-ca-file=/etc/etcd/ca.pem \\
  --peer-trusted-ca-file=/etc/etcd/ca.pem \\
  --peer-client-cert-auth \\
  --client-cert-auth \\
  --initial-advertise-peer-urls https://${INTERNAL_IP}:2380 \\
  --listen-peer-urls https://${INTERNAL_IP}:2380 \\
  --listen-client-urls https://${INTERNAL_IP}:2379,https://127.0.0.1:2379 \\
  --advertise-client-urls https://${INTERNAL_IP}:2379 \\
  --initial-cluster-token etcd-cluster-0 \\
  --initial-cluster ${INITIAL_CLUSTER} \\
  --initial-cluster-state new \\
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
Environment="ETCD_UNSUPPORTED_ARCH=arm64"

[Install]
WantedBy=multi-user.target
EOF

# Strat etcd

sudo systemctl daemon-reload
sudo systemctl enable etcd
sudo systemctl start etcd
sudo systemctl status etcd

# Verification

sudo ETCDCTL_API=3 etcdctl member list \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/etcd/ca.pem \
  --cert=/etc/etcd/kubernetes.pem \
  --key=/etc/etcd/kubernetes-key.pem

7. Bootstrapping the Kubernetes Control Plane

The following components will be installed on each master node : Kubernetes API Server, Scheduler and Controller Manager

Kubernetes API Server

# Download and move them to the right position

sudo mkdir -p /etc/kubernetes/config

wget -q --show-progress --https-only --timestamping \
  "https://storage.googleapis.com/kubernetes-release/release/v1.21.0/bin/linux/arm64/kube-apiserver" \
  "https://storage.googleapis.com/kubernetes-release/release/v1.21.0/bin/linux/arm64/kube-controller-manager" \
  "https://storage.googleapis.com/kubernetes-release/release/v1.21.0/bin/linux/arm64/kube-scheduler" \
  "https://storage.googleapis.com/kubernetes-release/release/v1.21.0/bin/linux/arm64/kubectl"

chmod +x kube-apiserver kube-controller-manager kube-scheduler kubectl
  sudo mv kube-apiserver kube-controller-manager kube-scheduler kubectl /usr/local/bin/

# Certs to their desired location

sudo mkdir -p /var/lib/kubernetes/

sudo cp ca.pem ca-key.pem kubernetes-key.pem kubernetes.pem \
  service-account-key.pem service-account.pem \
  encryption-config.yaml /var/lib/kubernetes/

# Create the kube.apiserver systemd unit file

주요 파라미터

- advertise-address : ${INTERNAL_IP}

- apiserver-count : 2

- etcd-servers : ${CONTROLLER0_IP}:2379,${CONTROLLER1_IP}:2379

- service-account-issuer : https://${KUBERNETES_PUBLIC_ADDRESS}:6443

- service-cluster-ip-range : 10.32.0.0/24

INTERNAL_IP=172.30.1.40
KUBERNETES_PUBLIC_ADDRESS=172.30.1.40
CONTROLLER0_IP=172.30.1.40
CONTROLLER1_IP=172.30.1.41
cat << EOF | sudo tee /etc/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-apiserver \\
  --advertise-address=${INTERNAL_IP} \\
  --allow-privileged=true \\
  --apiserver-count=2 \\
  --audit-log-maxage=30 \\
  --audit-log-maxbackup=3 \\
  --audit-log-maxsize=100 \\
  --audit-log-path=/var/log/audit.log \\
  --authorization-mode=Node,RBAC \\
  --bind-address=0.0.0.0 \\
  --client-ca-file=/var/lib/kubernetes/ca.pem \\
  --enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \\
  --etcd-cafile=/var/lib/kubernetes/ca.pem \\
  --etcd-certfile=/var/lib/kubernetes/kubernetes.pem \\
  --etcd-keyfile=/var/lib/kubernetes/kubernetes-key.pem \\
  --etcd-servers=https://${CONTROLLER0_IP}:2379,https://${CONTROLLER1_IP}:2379 \\
  --event-ttl=1h \\
  --encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml \\
  --kubelet-certificate-authority=/var/lib/kubernetes/ca.pem \\
  --kubelet-client-certificate=/var/lib/kubernetes/kubernetes.pem \\
  --kubelet-client-key=/var/lib/kubernetes/kubernetes-key.pem \\
  --runtime-config='api/all=true' \\
  --service-account-key-file=/var/lib/kubernetes/service-account.pem \\
  --service-account-signing-key-file=/var/lib/kubernetes/service-account-key.pem \\
  --service-account-issuer=https://${KUBERNETES_PUBLIC_ADDRESS}:6443 \\
  --service-cluster-ip-range=10.32.0.0/24 \\
  --service-node-port-range=30000-32767 \\
  --tls-cert-file=/var/lib/kubernetes/kubernetes.pem \\
  --tls-private-key-file=/var/lib/kubernetes/kubernetes-key.pem \\
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

# Start kube-apiserver

sudo systemctl daemon-reload
sudo systemctl enable kube-apiserver
sudo systemctl start kube-apiserver

sudo systemctl status kube-apiserver

Controller Manager

# Certs to their desired location

sudo cp kube-controller-manager.kubeconfig /var/lib/kubernetes/

# Create the kube-controller-manager systemd unit file

주요 파라미터

- cluster-cidr : 10.200.0.0/16

- service-cluster-ip-range : 10.32.0.0/24

cat << EOF | sudo tee /etc/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-controller-manager \\
  --address=0.0.0.0 \\
  --cluster-cidr=10.200.0.0/16 \\
  --cluster-name=kubernetes \\
  --cluster-signing-cert-file=/var/lib/kubernetes/ca.pem \\
  --cluster-signing-key-file=/var/lib/kubernetes/ca-key.pem \\
  --kubeconfig=/var/lib/kubernetes/kube-controller-manager.kubeconfig \\
  --leader-elect=true \\
  --root-ca-file=/var/lib/kubernetes/ca.pem \\
  --service-account-private-key-file=/var/lib/kubernetes/service-account-key.pem \\
  --service-cluster-ip-range=10.32.0.0/24 \\
  --use-service-account-credentials=true \\
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

# Start kube-controller-manager

sudo systemctl daemon-reload
sudo systemctl enable kube-controller-manager
sudo systemctl start kube-controller-manager

sudo systemctl status kube-controller-manager

Kube Scheduler

# Certs to their desired location

sudo cp kube-scheduler.kubeconfig /var/lib/kubernetes/

# Create the kube-scheduler systemd unit file

cat << EOF | sudo tee /etc/kubernetes/config/kube-scheduler.yaml
apiVersion: kubescheduler.config.k8s.io/v1beta1
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: "/var/lib/kubernetes/kube-scheduler.kubeconfig"
leaderElection:
  leaderElect: true
EOF


cat << EOF | sudo tee /etc/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-scheduler \\
  --config=/etc/kubernetes/config/kube-scheduler.yaml \\
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

# Start kube-scheduler

sudo systemctl daemon-reload
sudo systemctl enable kube-scheduler
sudo systemctl start kube-scheduler

sudo systemctl status kube-scheduler

RBAC for Kubelet Authoriztion

- kube-apiserver가 worker node에 대한 정보를 요청 권한을 가지도록

- Access to tke kubelet API is requried for retrieving metrics, logs and executing commands in pods.

# Create an Admin ClusterRole and bind it to kubeconfig

ClusterRole

cat << EOF | kubectl apply --kubeconfig admin.kubeconfig -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
    verbs:
      - "*"
EOF

ClusterRoleBinding

cat << EOF | kubectl apply --kubeconfig admin.kubeconfig -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kubernetes
EOF

# Verify All component working properly

kubectl get componentstatuses --kubeconfig admin.kubeconfig
NAME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok                   
scheduler            Healthy   ok                   
etcd-0               Healthy   {"health": "true"}   
etcd-1               Healthy   {"health": "true"}

(Optional) Enable HTTP Health Checks

# Install nginx and expose health check over http

sudo apt-get update
sudo apt-get install -y nginx

# Config proxying health check

cat > kubernetes.default.svc.cluster.local <<EOF
server {
  listen      80;
  server_name kubernetes.default.svc.cluster.local;

  location /healthz {
     proxy_pass                    https://127.0.0.1:6443/healthz;
     proxy_ssl_trusted_certificate /var/lib/kubernetes/ca.pem;
  }
}
EOF

sudo mv kubernetes.default.svc.cluster.local \
	/etc/nginx/sites-available/kubernetes.default.svc.cluster.local

sudo ln -s /etc/nginx/sites-available/kubernetes.default.svc.cluster.local /etc/nginx/sites-enabled/

# start nginx and verify health check

sudo systemctl restart nginx
sudo systemctl enable nginx

kubectl cluster-info --kubeconfig admin.kubeconfig
curl -H "Host: kubernetes.default.svc.cluster.local" -i http://127.0.0.1/healthz

(Optional) 8. Kubectl remote access from local machine

kubectl config set-cluster kubernetes-the-hard-way \
  --certificate-authority='ca.pem' \
  --embed-certs=true \
  --server=https://172.30.1.40:6443

kubectl config set-credentials admin \
  --client-certificate='admin.pem' \
  --client-key='admin-key.pem'

kubectl config set-context kubernetes-the-hard-way \
  --cluster=kubernetes-the-hard-way \
  --user=admin

kubectl config use-context kubernetes-the-hard-way

9. Bootstrapping the Kubernetes Worker Nodes

- The following components will be installed on each node: runc, container, networking plugins, containerd, kubelet, and kube-proxy

# Install the OS dependencies & Disable Swap

sudo apt-get update
sudo apt-get -y install socat conntrack ipset

sudo swapoff -a

# Download binaries and move them to desired location

wget -q --show-progress --https-only --timestamping \
  https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.21.0/crictl-v1.21.0-linux-arm64.tar.gz \
 https://github.com/opencontainers/runc/releases/download/v1.1.0/runc.arm64 \
  https://github.com/containernetworking/plugins/releases/download/v0.9.1/cni-plugins-linux-arm64-v0.9.1.tgz \
  https://github.com/containerd/containerd/releases/download/v1.6.1/containerd-1.6.1-linux-arm64.tar.gz \
  https://storage.googleapis.com/kubernetes-release/release/v1.21.0/bin/linux/arm64/kubectl \
  https://storage.googleapis.com/kubernetes-release/release/v1.21.0/bin/linux/arm64/kube-proxy \
  https://storage.googleapis.com/kubernetes-release/release/v1.21.0/bin/linux/arm64/kubelet

sudo mkdir -p \
  /etc/cni/net.d \
  /opt/cni/bin \
  /var/lib/kubelet \
  /var/lib/kube-proxy \
  /var/lib/kubernetes \
  /var/run/kubernetes

mkdir containerd
tar -xvf crictl-v1.21.0-linux-arm64.tar.gz
tar -xvf containerd-1.6.1-linux-arm64.tar.gz -C containerd
sudo tar -xvf cni-plugins-linux-arm64-v0.9.1.tgz -C /opt/cni/bin/
sudo mv runc.arm64 runc
chmod +x crictl kubectl kube-proxy kubelet runc 
sudo mv crictl kubectl kube-proxy kubelet runc /usr/local/bin/
sudo mv containerd/bin/* /bin/

# Configure CNI Networking

- Create the bridge network and loopback network configuration

POD_CIDR=10.200.1.0/24

cat <<EOF | sudo tee /etc/cni/net.d/10-bridge.conf
{
    "cniVersion": "0.4.0",
    "name": "bridge",
    "type": "bridge",
    "bridge": "cnio0",
    "isGateway": true,
    "ipMasq": true,
    "ipam": {
        "type": "host-local",
        "ranges": [
          [{"subnet": "${POD_CIDR}"}]
        ],
        "routes": [{"dst": "0.0.0.0/0"}]
    }
}
EOF

cat <<EOF | sudo tee /etc/cni/net.d/99-loopback.conf
{
    "cniVersion": "0.4.0",
    "name": "lo",
    "type": "loopback"
}
EOF

CNI

# Configure Containerd

sudo mkdir -p /etc/containerd/

cat << EOF | sudo tee /etc/containerd/config.toml
[plugins]
  [plugins.cri.containerd]
    snapshotter = "overlayfs"
    [plugins.cri.containerd.default_runtime]
      runtime_type = "io.containerd.runtime.v1.linux"
      runtime_engine = "/usr/local/bin/runc"
      runtime_root = ""
EOF

# Create the containerd systemd unit file

cat <<EOF | sudo tee /etc/systemd/system/containerd.service
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target

[Service]
ExecStartPre=/sbin/modprobe overlay
ExecStart=/bin/containerd
Restart=always
RestartSec=5
Delegate=yes
KillMode=process
OOMScoreAdjust=-999
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity

[Install]
WantedBy=multi-user.target
EOF

# Start containerd

sudo systemctl daemon-reload
sudo systemctl enable containerd
sudo systemctl start containerd

Kubelet

# Certi to the desired location

sudo cp k8s-worker-1-key.pem k8s-worker-1.pem /var/lib/kubelet/
sudo cp k8s-worker-1.kubeconfig /var/lib/kubelet/kubeconfig

# Configure the kubelet

- authorization mode : webhook

POD_CIDR=10.200.0.0/24

cat <<EOF | sudo tee /var/lib/kubelet/kubelet-config.yaml
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true
  x509:
    clientCAFile: "/var/lib/kubernetes/ca.pem"
authorization:
  mode: Webhook
clusterDomain: "cluster.local"
clusterDNS:
  - "10.32.0.10"
podCIDR: "${POD_CIDR}"
resolvConf: "/run/systemd/resolve/resolv.conf"
runtimeRequestTimeout: "15m"
tlsCertFile: "/var/lib/kubelet/${HOSTNAME}.pem"
tlsPrivateKeyFile: "/var/lib/kubelet/${HOSTNAME}-key.pem"
EOF

# Create the kubelet systemd unit file

cat <<EOF | sudo tee /etc/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=containerd.service
Requires=containerd.service

[Service]
ExecStart=/usr/local/bin/kubelet \\
  --config=/var/lib/kubelet/kubelet-config.yaml \\
  --container-runtime=remote \\
  --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock \\
  --image-pull-progress-deadline=2m \\
  --kubeconfig=/var/lib/kubelet/kubeconfig \\
  --network-plugin=cni \\
  --register-node=true \\
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

# Start kubelet

sudo systemctl daemon-reload
sudo systemctl enable kubelet
sudo systemctl start kubelet

* reboot 후 아래 메시지 없어짐.

"Failed to get the kubelet's cgroup. Kubelet system container metrics may be missing." err="mountpoint for memory not found"

Kube-proxy

# Certi to the desired location

sudo mv kube-proxy.kubeconfig /var/lib/kube-proxy/kubeconfig

# Configuration & Systemd unit

cat <<EOF | sudo tee /var/lib/kube-proxy/kube-proxy-config.yaml
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
clientConnection:
  kubeconfig: "/var/lib/kube-proxy/kubeconfig"
mode: "iptables"
clusterCIDR: "10.200.0.0/16"
EOF

cat <<EOF | sudo tee /etc/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-proxy \\
  --config=/var/lib/kube-proxy/kube-proxy-config.yaml
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

# Start kubelet, kube-proxy

sudo systemctl daemon-reload
sudo systemctl enable kubelet kube-proxy
sudo systemctl start kubelet kube-proxy

# Verification

ubectl get no --kubeconfig=admin.kubeconfig
\NAME           STATUS   ROLES    AGE   VERSION
k8s-worker-1   Ready    <none>   40m   v1.21.0
k8s-worker-2   Ready    <none>   40m   v1.21.0

10. Networking

Pods scheduled to a node receive an IP address from the node's Pod CIDR range. At this point pods can not communicate with other pods running on different nodes due to missing network routes.

We are going to install Flannel to implement the Kubernetes networking model.

Flannel

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml --kubeconfig=admin.kubeconfig

- pod cidr not assigned

kubectl logs -f kube-flannel-ds-x56l5 -n kube-system --kubeconfig=admin.kubeconfig

I0316 05:37:58.801461       1 main.go:205] CLI flags config: {etcdEndpoints:http://127.0.0.1:4001,http://127.0.0.1:2379 etcdPrefix:/coreos.com/network etcdKeyfile: etcdCertfile: etcdCAFile: etcdUsername: etcdPassword: version:false kubeSubnetMgr:true kubeApiUrl: kubeAnnotationPrefix:flannel.alpha.coreos.com kubeConfigFile: iface:[] ifaceRegex:[] ipMasq:true subnetFile:/run/flannel/subnet.env publicIP: publicIPv6: subnetLeaseRenewMargin:60 healthzIP:0.0.0.0 healthzPort:0 iptablesResyncSeconds:5 iptablesForwardRules:true netConfPath:/etc/kube-flannel/net-conf.json setNodeNetworkUnavailable:true}
W0316 05:37:58.801882       1 client_config.go:614] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0316 05:37:59.907489       1 kube.go:120] Waiting 10m0s for node controller to sync
I0316 05:37:59.907871       1 kube.go:378] Starting kube subnet manager
I0316 05:38:00.908610       1 kube.go:127] Node controller sync successful
I0316 05:38:00.908732       1 main.go:225] Created subnet manager: Kubernetes Subnet Manager - k8s-worker-1
I0316 05:38:00.908762       1 main.go:228] Installing signal handlers
I0316 05:38:00.909447       1 main.go:454] Found network config - Backend type: vxlan
I0316 05:38:00.909557       1 match.go:189] Determining IP address of default interface
I0316 05:38:00.911032       1 match.go:242] Using interface with name eth0 and address 172.30.1.42
I0316 05:38:00.911156       1 match.go:264] Defaulting external address to interface address (172.30.1.42)
I0316 05:38:00.911399       1 vxlan.go:138] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false
E0316 05:38:00.912700       1 main.go:317] Error registering network: failed to acquire lease: node "k8s-worker-1" pod cidr not assigned
I0316 05:38:00.913055       1 main.go:434] Stopping shutdownHandler...
W0316 05:38:00.913524       1 reflector.go:436] github.com/flannel-io/flannel/subnet/kube/kube.go:379: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding

아래 명령으로 해결

kubectl patch node k8s-worker-1 -p '{"spec":{"podCIDR":"10.200.0.0/24"}}' --kubeconfig=admin.kubeconfig
kubectl patch node k8s-worker-2 -p '{"spec":{"podCIDR":"10.200.1.0/24"}}' --kubeconfig=admin.kubeconfig

10. 클러스터 구축 결과 및 테스트

워커노드 확인

kubectl get no --kubeconfig admin.kubeconfig
NAME           STATUS   ROLES    AGE    VERSION
k8s-worker-1   Ready    <none>   110m   v1.21.0
k8s-worker-2   Ready    <none>   53m    v1.21.0

Flannel 배포 확인

kubectl get po -n kube-system -o wide --kubeconfig admin.kubeconfig -w
NAME                    READY   STATUS    RESTARTS   AGE     IP            NODE           NOMINATED NODE   READINESS GATES
kube-flannel-ds-hbk2w   1/1     Running   0          3m18s   172.30.1.43   k8s-worker-2   <none>           <none>
kube-flannel-ds-kmx9k   1/1     Running   0          3m18s   172.30.1.42   k8s-worker-1   <none>           <none>

Nginx 배포 및 확인

cat << EOF | kubectl apply -f -
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: nginx
 spec:
   selector:
     matchLabels:
       run: nginx
   replicas: 2
   template:
     metadata:
       labels:
         run: nginx
     spec:
       containers:
       - name: my-nginx
         image: nginx
         ports:
         - containerPort: 80
EOF
deployment.apps/nginx created

kubectl get po -o wide  --kubeconfig admin.kubeconfig -w
NAME                     READY   STATUS    RESTARTS   AGE   IP            NODE           NOMINATED NODE   READINESS GATES
nginx-7bddd8c596-6x7gn   1/1     Running   0          44s   10.200.0.10   k8s-worker-1   <none>           <none>
nginx-7bddd8c596-vptkl   1/1     Running   0          44s   10.200.1.2    k8s-worker-2   <none>           <none>

kubectl run curl-tester --image=nginx --kubeconfig admin.kubeconfig

kubectl exec -it curl-tester --kubeconfig admin.kubeconfig -- curl http://10.200.1.2 
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

kubectl exec -it curl-tester --kubeconfig admin.kubeconfig -- curl http://10.200.0.10 
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

11. Deploying the DNS Cluster Add-on

DNS add-on provides DNS based service discovery, backed by CoreDNS, to applications running inside the Kubernetes cluster.

# Deploy the coredns cluster add-on

kubectl apply -f https://storage.googleapis.com/kubernetes-the-hard-way/coredns-1.8.yaml

kubectl get svc -n kube-system --kubeconfig admin.kubeconfig
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.32.0.10   <none>        53/UDP,53/TCP,9153/TCP   44m

kubectl get svc --kubeconfig admin.kubeconfig
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.32.0.1    <none>        443/TCP   6h19m
nginx        ClusterIP   10.32.0.37   <none>        80/TCP    40m

kubectl get po -n kube-system --kubeconfig admin.kubeconfig -o wide
NAME                       READY   STATUS    RESTARTS   AGE   IP            NODE           NOMINATED NODE   READINESS GATES
coredns-8494f9c688-658c2   1/1     Running   0          69m   10.200.0.12   k8s-worker-1   <none>           <none>
coredns-8494f9c688-k8lql   1/1     Running   0          69m   10.200.1.4    k8s-worker-2   <none>           <none>

# Verification

kubectl exec -it busybox --kubeconfig admin.kubeconfig -- nslookup nginx
Server:		10.32.0.10
Address:	10.32.0.10:53

Name:	nginx.default.svc.cluster.local
Address: 10.32.0.37

kubectl exec -it curl-tester --kubeconfig admin.kubeconfig -- curl http://nginx.default.svc.cluster.local
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

# 디버깅

kubectl exec -it busybox --kubeconfig admin.kubeconfig -- cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.32.0.10
options ndots:5
E0316 08:14:40.416650   23982 v3.go:79] EOF

[INFO] 10.200.0.11:48289 - 37888 "A IN kubernetes.default.svc.cluster.local. udp 54 false 512" NOERROR qr,aa,rd 106 0.000881653s
[INFO] 10.200.0.11:48289 - 37888 "A IN kubernetes.svc.cluster.local. udp 46 false 512" NXDOMAIN qr,aa,rd 139 0.001551175s
[INFO] 10.200.0.11:48289 - 37888 "AAAA IN kubernetes.default.svc.cluster.local. udp 54 false 512" NOERROR qr,aa,rd 147 0.000719937s
[INFO] 10.200.0.11:48289 - 37888 "AAAA IN kubernetes.svc.cluster.local. udp 46 false 512" NXDOMAIN qr,aa,rd 139 0.001028995s
[INFO] 10.200.0.11:48289 - 37888 "A IN kubernetes.cluster.local. udp 42 false 512" NXDOMAIN qr,aa,rd 135 0.001927212s
[INFO] 10.200.0.11:48289 - 37888 "AAAA IN kubernetes.cluster.local. udp 42 false 512" NXDOMAIN qr,aa,rd 135 0.00215127s

kubectl edit -n kube-system configmaps coredns --kubeconfig admin.kubeconfig

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        cache 30
        loop
        reload
        loadbalance
        log
    }

간단하게 Corefile 구성을 살펴보자.

errors : 에러가 발생하면 stdout으로 보낸다.
health : http://localhost:8080/health를 통해 CoreDNS 상태를 확인할 수 있다.
ready : 준비 요청이 되어 있는지 확인하기 위해 포트 http://localhost:8181/ready로 HTTP 요청을 보니면 200 OK가 반환된다.
kubenetes : 쿠버네티스의 Service 도메인과 POD IP 기반으로 DNS 쿼리를 응답한다. ttl 설정으로 타임아웃을 제어할 수 있다.
- pods 옵션은 POD IP를 기반으로 DNS 질의를 제어하기 위한 옵션이다. 기본값은 disabled이며, insecure값은 kube-dns 하위 호환성을 위해서 사용한다.
- pods disabled옵션을 설정하면 POD IP 기반 DNS 질의가 불가능하다. 예를 들어 testbed 네임스페이스에 있는 POD IP가 10.244.2.16라고 한다면, 10-244-2-16.testbed.pod.cluster.local질의에 A 레코드를 반환하지 않게 된다.
- pods insecure 옵션은 같은 네임스페이스에 일치하는 POD IP가 있는 경우에만 A 레코드를 반환한다고 되어 있다. 하지만 간단하게 테스트 해보기 위해 다른 네임스페이스 상에 POD를 만들고 서로 호출했을 때 계속 통신이 되었다. 제대로 이해를 못했거나 테스트 방식이 잘못된 것인지 잘 모르겠다. ㅠㅠ
prometheus : 지정한 포트(:9153)로 프로메테우스 포맷의 메트릭 정보를 확인할 수 있다. 위에서 다룬 health의 :8080포트나 ready 옵션의 :8181포트를 포함해서 CoreDNS로 HTTP 요청을 보내려면 CoreDNS Service 오브젝트 설정에 :9153, :8080, :8181 포트를 바인딩을 설정해야 한다.

12. Pod to Internet

참조 : https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/#configuration-of-stub-domain-and-upstream-nameserver-using-coredns

Corefile에 forward 추가.

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        cache 30
        loop
        reload
        loadbalance
        log
        forward . 8.8.8.8
    }

구글에 핑을 날려 확인

kubectl exec -it busybox --kubeconfig admin.kubeconfig -- ping www.google.com
PING www.google.com (172.217.175.36): 56 data bytes
64 bytes from 172.217.175.36: seq=0 ttl=113 time=31.225 ms
64 bytes from 172.217.175.36: seq=1 ttl=113 time=31.720 ms
64 bytes from 172.217.175.36: seq=2 ttl=113 time=32.036 ms
64 bytes from 172.217.175.36: seq=3 ttl=113 time=32.769 ms
64 bytes from 172.217.175.36: seq=4 ttl=113 time=31.890 ms

kubectl exec -it busybox --kubeconfig admin.kubeconfig -- nslookup nginx
Server:		10.32.0.10
Address:	10.32.0.10:53

Name:	nginx.default.svc.cluster.local
Address: 10.32.0.37

디버깅

[INFO] Reloading complete
[INFO] 127.0.0.1:51732 - 10283 "HINFO IN 790223070724377814.6189194672555963484. udp 56 false 512" NXDOMAIN qr,rd,ra,ad 131 0.033668927s
[INFO] 10.200.0.11:44705 - 5 "AAAA IN www.google.com. udp 32 false 512" NOERROR qr,rd,ra 74 0.034105169s
[INFO] 10.200.0.11:44224 - 6 "A IN www.google.com. udp 32 false 512" NOERROR qr,rd,ra 62 0.034133246s
[INFO] 10.200.0.11:41908 - 6656 "A IN nginx.svc.cluster.local. udp 41 false 512" NXDOMAIN qr,aa,rd 134 0.000872899s
[INFO] 10.200.0.11:41908 - 6656 "A IN nginx.cluster.local. udp 37 false 512" NXDOMAIN qr,aa,rd 130 0.001095134s
[INFO] 10.200.0.11:41908 - 6656 "AAAA IN nginx.default.svc.cluster.local. udp 49 false 512" NOERROR qr,aa,rd 142 0.001334348s
[INFO] 10.200.0.11:41908 - 6656 "AAAA IN nginx.svc.cluster.local. udp 41 false 512" NXDOMAIN qr,aa,rd 134 0.001338932s
[INFO] 10.200.0.11:41908 - 6656 "A IN nginx.default.svc.cluster.local. udp 49 false 512" NOERROR qr,aa,rd 96 0.002403233s
[INFO] 10.200.0.11:41908 - 6656 "AAAA IN nginx.cluster.local. udp 37 false 512" NXDOMAIN qr,aa,rd 130 0.00264s
[INFO] 10.200.0.11:41908 - 6656 "A IN nginx.default.svc.cluster.local. udp 49 false 512" NOERROR qr,aa,rd 96 0.000500251s
[INFO] 10.200.0.11:41908 - 6656 "AAAA IN nginx.cluster.local. udp 37 false 512" NXDOMAIN qr,aa,rd 130 0.000283484s
[INFO] 10.200.0.11:41908 - 6656 "AAAA IN nginx.default.svc.cluster.local. udp 49 false 512" NOERROR qr,aa,rd 142 0.000317755s
[INFO] 10.200.0.11:41908 - 6656 "A IN nginx.svc.cluster.local. udp 41 false 512" NXDOMAIN qr,aa,rd 134 0.000322389s
[INFO] 10.200.0.11:41908 - 6656 "AAAA IN nginx.svc.cluster.local. udp 41 false 512" NXDOMAIN qr,aa,rd 134 0.00025812s
[INFO] 10.200.0.11:41908 - 6656 "A IN nginx.cluster.local. udp 37 false 512" NXDOMAIN qr,aa,rd 130 0.000913992s
[INFO] 10.200.0.11:56880 - 7936 "A IN nginx.default.svc.cluster.local. udp 49 false 512" NOERROR qr,aa,rd 96 0.000680195s
[INFO] 10.200.0.11:56880 - 7936 "AAAA IN nginx.default.svc.cluster.local. udp 49 false 512" NOERROR qr,aa,rd 142 0.000718059s
[INFO] 10.200.0.11:56880 - 7936 "A IN nginx.cluster.local. udp 37 false 512" NXDOMAIN qr,aa,rd 130 0.001300027s
[INFO] 10.200.0.11:56880 - 7936 "AAAA IN nginx.cluster.local. udp 37 false 512" NXDOMAIN qr,aa,rd 130 0.002253237s
[INFO] 10.200.0.11:56880 - 7936 "AAAA IN nginx.svc.cluster.local. udp 41 false 512" NXDOMAIN qr,aa,rd 134 0.001340651s
[INFO] 10.200.0.11:56880 - 7936 "A IN nginx.svc.cluster.local. udp 41 false 512" NXDOMAIN qr,aa,rd 134 0.000766964s
[INFO] 10.200.0.11:56880 - 7936 "A IN nginx.default.svc.cluster.local. udp 49 false 512" NOERROR qr,aa,rd 96 0.000423221s
[INFO] 10.200.0.11:56880 - 7936 "A IN nginx.cluster.local. udp 37 false 512" NXDOMAIN qr,aa,rd 130 0.000411711s
[INFO] 10.200.0.11:56880 - 7936 "A IN nginx.svc.cluster.local. udp 41 false 512" NXDOMAIN qr,aa,rd 134 0.000823578s
[INFO] 10.200.0.11:56880 - 7936 "AAAA IN nginx.default.svc.cluster.local. udp 49 false 512" NOERROR qr,aa,rd 142 0.001064718s
[INFO] 10.200.0.11:56880 - 7936 "AAAA IN nginx.cluster.local. udp 37 false 512" NXDOMAIN qr,aa,rd 130 0.001470023s
[INFO] 10.200.0.11:56880 - 7936 "AAAA IN nginx.svc.cluster.local. udp 41 false 512" NXDOMAIN qr,aa,rd 134 0.001802204s

[INFO] Reloading
W0316 08:37:06.872387       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0316 08:37:06.878515       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
[INFO] plugin/reload: Running configuration MD5 = 4507d64c02fd8d12322b2944d3f2f975
[INFO] Reloading complete
[INFO] 127.0.0.1:37902 - 16284 "HINFO IN 968904221266650254.5526836193363943015. udp 56 false 512" NXDOMAIN qr,rd,ra,ad 131 0.034053296s
[INFO] 10.200.0.0:56308 - 7 "A IN www.google.com. udp 32 false 512" NOERROR qr,rd,ra 62 0.03481522s
[INFO] 10.200.0.0:60813 - 9216 "A IN nginx.default.svc.cluster.local. udp 49 false 512" NOERROR qr,aa,rd 96 0.001246092s
[INFO] 10.200.0.0:60813 - 9216 "A IN nginx.svc.cluster.local. udp 41 false 512" NXDOMAIN qr,aa,rd 134 0.001978799s
[INFO] 10.200.0.0:60813 - 9216 "A IN nginx.cluster.local. udp 37 false 512" NXDOMAIN qr,aa,rd 130 0.002515881s
[INFO] 10.200.0.0:60813 - 9216 "AAAA IN nginx.cluster.local. udp 37 false 512" NXDOMAIN qr,aa,rd 130 0.003136974s
[INFO] 10.200.0.0:60813 - 9216 "AAAA IN nginx.svc.cluster.local. udp 41 false 512" NXDOMAIN qr,aa,rd 134 0.004195045s
[INFO] 10.200.0.0:60813 - 9216 "AAAA IN nginx.default.svc.cluster.local. udp 49 false 512" NOERROR qr,aa,rd 142 0.004489419s
[INFO] 10.200.0.0:60813 - 9216 "A IN nginx.svc.cluster.local. udp 41 false 512" NXDOMAIN qr,aa,rd 134 0.000557082s
[INFO] 10.200.0.0:60813 - 9216 "A IN nginx.cluster.local. udp 37 false 512" NXDOMAIN qr,aa,rd 130 0.000709165s
[INFO] 10.200.0.0:60813 - 9216 "AAAA IN nginx.svc.cluster.local. udp 41 false 512" NXDOMAIN qr,aa,rd 134 0.002432965s
[INFO] 10.200.0.0:60813 - 9216 "AAAA IN nginx.cluster.local. udp 37 false 512" NXDOMAIN qr,aa,rd 130 0.002911193s
[INFO] 10.200.0.0:60813 - 9216 "AAAA IN nginx.default.svc.cluster.local. udp 49 false 512" NOERROR qr,aa,rd 142 0.003832859s
W0316 08:45:26.938383       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice

- https://www.daveevans.us/posts/kubernetes-on-raspberry-pi-the-hard-way-part-1/

Kubernetes on Raspberry Pi, The Hard Way - Part 1

Starting a Kubernetes project for Raspberry Pi.

www.daveevans.us

- https://www.daveevans.us/posts/kubernetes-on-raspberry-pi-the-hard-way-part-1/

- https://github.com/cloudfoundry-incubator/kubo-deployment/issues/346

- https://github.com/robertojrojas/kubernetes-the-hard-way-raspberry-pi/blob/master/docs/04-kubernetes-controller.md

- https://phoenixnap.com/kb/enable-ssh-raspberry-pi

- https://github.com/Nek0trkstr/Kubernetes-The-Hard-Way-Raspberry-Pi/tree/master/00-before_you_begin

- https://github.com/kelseyhightower/kubernetes-the-hard-way/tree/master/docs

- k8s 인증 : https://coffeewhale.com/kubernetes/authentication/x509/2020/05/02/auth01/

- coredns : https://jonnung.dev/kubernetes/2020/05/11/kubernetes-dns-about-coredns/

'클라우드 컴퓨팅 > 쿠버네티스' 카테고리의 다른 글

Build k8s Cluster on Raspberry Pi 4B(Ubuntu) through Kubeadm (0)	2022.01.18
Kubernetes Networking (0)	2021.12.05
Docker Network (0)	2021.12.05
Kubernetes Storage (0)	2021.11.22
Kubernetes Security - Authentication/Authorization (0)	2021.11.10

Slow And Steady Wins The Race