てっくらのーと
てっくらのーとは、触れた技術のメモと日常の記録が少し合わさった個人のサイトです。

k8s アップグレード作業メモ

January 17, 2021

Kubernetesをv1.20.2にアップグレードしたのでそのメモ。公式にアップグレードのコマンド等が記載されているのでこちらをもとに進めます。

環境

❯ kubectl get node
NAME         STATUS   ROLES                  AGE   VERSION
k8s-master   Ready    control-plane,master   17d   v1.20.1
k8s-node-1   Ready    <none>                 17d   v1.20.1
k8s-node-2   Ready    <none>                 13d   v1.20.1

❯ kubectl get pod -o wide
No resources found in default namespace.

Pod は特に動かしていませんでした。

アップグレード(control-node)

drain

(後から気づいたので実施することを忘れました。。)

plan

アップグレードが可能なバージョンを確認します。

work@k8s-master:~$ sudo kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.20.1
[upgrade/versions] kubeadm version: v1.20.1
W0114 22:55:24.989943    8592 version.go:101] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable.txt": Get "https://storage.googleapis.com/kubernetes-release/release/stable.txt": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
W0114 22:55:24.990114    8592 version.go:102] falling back to the local client version: v1.20.1
[upgrade/versions] Latest stable version: v1.20.1
[upgrade/versions] Latest stable version: v1.20.1
[upgrade/versions] Latest version in the v1.20 series: v1.20.2
[upgrade/versions] Latest version in the v1.20 series: v1.20.2

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       AVAILABLE
kubelet     3 x v1.20.1   v1.20.2

Upgrade to the latest version in the v1.20 series:

COMPONENT                 CURRENT    AVAILABLE
kube-apiserver            v1.20.1    v1.20.2
kube-controller-manager   v1.20.1    v1.20.2
kube-scheduler            v1.20.1    v1.20.2
kube-proxy                v1.20.1    v1.20.2
CoreDNS                   1.7.0      1.7.0
etcd                      3.4.13-0   3.4.13-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.20.2

Note: Before you can perform this upgrade, you have to update kubeadm to v1.20.2.

_____________________________________________________________________


The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.

API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no
kubelet.config.k8s.io     v1beta1           v1beta1             no
_____________________________________________________________________

v1.20.1からv1.20.2にアップグレードできる
kubeadm upgrade apply v1.20.2でアップグレードできる
アップグレードする前にkubeadmを同バージョンにアップグレードする

とあるのでその通り進めます。

update kubeadm

kubeadmをアップグレードします。kubeadmはOSのパッケージマネージャーでインストールしているので、同じくパッケージマネージャー経由でアップグレードします。

work@k8s-master:~$ sudo apt install kubeadm
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  kubeadm
1 upgraded, 0 newly installed, 0 to remove and 19 not upgraded.
Need to get 6,489 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main armhf kubeadm armhf 1.20.2-00 [6,489 kB]
Fetched 6,489 kB in 13s (495 kB/s)
Reading changelogs... Done
(Reading database ... 41927 files and directories currently installed.)
Preparing to unpack .../kubeadm_1.20.2-00_armhf.deb ...
Unpacking kubeadm (1.20.2-00) over (1.20.1-00) ...
Setting up kubeadm (1.20.2-00) ...

完了しました。バージョンを確認します。

work@k8s-master:~$ sudo kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:25:59Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/arm"}

v1.20.2になっていました。

upgrade kubernetes

kubernetes自身をアップグレードしていきます。途中実行確認があるのでyを押下します。

work@k8s-master:~$ sudo kubeadm upgrade apply v1.20.2
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.20.2"
[upgrade/versions] Cluster version: v1.20.1
[upgrade/versions] kubeadm version: v1.20.2
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
[upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection
[upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.20.2"...
Static pod: kube-apiserver-k8s-master hash: a3be84725e7f93b76061ccf7862e9165
Static pod: kube-controller-manager-k8s-master hash: cf6a9238a74837b3afd7c649b346745b
Static pod: kube-scheduler-k8s-master hash: 9be8cb4627e7e5ad4c3f8acabd4b49b3
[upgrade/etcd] Upgrading to TLS for etcd
Static pod: etcd-k8s-master hash: 8dfbba74307f47100b61bba3c071ed92
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Current and new manifests of etcd are equal, skipping upgrade
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests441184868"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2021-01-14-23-09-23/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-apiserver-k8s-master hash: a3be84725e7f93b76061ccf7862e9165
Static pod: kube-apiserver-k8s-master hash: a3be84725e7f93b76061ccf7862e9165
：
Static pod: kube-apiserver-k8s-master hash: a3be84725e7f93b76061ccf7862e9165
Static pod: kube-apiserver-k8s-master hash: b064939bdf860fc00d7e77fc075e16c8
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2021-01-14-23-09-23/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-controller-manager-k8s-master hash: cf6a9238a74837b3afd7c649b346745b
Static pod: kube-controller-manager-k8s-master hash: 3456cf17d1057cfffaa60b9ccb6eaf2d
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2021-01-14-23-09-23/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-scheduler-k8s-master hash: 9be8cb4627e7e5ad4c3f8acabd4b49b3
Static pod: kube-scheduler-k8s-master hash: 69cd289b4ed80ced4f95a59ff60fa102
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upgrade/postupgrade] Applying label node-role.kubernetes.io/control-plane='' to Nodes with label node-role.kubernetes.io/master='' (deprecated)
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.20" in namespace kube-system with the configuration for the kubelets in the cluster
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.20.2". Enjoy!

[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.

成功したっぽい応答です。kubernetes自身のアップグレードと言いつつ何をやっているのだろうと思いましたが上のログを見ると、

kube-apiserver, kube-controller-manager, kube-schedulerについて新しいManifestをもとにPodを立ち上げている
- etcd等も同様のことをやっているようだが現行と差分が無いためアップグレードがスキップされた
kubelet-config-1.20を作成している
- アップグレードがトリガーかどうかよくわからない

のように見えます。そして最後にkubeletのアップグレードをお願いされています。

upgrade kubelet/kubectl

kubeletとkubectlをアップグレードします。

work@master:~ $ sudo apt install kubectl kubelet
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  kubeadm kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 16 not upgraded.
Need to get 22.7 MB of archives.
After this operation, 24.6 kB of additional disk space will be used.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main armhf kubelet armhf 1.20.2-00 [16.0 MB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main armhf kubectl armhf 1.20.2-00 [6704 kB]
Fetched 22.7 MB in 10s (2118 kB/s)
apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
Reading changelogs... Done
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = (unset),
        LC_ALL = (unset),
        LANG = "en_US.utf8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
(Reading database ... 41927 files and directories currently installed.)
Preparing to unpack .../kubelet_1.20.2-00_armhf.deb ...
Unpacking kubelet (1.20.2-00) over (1.20.1-00) ...
Preparing to unpack .../kubectl_1.20.2-00_armhf.deb ...
Unpacking kubectl (1.20.2-00) over (1.20.1-00) ...
Setting up kubectl (1.20.2-00) ...
Setting up kubelet (1.20.2-00) ...
$ sudo systemctl restart kubelet
$

各nodeのバージョンを確認します。

❯ kubectl get nodes
NAME         STATUS   ROLES                  AGE   VERSION
k8s-master   Ready    control-plane,master   17d   v1.20.2
k8s-node-1   Ready    <none>                 17d   v1.20.1
k8s-node-2   Ready    <none>                 13d   v1.20.1

k8s-masterのみがv1.20.2となりました。

アップグレード(worker-node)

続いてworker-nodeのアップグレードを行います。

drain

この時にふと「そういえばドレインとかしてクラスタから外すのが必要なのでは？？」と考えだしました。調べてみると皆様当然のように外していらっしゃいましたので「ですよね～」と言いながら今頃実施します。

work@k8s-master:~$ kubectl drain k8s-node-1 --ignore-daemonsets
node/k8s-node-1 cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-flannel-ds-bm57x, kube-system/kube-proxy-2pmgv
node/k8s-node-1 drained
work@k8s-master:~$ kubectl get nodes
NAME         STATUS                     ROLES                  AGE   VERSION
k8s-master   Ready                      control-plane,master   17d   v1.20.2
k8s-node-1   Ready,SchedulingDisabled   <none>                 17d   v1.20.1
k8s-node-2   Ready                      <none>                 13d   v1.20.1

upgrade kubeadm

control-nodeと同様にsudo apt install kubeadmでアップグレードします(ログ略)。こちらはnodeにログインして実施することを忘れずに。

upgrade kubernetes

control-nodeでkubeadm upgrade nodeを実行します。

work@k8s-master:~$ sudo kubeadm upgrade node
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[upgrade] Upgrading your Static Pod-hosted control plane instance to version "v1.20.2"...
Static pod: kube-apiserver-k8s-master hash: b064939bdf860fc00d7e77fc075e16c8
Static pod: kube-controller-manager-k8s-master hash: 3456cf17d1057cfffaa60b9ccb6eaf2d
Static pod: kube-scheduler-k8s-master hash: 69cd289b4ed80ced4f95a59ff60fa102
[upgrade/etcd] Upgrading to TLS for etcd
Static pod: etcd-k8s-master hash: 8dfbba74307f47100b61bba3c071ed92
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Current and new manifests of etcd are equal, skipping upgrade
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests791970212"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Current and new manifests of kube-apiserver are equal, skipping upgrade
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Current and new manifests of kube-controller-manager are equal, skipping upgrade
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Current and new manifests of kube-scheduler are equal, skipping upgrade
[upgrade] The control plane instance for this node was successfully updated!
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.

control-nodeの時と異なり、kube-apiserver等は既にアップグレードされているためスキップされているようです。では何のために実施するんだ？と思ったのですが下に記載があるnodeのconfigurationをアップグレードしていそうなのでそのように捉えています。

upgrade kubelet/kubectl

control-nodeと同様にsudo apt install kubelet kubectlでアップグレードします(ログ略)。こちらもnodeにログインして実施することを忘れずに。

uncordon

クラスタに再び組み込みます。

work@k8s-master:~$ kubectl uncordon k8s-node-1
node/k8s-node-1 uncordoned

確認

worker-nodeの台数分のアップグレード作業を完了後、確認します。

❯ kubectl get nodes
NAME         STATUS   ROLES                  AGE   VERSION
k8s-master   Ready    control-plane,master   17d   v1.20.2
k8s-node-1   Ready    <none>                 17d   v1.20.2
k8s-node-2   Ready    <none>                 13d   v1.20.2

Version が揃うと気持ちいい。

❯ kubectl get pod -n kube-system -o wide
NAME                                 READY   STATUS    RESTARTS   AGE     IP             NODE         NOMINATED NODE   READINESS GATES
coredns-74ff55c5b-n8qvl              1/1     Running   0          17d     10.244.0.2     k8s-master   <none>           <none>
coredns-74ff55c5b-rfvj5              1/1     Running   0          17d     10.244.0.3     k8s-master   <none>           <none>
etcd-k8s-master                      1/1     Running   0          17d     192.168.13.2   k8s-master   <none>           <none>
kube-apiserver-k8s-master            1/1     Running   0          2d23h   192.168.13.2   k8s-master   <none>           <none>
kube-controller-manager-k8s-master   1/1     Running   0          2d23h   192.168.13.2   k8s-master   <none>           <none>
kube-flannel-ds-2pzhn                1/1     Running   3          17d     192.168.13.2   k8s-master   <none>           <none>
kube-flannel-ds-7g89m                1/1     Running   0          13d     192.168.13.6   k8s-node-2   <none>           <none>
kube-flannel-ds-bm57x                1/1     Running   4          17d     192.168.13.5   k8s-node-1   <none>           <none>
kube-proxy-2b5zz                     1/1     Running   0          2d23h   192.168.13.2   k8s-master   <none>           <none>
kube-proxy-2g9r8                     1/1     Running   0          2d23h   192.168.13.6   k8s-node-2   <none>           <none>
kube-proxy-2pmgv                     1/1     Running   0          2d23h   192.168.13.5   k8s-node-1   <none>           <none>
kube-scheduler-k8s-master            1/1     Running   0          2d23h   192.168.13.2   k8s-master   <none>           <none>

そういえば kube-apiserver は？

以前のエントリでkube-apiserverのCPU使用率が高騰していたことを確認していまして、それの対処反映を含めたアップグレード作業だったわけですが、中々温厚な使用率に戻りました。よかった。

top - 21:44:18 up 17 days, 23:18,  1 user,  load average: 0.71, 0.68, 0.71
Tasks: 148 total,   1 running, 147 sleeping,   0 stopped,   0 zombie
%Cpu(s):  8.3 us,  3.4 sy,  0.0 ni, 88.1 id,  0.1 wa,  0.0 hi,  0.1 si,  0.0 st
MiB Mem :   7875.9 total,   4244.4 free,    588.8 used,   3042.7 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   6738.6 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
14979 root      20   0  893920 277668  60152 S  15.5   3.4 890:43.67 kube-apiserver <--- これ
20254 root      20   0 1035488  90488  59284 S  15.2   1.1 433:27.93 kubelet
  555 root      20   0 1027992  86020  33608 S   6.3   1.1 810:42.36 dockerd
15804 root      20   0  885328  85384  53212 S   2.3   1.1 126:54.82 kube-controller

感想

公式にコマンドが羅列されているので助かった。今回はPod(Service)を動作させず確認したので、次回は動作させてサービスの継続性とかも確認したい。

コマンドを打つこと自体のリスクは Ansible あたりが正解になるのかしら。デプロイのplaybookはそれっぽいものを作成したのでアップグレードもそうしてしまいたい。