k8s入门教程三:多主集群部署

发表于 | 更新于 | 分类于 | 评论数： | 阅读次数：
本文字数： 16k | 阅读时长 ≈ 15 分钟

高可用架构

kubernetes高可用集群通常由3或5个节点组成高可用集群，需要保障各个节点的高可用性

  • etcd 内置集群机制，保障数据持久存储
  • kube-apiserver 无状态api服务，有负载均衡调度器做负载分发，如haproxy或nginx
  • kube-scheduler 内置选举机制，保障调度器高可用，确保同个时刻一个leader节点工作，其他处于阻塞，防止脑裂
  • kube-controller-manager 内置的选举机制保障控制器高可用，机制和kube-scheduler一致。

如下图所示：

从上图可以得出：有多个worker node，那计算节点就是高可用。而master节点上面，由于etcd、scheduler、controller-manager由k8s系统自身实现了高可用。所以需要高用可的是apiserver这个组件。以下是以keepalived+haproxy实现apiserver的高可用。

环境准备

由于条件有限，拿三台机器实现master的高可用。环境如下：

名称 IP 系统版本 角色 安装软件
k8s01 192.168.1.90 centos 7.7 master kubeadm、kubelet、kubectl、docker、haproxy、keepalived
k8s02 192.168.1.91 centos 7.7 master kubeadm、kubelet、kubectl、docker、haproxy、keepalived
k8s03 192.168.1.92 centos 7.7 master kubeadm、kubelet、kubectl、docker
VIP 192.168.1.99

三台都准备安装kubelet、kubeadm、kubectl、docker，前一篇文章里面的 准备工作 都需要运行。

HA高可用

安装软件

在k8s01、k8s02直接使用yum安装 keepalived以及haproxy这2个软件。

1
yum install -y keepalived haproxy

keepalived配置

/etc/keepalived/keepalived.conf的配置如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
! Configuration File for keepalived

global_defs {
    router_id lvs_vip
}

vrrp_instance VI_1 {
    state MASTER	#k8s02备用节点配置为：BACKUP
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.1.99 dev eth0 label eth0:0
    }
}

注意router_id、virtual_router_id以及state这几个配置项。virtual_router_id必须不一样，值越高，其优先级越高。

haproxy配置

/etc/haproxy/haproxy.cfg配置如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
global
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    stats socket /var/lib/haproxy/stats

defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

listen k8s-apiserver
    bind *:6444
    mode tcp
    balance roundrobin
    timeout server 900s
    timeout connect 15s

    server  app1 192.168.1.90:6443 check  port 6443 inter 5000 fall 5
    server  app2 192.168.1.91:6443 check  port 6443 inter 5000 fall 5
    server  app3 192.168.1.92:6443 check  port 6443 inter 5000 fall 5

frontend stats-front
    bind *:8081
    mode http
    default_backend stats-back

backend stats-back
    mode http
    balance roundrobin
    stats hide-version
    stats uri /haproxy/stats
    stats auth admin:admin

haproxy配置主要估了2个事情：

  • 监听6444端口，对应后端为三台master节点的6443端口
  • 开启 /haproxy/stats 状态查看后端服务是否有正常

完成之后，启动服务。观察VIP是否在master节点上，访问 http://192.168.1.99/haproxy/stats；如果正常，将keepalived停止，再访问URL，如果正常就继续下一步操作。

1
2
systemctl enable keepalived && systemctl start keepalived
systemctl enable haproxy && systemctl start haproxy

初始化master节点

  1. 使用 kubeadm config print init-defaults 2>/dev/null >kubeadm-init.yaml 生成配置，再修改相应的配置如下。主要是controlPlaneEndpoint配置 192.168.1.99:6444，这个VIP加haproxy监听的端口。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.1.90
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: k8s01
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: "192.168.1.99:6444"
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.17.3
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/16
  podSubnet: 10.100.0.0/16
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
featureGates:
  SupportIPVSProxyMode: true
mode: ipvs
  1. 下载镜像，每一台都需要操作：
1
2
3
4
5
6
7
8
[root@k8s-1 ~]# kubeadm config images pull --config kubeadm-init.yaml
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.17.3
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.17.3
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.17.3
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.17.3
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.3-0
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.6.5
  1. 在k8s01上面做初始化
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
[root@k8s01 ~]# kubeadm init --config kubeadm-init.yaml
W0220 08:53:24.018854   15694 validation.go:28] Cannot validate kube-proxy config - no validator is available
W0220 08:53:24.019041   15694 validation.go:28] Cannot validate kubelet config - no validator is available
[init] Using Kubernetes version: v1.17.3
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.1.90 192.168.1.99]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s01 localhost] and IPs [192.168.1.90 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s01 localhost] and IPs [192.168.1.90 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "admin.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0220 08:53:37.967621   15694 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0220 08:53:38.025082   15694 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 31.011867 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.17" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s01 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

  kubeadm join 192.168.1.99:6444 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:fe1e721f5627faeb20bc8ce85e806d1410ea67130bf7975ad9924835e3ac0e43 \
    --control-plane

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.1.99:6444 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:fe1e721f5627faeb20bc8ce85e806d1410ea67130bf7975ad9924835e3ac0e43
[root@k8s-1 ~]# mkdir -p $HOME/.kube
[root@k8s-1 ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-1 ~]# chown $(id -u):$(id -g) $HOME/.kube/config

上面有2个 kubeadm join ，之前在单主模式下，只会出现worker node的加入命令。

  1. 需要上传证书到其他的机器上，脚本参考了官方文档：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
[root@k8s01 ~]# cat upload_cert.sh
#!/bin/bash

USER=root # customizable
CONTROL_PLANE_IPS="k8s02 k8s03"
for host in ${CONTROL_PLANE_IPS}; do
    ssh "${USER}"@$host "mkdir -p /etc/kubernetes/pki/etcd"
    scp /etc/kubernetes/pki/ca.crt "${USER}"@$host:/etc/kubernetes/pki/
    scp /etc/kubernetes/pki/ca.key "${USER}"@$host:/etc/kubernetes/pki/
    scp /etc/kubernetes/pki/sa.key "${USER}"@$host:/etc/kubernetes/pki/
    scp /etc/kubernetes/pki/sa.pub "${USER}"@$host:/etc/kubernetes/pki/
    scp /etc/kubernetes/pki/front-proxy-ca.crt "${USER}"@$host:/etc/kubernetes/pki/
    scp /etc/kubernetes/pki/front-proxy-ca.key "${USER}"@$host:/etc/kubernetes/pki/
    scp /etc/kubernetes/pki/etcd/ca.crt "${USER}"@$host:/etc/kubernetes/pki/etcd/
    # Quote this line if you are using external etcd
    scp /etc/kubernetes/pki/etcd/ca.key "${USER}"@$host:/etc/kubernetes/pki/etcd/
done
  1. 必须保证证书已经上传到了节点，然后在k8s02 k8s03进行join的操作。
1
2
3
kubeadm join 192.168.1.99:6444 --token abcdef.0123456789abcdef \
  --discovery-token-ca-cert-hash sha256:fe1e721f5627faeb20bc8ce85e806d1410ea67130bf7975ad9924835e3ac0e43 \
  --control-plane
  1. 在k8s01上面查看状态：
1
2
3
4
5
6
7
8
9
10
[root@k8s01 ~]# kubectl get nodes
NAME    STATUS     ROLES    AGE     VERSION
k8s01   NotReady   master   31m     v1.17.3
k8s02   NotReady   master   12m     v1.17.3
k8s03   NotReady   master   8m14s   v1.17.3
[root@k8s01 ~]# kubectl get componentstatuses
NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-0               Healthy   {"health":"true"}
  1. 在k8s01上面安装网络组件calico：
1
2
3
4
5
6
#https://docs.projectcalico.org/getting-started/kubernetes/quickstart
wget https://docs.projectcalico.org/manifests/calico.yaml
#将网段替换为podSubnet所设置的网段，calico默认值为192.168.0.0.16
sed -i 's#192.168.0.0/16#10.100.0.0/16#' calico.yaml
kubectl apply -f calico.yaml
kubectl taint nodes --all node-role.kubernetes.io/master-
  1. 查看状态
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
[root@k8s01 ~]# k get nodes
NAME    STATUS   ROLES    AGE   VERSION
k8s01   Ready    master   43m   v1.17.3
k8s02   Ready    master   25m   v1.17.3
k8s03   Ready    master   20m   v1.17.3
[root@k8s01 ~]# kubectl get pod -n kube-system
NAME                                      READY   STATUS    RESTARTS   AGE
calico-kube-controllers-77c4b7448-k5n72   1/1     Running   0          5m6s
calico-node-25gww                         1/1     Running   0          5m7s
calico-node-s8rmm                         1/1     Running   0          5m7s
calico-node-t5pw9                         1/1     Running   0          5m7s
coredns-7f9c544f75-6w798                  1/1     Running   0          46m
coredns-7f9c544f75-b745b                  1/1     Running   0          46m
etcd-k8s01                                1/1     Running   0          46m
etcd-k8s02                                1/1     Running   0          28m
etcd-k8s03                                1/1     Running   0          24m
kube-apiserver-k8s01                      1/1     Running   0          46m
kube-apiserver-k8s02                      1/1     Running   0          28m
kube-apiserver-k8s03                      1/1     Running   0          24m
kube-controller-manager-k8s01             1/1     Running   1          46m
kube-controller-manager-k8s02             1/1     Running   0          28m
kube-controller-manager-k8s03             1/1     Running   0          24m
kube-proxy-76njt                          1/1     Running   0          24m
kube-proxy-nl2kl                          1/1     Running   0          28m
kube-proxy-sz6mw                          1/1     Running   0          46m
kube-scheduler-k8s01                      1/1     Running   1          46m
kube-scheduler-k8s02                      1/1     Running   0          28m
kube-scheduler-k8s03                      1/1     Running   0          24m
[root@k8s01 ~]#
[root@k8s01 ~]#
[root@k8s01 ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.96.0.1:443 rr
  -> 192.168.1.90:6443            Masq    1      1          0
  -> 192.168.1.91:6443            Masq    1      0          0
  -> 192.168.1.92:6443            Masq    1      1          0
TCP  10.96.0.10:53 rr
  -> 10.100.235.129:53            Masq    1      0          0
  -> 10.100.235.131:53            Masq    1      0          0
TCP  10.96.0.10:9153 rr
  -> 10.100.235.129:9153          Masq    1      0          0
  -> 10.100.235.131:9153          Masq    1      0          0
UDP  10.96.0.10:53 rr
  -> 10.100.235.129:53            Masq    1      0          0
  -> 10.100.235.131:53            Masq    1      0          0

看到三个节点都是master时，说明部署已经成功了。

参考资料

相关文章
0%