一、kubeasz 部署高可用 k8s 集群

官方网址：https://github.com/easzlab/kubeasz

kubeasz 致力于提供快速部署高可用k8s集群的工具, 同时也努力成为k8s实践、使用的参考书；基于二进制方式部署和利用 ansible-playbook 实现自动化；既提供一键安装脚本, 也可以根据安装指南分步执行安装各个组件。

image-20230810231009451

1.1 k8s 集群信息

主机类型 服务器IP 主机名 VIP
K8S-master1 10.243.20.230 k8s-master1-230 10.243.20.250
K8S-Master2 10.243.20.231 k8s-master2-231 10.243.20.250
K8S-Master3 10.243.20.232 k8s-master3-232 10.243.20.250
Harbor1 10.243.20.233 k8s-harbor1-233
Harbor2 10.243.20.234 k8s-harbor2-234
etcd1 10.243.20.235 k8s-etcd1-235
etcd2 10.243.20.236 k8s-etcd2-236
etcd3 10.243.20.237 k8s-etcd3-237
Haproxy1+keepalived 10.243.20.238 k8s-ha1-238
Haproxy2+keepalived 10.243.20.239 k8s-ha2-deploy-239 K8S 部署节点
Node1 10.243.20.240 k8s-node1-240
Node2 10.243.20.241 k8s-node2-241
Node3 10.243.20.242 k8s-node3-242

1.2 k8s 集群环境准备

1.2.1 软件清单

1
2
3
4
5
6
7
8
9
10
OS version: Ubuntu 22.04.2 LTS
keepalived version: Keepalived v2.2.4 (08/21,2021)
HAProxy version 2.4.22
k8s version: 1.27.2
Docker version 20.10.24, build 297e128
containerd github.com/containerd/containerd v1.6.20 2806fc1057397dbaeefbea0e4e17bddfbd388f38
nerdctl version 1.4.0
harbor version: v2.8.2
CoreDNS version: v1.10.1
calico version: v3.26.1

1.2.2 相关参数初始化

系统内核优化

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
cat >> /etc/security/limits.conf << EOF 
*             soft    core            unlimited
*             hard    core            unlimited
*       soft    nproc           1000000
*             hard    nproc           1000000
*             soft    nofile          1000000
*             hard    nofile          1000000
*             soft    memlock         32000
*             hard    memlock         32000
*             soft    msgqueue        8192000
*             hard    msgqueue        8192000
EOF


cat >> /etc/sysctl.conf << EOF
net.ipv4.ip_forward=1
vm.max_map_count=262144
kernel.pid_max=4194303
fs.file-max=1000000
net.ipv4.tcp_max_tw_buckets=6000
net.netfilter.nf_conntrack_max=2097152

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness=0
EOF

sudo modprobe nf_conntrack
sudo modprobe br_netfilter

sysctl -p

实验之前确保各节点时区一致、时间同步。

1
2
3
4
# 确保时区一致
timedatectl set-timezone Asia/Shanghai
# 每隔5分钟同步一次时间
*/5 * * * * /usr/sbin/ntpdate time1.aliyun.com &> /dev/null && hwclock -w &> /dev/null

1.3 负载均衡配置

1.3.1 部署 keepalived

（1）第一台高可用节点 HA1 部署 keepalived

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
root@k8s-ha1-238:~# apt -y install keepalived
root@k8s-ha1-238:/etc/default# vim /etc/keepalived/keepalived.conf
global_defs {
   router_id LVS_DEVEL1
   }
                                                                                                                                                                                          
vrrp_script check_haproxy {
  script "/etc/keepalived/check_haproxy.sh"
  interval 3
}


vrrp_instance VI_1 {
    state BACKUP
    interface ens160
    virtual_router_id 250
    priority 100
    advert_int 1
    nopreempt
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.243.20.250 label ens160:1        # 定义 VIP
    }
    unicast_src_ip 10.243.20.238
    unicast_peer {
      10.243.20.239
    }
    track_script {
      check_haproxy
    }
}

root@k8s-ha1-238:/etc/default# systemctl restart keepalived.service

第二台高可用节点 HA2 部署 keepalived 同理，只需修改部分参数

global_defs {
router_id LVS_DEVEL2
}

​ priority 80

​ unicast_src_ip 10.243.20.239
​ unicast_peer {
​ 10.243.20.238
​ }

检查HAproxy脚本

root@k8s-ha1-238:/etc/keepalived# cat check_haproxy.sh
#!/bin/bash

if ! /usr/bin/killall -0 haproxy; then
systemctl restart haproxy.service &> /dev/null
if ! [ $? -eq 0 ]; then
systemctl stop keepalived.service
fi
fi

1.3.2 部署 HAproxy

（1）在 HA1 上部署 HAproxy 服务，HA2 配置相同

1
2
3
4
5
6
7
8
9
root@k8s-ha1-238:~# apt -y install haproxy
root@k8s-ha1-238:~# vim /etc/haproxy/haproxy.cfg
listen k8s-api-6443
  bind 10.243.20.250:6443            # 监听k8s-Master的6443端口
  mode tcp
  server 10.243.20.230 10.243.20.230:6443 check inter 3s fall 3 rise 3
  server 10.243.20.231 10.243.20.231:6443 check inter 3s fall 3 rise 3
  server 10.243.20.232 10.243.20.232:6443 check inter 3s fall 3 rise 3
root@k8s-ha1-238:~# systemctl restart haproxy.service

为了保证后续 HAproxy 节点可以正常使用需要开启 “net.ipv4.ip_nonlocal_bind=1”

echo “net.ipv4.ip_nonlocal_bind=1” >> /etc/sysctl.conf
sysctl -p

1.4 部署 Harbor

1.4.1 部署基于 https 的 Harbor 服务

（1）在 Harbor1 主机上部署 Harbor 服务，Harbor2 主机同理

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# 准备 Harbor 安装包
root@k8s-harbor1-233:~# ls
harbor-offline-installer-v2.8.2.tgz

root@k8s-harbor1-233:~# mkdir /apps
root@k8s-harbor1-233:~# tar xvf harbor-offline-installer-v2.8.2.tgz -C /apps/
root@k8s-harbor1-233:~# cd /apps/harbor/
root@k8s-harbor1-233:/apps/harbor# cp harbor.yml.tmpl harbor.yml

# 编辑配置文件，修改相关选项，不开启harbor的https，后面使用HAproxy做代理实现https的harbor
hostname: 10.243.20.233                         # 修改主机名为本机IP地址
#https:
  # https port for harbor, default is 443
#  port: 443
  # The path of cert and key files for nginx
#  certificate: /apps/harbor/certs/harbor.wuhaolam.top.pem          
#  private_key: /apps/harbor/certs/harbor.wuhaolam.top.key
harbor_admin_password: 12345                                          # admin 登录密码
data_volume: /data/harbor/                                            # 数据卷存放目录

# 开始安装，--with-trivy 是开启安全扫描
root@k8s-harbor1-233:/apps/harbor# ./install.sh --with-trivy
image-20231231101646039

（2）配置 Harbor 服务开机自启动

1
2
3
4
root@k8s-harbor1-233:~# cat /etc/rc.local
#!/bin/bash
cd /apps/harbor && /usr/bin/docker-compose up -d
root@k8s-harbor1-233:~# chmod +x /etc/rc.local

Docker22.04 安装见此文章
https://wuhaolam.top/archives/99c33715.html

harobor 离线安装包下载地址
https://github.com/goharbor/harbor/releases/download/v2.8.2/harbor-offline-installer-v2.8.2.tgz

后续更改harbor.yml的配置文件，使用如下命令重启服务（在harbor安装目录）

  • docker-compose down
  • ./prepare
  • docker-compose up -d

（3）配置 HAproxy https 反向代理至 Harbor 实现高可用

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# 准备证书文件
## HAproxy是将私钥文件和服务器证书合并; 而 nginx 是将服务器证书和CA证书合并
root@k8s-ha1-238:~# mkdir /etc/haproxy/certs
root@k8s-ha1-238:/etc/haproxy/certs# ls
haproxy.pem  www.wuhaolam.top.key  www.wuhaolam.top.pem

# HA1 和 HA2 做相同配置
root@k8s-ha1-238:~# vim /etc/haproxy/haproxy.cfg
listen harbor-api-443
  bind 10.243.20.250:80
  bind 10.243.20.250:443 ssl crt /etc/haproxy/certs/haproxy.pem
  redirect scheme https if !{ ssl_fc }
  mode http
  balance source
  option forwardfor
  server 10.243.20.233 10.243.20.233:80 check inter 3s fall 3 rise 3
  server 10.243.20.234 10.243.20.234:80 check inter 3s fall 3 rise 3
  
root@k8s-ha1-238:~# systemctl restart haproxy.service

使用NGINX做反向代理负载均衡简要配置

==在http语句块中配置==

client_max_body_size 100m; # 允许客户端请求主体的最大大小，可以根据需要调整

upstream harbor_servers {
hash $remote_addr consistent;
server 192.168.119.112:80 weight=1 fail_timeout=5s max_fails=3;
server 192.168.119.113:80 weight=1 fail_timeout=5s max_fails=3;
}

server {
listen 80;
listen 443 ssl;
ssl_certificate /apps/nginx/certs/harbor.wuhaolam.top.pem;
ssl_certificate_key /apps/nginx/certs/harbor.wuhaolam.top.key;
ssl_session_cache shared:sslcache:20m;
ssl_session_timeout 10m;
server_name harbor.wuhaolam.top;
add_header Strict-Transport-Security “max-age=31536000; includeSubDomains” always;

location / {
if ( $scheme = http ) {
rewrite / https://harbor.wuhaolam.top permanent;
}
proxy_pass http://harbor_servers;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 300s;
}
}

1.4.2 配置 Harbor 之间的镜像复制

:warning:==此节中harbor的镜像复制有些使用了域名，有误；如果没有配置域名则需要替换成IP地址==

（1）在浏览器输入 Harbor 主机的IP地址，然后输入用户名和密码登录 Harbor UI

image-20231231160954714

image-20231231161056065

（2）在 Harbor1 节点中，仓库管理 –> 新建目标；目标创建完成后，点击确定即可

image-20231231161214877

image-20231231170156895

（3）点击 系统管理 –> 复制管理 –> 新建规则；

image-20231231162641649

（4）手动复制验证规则

Harbor1 节点

image-20231231163215994

Harbor2 节点

image-20231231163325779

（5）在 Harbor2 节点创建等同的复制规则

image-20231231181158998

image-20231231181223789

（6）推送镜像到 Harbor2 节点，测试是否可以自动推送镜像至 Harbor1节点

1
2
3
4
root@k8s-harbor2-234:/apps/harbor# docker login harbor2.wuhaolam.top
root@k8s-harbor2-234:/apps/harbor# docker pull nginx
root@k8s-harbor2-234:/apps/harbor# docker tag nginx:latest harbor2.wuhaolam.top/myserver/nginx:v1
root@k8s-harbor2-234:/apps/harbor# docker push harbor2.wuhaolam.top/myserver/nginx:v1

image-20231231182134334

image-20231231182335950

1.5 kubeasz 部署高可用 kubernets

1.5.1 在部署节点上配置Ansible免密登录至master、node、etcd节点

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
root@k8s-ha2-deploy-239:~# apt -y install ansible git

# 配置免密登录
root@k8s-ha2-deploy-239:~# cat ssh-key.sh 
#!/bin/bash
  
IPLIST='
10.243.20.230
10.243.20.231
10.243.20.232
10.243.20.235
10.243.20.236
10.243.20.237
10.243.20.240
10.243.20.241
10.243.20.242
'
PASSWORD='wuhaolam'

if [ ! -e /root/.ssh/id_rsa ];then
  echo -e "\E[1;32m开始生成密钥对...\E[0m"
  ssh-keygen -P "" -f /root/.ssh/id_rsa &> /dev/null
else
  echo -e "\E[1;34m密钥对已经存在\E[0m"
fi

if ! dpkg -L sshpass &> /dev/null;then
  apt -y install sshpass &> /dev/null || { echo -e '\E[1;31m"error: sshpass packet install false!"\E[0m'; exit; }
fi

for IP in $IPLIST; do
  sshpass -p $PASSWORD ssh-copy-id -o StrictHostKeyChecking=no $IP &> /dev/null
  echo $IP key_authentication already done.
done

echo -e "\E[1;32msuccessful\E[0m"
root@k8s-ha2-deploy-239:~# bash ssh-key.sh
image-20240101153034767

1.5.2 在部署节点下载 kubeasz 项目及其组件

（1）下载项目源码

1
2
3
4
root@k8s-ha2-deploy-239:~# wget https://github.com/easzlab/kubeasz/releases/download/3.6.1/ezdown
root@k8s-ha2-deploy-239:~# ls ezdown   # 此文件是一个k8s环境集群的脚本
ezdown
root@k8s-ha2-deploy-239:~# chmod +x ezdown

（2）在部署节点安装Docker，不使用 ezdown 脚本安装，自己安装Docker

1
2
3
4
5
Docker22.04 安装见此文章
https://wuhaolam.top/archives/99c33715.html

root@k8s-ha2-deploy-239:~# docker --version
Docker version 24.0.2, build cb74dfc

（3）开始下载kubeasz项目及其组件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
root@k8s-ha2-deploy-239:~# ./ezdown -D
# 下载完成
root@k8s-ha2-deploy-239:~# docker images
REPOSITORY                                           TAG       IMAGE ID       CREATED         SIZE
easzlab/kubeasz                                      3.6.1     0ae1e2a7c7f0   7 months ago    157MB
easzlab/kubeasz-k8s-bin                              v1.27.2   a9d4ca76c91b   7 months ago    1.12GB
calico/kube-controllers                              v3.24.6   baf4466ddf40   7 months ago    77.5MB
easzlab.io.local:5000/calico/kube-controllers        v3.24.6   baf4466ddf40   7 months ago    77.5MB
easzlab.io.local:5000/calico/cni                     v3.24.6   ca9fea5e07cb   7 months ago    212MB
calico/cni                                           v3.24.6   ca9fea5e07cb   7 months ago    212MB
calico/node                                          v3.24.6   3953a481aa9d   7 months ago    245MB
easzlab.io.local:5000/calico/node                    v3.24.6   3953a481aa9d   7 months ago    245MB
easzlab/kubeasz-ext-bin                              1.7.1     5c1895de99b2   9 months ago    606MB
easzlab/metrics-server                               v0.6.3    1da5af8117da   9 months ago    68.9MB
easzlab.io.local:5000/easzlab/metrics-server         v0.6.3    1da5af8117da   9 months ago    68.9MB
easzlab/k8s-dns-node-cache                           1.22.20   d1157efdd316   10 months ago   67.8MB
easzlab.io.local:5000/easzlab/k8s-dns-node-cache     1.22.20   d1157efdd316   10 months ago   67.8MB
easzlab/pause                                        3.9       78d53e70b442   14 months ago   744kB
easzlab.io.local:5000/easzlab/pause                  3.9       78d53e70b442   14 months ago   744kB
kubernetesui/dashboard                               v2.7.0    07655ddf2eeb   15 months ago   246MB
easzlab.io.local:5000/kubernetesui/dashboard         v2.7.0    07655ddf2eeb   15 months ago   246MB
kubernetesui/metrics-scraper                         v1.0.8    115053965e86   19 months ago   43.8MB
easzlab.io.local:5000/kubernetesui/metrics-scraper   v1.0.8    115053965e86   19 months ago   43.8MB
coredns/coredns                                      1.9.3     5185b96f0bec   19 months ago   48.8MB
easzlab.io.local:5000/coredns/coredns                1.9.3     5185b96f0bec   19 months ago   48.8MB
registry                                             2         b8604a3fe854   2 years ago     26.2MB
root@k8s-ha2-deploy-239:~# ls /etc/kubeasz/
ansible.cfg  bin  docs  down  example  ezctl  ezdown  manifests  pics  playbooks  README.md  roles  tools

1.5.3 生成并自定义 hosts 文件

1
2
3
4
5
6
7
root@k8s-ha2-deploy-239:~# cd /etc/kubeasz/ 
root@k8s-ha2-deploy-239:/etc/kubeasz# ./ezctl new k8s-cluster1   # 新建一个容器
2024-01-02 14:05:59 DEBUG generate custom cluster files in /etc/kubeasz/clusters/k8s-cluster1
2024-01-02 14:05:59 DEBUG set versions
2024-01-02 14:05:59 DEBUG cluster k8s-cluster1: files successfully created.
2024-01-02 14:05:59 INFO next steps 1: to config '/etc/kubeasz/clusters/k8s-cluster1/hosts'
2024-01-02 14:05:59 INFO next steps 2: to config '/etc/kubeasz/clusters/k8s-cluster1/config.yml'

1.5.3.1 编辑生成的 Ansible hosts 文件

指定 etcd 节点、Master节点、node节点、VIP、运行时、网络组件类型、Service IP 与 pod IP 范围等配置信息。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
root@k8s-ha2-deploy-239:/etc/kubeasz# vim clusters/k8s-cluster1/hosts
root@k8s-ha2-deploy-239:/etc/kubeasz# cat clusters/k8s-cluster1/hosts
# 'etcd' cluster should have odd member(s) (1,3,5,...)
[etcd]
10.243.20.235
10.243.20.236
10.243.20.237

# master node(s), set unique 'k8s_nodename' for each node
# CAUTION: 'k8s_nodename' must consist of lower case alphanumeric characters, '-' or '.',
# and must start and end with an alphanumeric character
[kube_master]
10.243.20.230 k8s_nodename='10.243.20.230'
10.243.20.231 k8s_nodename='10.243.20.231'
10.243.20.232 k8s_nodename='10.243.20.232'

# work node(s), set unique 'k8s_nodename' for each node
# CAUTION: 'k8s_nodename' must consist of lower case alphanumeric characters, '-' or '.',
# and must start and end with an alphanumeric character
[kube_node]
10.243.20.240 k8s_nodename='10.243.20.240'
10.243.20.241 k8s_nodename='10.243.20.241'
10.243.20.242 k8s_nodename='10.243.20.242'

# [optional] harbor server, a private docker registry
# 'NEW_INSTALL': 'true' to install a harbor server; 'false' to integrate with existed one
[harbor]
#192.168.1.8 NEW_INSTALL=false

# [optional] loadbalance for accessing k8s from outside
[ex_lb]
#192.168.1.6 LB_ROLE=backup EX_APISERVER_VIP=192.168.1.250 EX_APISERVER_PORT=8443
#192.168.1.7 LB_ROLE=master EX_APISERVER_VIP=192.168.1.250 EX_APISERVER_PORT=8443

# [optional] ntp server for the cluster
[chrony]
#192.168.1.1

[all:vars]
# --------- Main Variables ---------------
# Secure port for apiservers
SECURE_PORT="6443"

# Cluster container-runtime supported: docker, containerd
# if k8s version >= 1.24, docker is not supported
CONTAINER_RUNTIME="containerd"

# Network plugins supported: calico, flannel, kube-router, cilium, kube-ovn
CLUSTER_NETWORK="calico"

# Service proxy mode of kube-proxy: 'iptables' or 'ipvs'
PROXY_MODE="ipvs"

# K8S Service CIDR, not overlap with node(host) networking
SERVICE_CIDR="10.100.0.0/16"

# Cluster CIDR (Pod CIDR), not overlap with node(host) networking
CLUSTER_CIDR="10.200.0.0/16"

# NodePort Range
NODE_PORT_RANGE="30000-52767"

# Cluster DNS Domain
CLUSTER_DNS_DOMAIN="cluster.local"

# -------- Additional Variables (don't change the default value right now) ---
# Binaries Directory
bin_dir="/usr/local/bin"

# Deploy Directory (kubeasz workspace)
base_dir="/etc/kubeasz"

# Directory for a specific cluster
cluster_dir="{{ base_dir }}/clusters/k8s-cluster1"

# CA and other components cert/key Directory
ca_dir="/etc/kubernetes/ssl"

# Default 'k8s_nodename' is empty
k8s_nodename=''

# Default python interpreter
ansible_python_interpreter=/usr/bin/python3

1.5.3.2 编辑集群 config.yml 文件

1
root@k8s-ha2-deploy-239:/etc/kubeasz# vim clusters/k8s-cluster1/config.yml

image-20240102160403619

image-20240102160709935

1.5.4 部署 k8s 集群

1.5.4.1 环境初始化

1
root@k8s-ha2-deploy-239:/etc/kubeasz# ./ezctl setup k8s-cluster1 01

image-20240102170520512

![image-20240102170043579](D:\Documents\TyporaPicture\基于 kubeasz 部署高可用 kubernetes\image-20240102170043579.png)

命令使用帮助
root@k8s-ha2-deploy-239:/etc/kubeasz# ./ezctl –help

root@k8s-ha2-deploy-239:/etc/kubeasz# ./ezctl setup –help
Usage: ezctl setup
available steps:
01 prepare to prepare CA/certs & kubeconfig & other system settings
02 etcd to setup the etcd cluster
03 container-runtime to setup the container runtime(docker or containerd)
04 kube-master to setup the master nodes
05 kube-node to setup the worker nodes
06 network to setup the network plugin
07 cluster-addon to setup other useful plugins
90 all to run 01~07 all at once
10 ex-lb to install external loadbalance for accessing k8s from outside
11 harbor to install a new harbor server or to integrate with an existed one

examples: ./ezctl setup test-k8s 01 (or ./ezctl setup test-k8s prepare)
./ezctl setup test-k8s 02 (or ./ezctl setup test-k8s etcd)
./ezctl setup test-k8s all
./ezctl setup test-k8s 04 -t restart_master

1.5.4.2 部署 etcd 集群

1
root@k8s-ha2-deploy-239:/etc/kubeasz# ./ezctl setup k8s-cluster1 02

image-20240103110940287

1
2
3
4
5
6
# 在其中一个 etcd 服务器验证集群状态
root@k8s-etcd1-235:~# export NODE_IPS="10.243.20.235 10.243.20.236 10.243.20.237"
root@k8s-etcd1-235:~# for IP in ${NODE_IPS};do ETCDCTL_API=3 /usr/local/bin/etcdctl --endpoints=https://${IP}:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/kubernetes/ssl/etcd.pem --key=/etc/kubernetes/ssl/etcd-key.pem endpoint health;done
https://10.243.20.235:2379 is healthy: successfully committed proposal: took = 11.440413ms
https://10.243.20.236:2379 is healthy: successfully committed proposal: took = 11.130143ms
https://10.243.20.237:2379 is healthy: successfully committed proposal: took = 12.942469ms
1
2
3
# 查看成员列表
root@k8s-etcd1-235:~# export NODE_IPS="10.243.20.235 10.243.20.236 10.243.20.237"
root@k8s-etcd1-235:~# ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table member list --endpoints=https://10.243.20.235:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/kubernetes/ssl/etcd.pem --key=/etc/kubernetes/ssl/etcd-key.pem

image-20240621092208445

1
2
3
4
5
6
# 验证节点心跳状态
root@k8s-etcd1-235:~# export NODE_IPS="10.243.20.235 10.243.20.236 10.243.20.237"
root@k8s-etcd1-235:~# for ip in ${NODE_IPS}; do ETCDCTL_API=3 /usr/local/bin/etcdctl --endpoints=https://${ip}:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/kubernetes/ssl/etcd.pem --key=/etc/kubernetes/ssl/etcd-key.pem endpoint health; done
https://10.243.20.235:2379 is healthy: successfully committed proposal: took = 10.120964ms
https://10.243.20.236:2379 is healthy: successfully committed proposal: took = 11.224057ms
https://10.243.20.237:2379 is healthy: successfully committed proposal: took = 11.444497ms
1
2
3
# 查看 etcd 详细信息
root@k8s-etcd1-235:~# export NODE_IPS="10.243.20.235 10.243.20.236 10.243.20.237"
root@k8s-etcd1-235:~# for ip in ${NODE_IPS}; do ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table endpoint status --endpoints=https://${ip}:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/kubernetes/ssl/etcd.pem --key=/etc/kubernetes/ssl/etcd-key.pem; done
image-20240621092654971

etcd 服务安装完成后，会监听在 2379 和 2380 两个端口
2379 是 客户端通信使用
2380 是 etcd 集群之间通信使用

1.5.4.3 部署容器运行时 containerd

（1）验证基础容器镜像

1
2
3
4
5
6
7
root@k8s-ha2-deploy-239:/etc/kubeasz# grep SANDBOX_IMAGE ./clusters/* -R
./clusters/k8s-cluster1/config.yml:SANDBOX_IMAGE: "easzlab.io.local:5000/easzlab/pause:3.9"

# 如果修改了基础容器镜像的位置，需要在配置文件中修改
root@k8s-ha2-deploy-239:/etc/kubeasz# vim ./clusters/k8s-cluster1/config.yml
# [containerd]基础容器镜像
SANDBOX_IMAGE: "easzlab.io.local:5000/easzlab/pause:3.9"

（2）配置本地域名仓库解析，如果配置了 DNS 服务器，可不用配置

1
2
3
4
5
6
7
root@k8s-ha2-deploy-239:/etc/kubeasz# vim roles/containerd/tasks/main.yml
- block:
    - name: DNS harbor1 resolved
      shell: "echo '10.243.20.233 harbor.wuhaolam.top' >> /etc/hosts"

    - name: DNS harbor2 resolved
      shell: "echo '10.243.20.234 harbor2.wuhaolam.top' >> /etc/hosts"

（3）配置 nerdctl 客户端

1
2
3
4
5
6
7
8
9
root@k8s-ha2-deploy-239:~# ls nerdctl-1.4.0-linux-amd64.tar.gz 
nerdctl-1.4.0-linux-amd64.tar.gz
root@k8s-ha2-deploy-239:/etc/kubeasz# tar xvf /root/nerdctl-1.4.0-linux-amd64.tar.gz -C /etc/kubeasz/bin/containerd-bin/
root@k8s-ha2-deploy-239:/etc/kubeasz# cat roles/containerd/templates/nerdctl.toml.j2 
namespace = "k8s.io"
debug = false
debug_full = false
insecure_registry = true
root@k8s-ha2-deploy-239:/etc/kubeasz# vim roles/containerd/tasks/main.yml
image-20240104135838145 image-20240104140043957

nerdctl 软件包下载
https://github.com/containerd/nerdctl/releases

（4）开始部署运行时

1
root@k8s-ha2-deploy-239:/etc/kubeasz# ./ezctl setup k8s-cluster1 03

image-20240104141049176

（5）验证 containerd 服务

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
root@k8s-master1-230:~# ln -sv /usr/local/bin/containerd-bin/* /usr/local/bin/
root@k8s-master1-230:~# nerdctl -v
nerdctl version 1.4.0
root@k8s-master1-230:~# containerd -v
containerd github.com/containerd/containerd v1.6.20 2806fc1057397dbaeefbea0e4e17bddfbd388f38

root@k8s-node1-240:~# nerdctl login harbor.wuhaolam.top
Enter Username: admin
Enter Password: 
WARN[0003] skipping verifying HTTPS certs for "harbor.wuhaolam.top" 
WARNING: Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

1.5.4.4 部署 k8s Master 节点

1
root@k8s-ha2-deploy-239:/etc/kubeasz# ./ezctl setup k8s-cluster1 04

image-20240104143611299

1.5.4.5 部署 k8s node 节点

1
root@k8s-ha2-deploy-239:/etc/kubeasz# ./ezctl setup k8s-cluster1 05

image-20240104144202040

1.5.4.6 部署网络服务组件 calico（手动部署）

1
2
3
4
5
# 准备calico部署文件
root@k8s-ha2-deploy-239:~# ls
calico3.26.1-ipip_ubuntu2204-k8s-1.27.x.yaml

# 注意 calico 的yaml 文件中，网卡名称是否与宿主机网卡名相同和 pod 子网范围是否与 hosts 文件中相同

image-20230810203519016

image-20230810203754510

calico 的 yaml 文件
http://file.wuhaolam.top/calico3.26.1-ipip_ubuntu2204-k8s-1.27.x.yaml?e=1691668755&token=miUmOnzUyd6iPZJ_Mb0lwezwsmi9rGBR-TEVXr1z:hfGQt3MFblFwXRJOTBZIQ5GhzG0=

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
# 部署calico
root@k8s-ha2-deploy-239:~# kubectl apply -f calico3.26.1-ipip_ubuntu2204-k8s-1.27.x.yaml 
root@k8s-ha2-deploy-239:/etc/kubeasz# kubectl get pod -A
NAMESPACE     NAME                                      READY   STATUS    RESTARTS      AGE
kube-system   calico-kube-controllers-6655b6c4b-dsj4f   1/1     Running   1 (11m ago)   13m
kube-system   calico-node-4vzkg                         1/1     Running   0             13m
kube-system   calico-node-gc754                         1/1     Running   0             13m
kube-system   calico-node-llm5n                         1/1     Running   0             13m
kube-system   calico-node-w2k8c                         1/1     Running   0             13m
kube-system   calico-node-wczv2                         1/1     Running   0             13m
kube-system   calico-node-xlts7                         1/1     Running   0             13m


# node 节点验证
root@k8s-ha2-deploy-239:/etc/kubeasz# scp ./bin/calicoctl root@10.243.20.240:/usr/local/bin/
# 查看node1节点的calico邻居状态
root@k8s-node1-240:~# calicoctl node status
Calico process is running.

IPv4 BGP status
+---------------+-------------------+-------+----------+-------------+
| PEER ADDRESS  |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+---------------+-------------------+-------+----------+-------------+
| 10.243.20.232 | node-to-node mesh | up    | 07:40:00 | Established |
| 10.243.20.241 | node-to-node mesh | up    | 07:39:59 | Established |
| 10.243.20.242 | node-to-node mesh | up    | 07:39:59 | Established |
| 10.243.20.231 | node-to-node mesh | up    | 07:40:22 | Established |
| 10.243.20.230 | node-to-node mesh | up    | 07:40:25 | Established |
+---------------+-------------------+-------+----------+-------------+

1.6 部署 k8s 内部 DNS 服务-CoreDNS

CoreDNS 官网：https://coredns.io/
部署清单文件：https://github.com/coredns/deployment/tree/master/kubernetes

CoreDNS 源文件来自于 k8s 源码文件中 kubernetes-1.27.9\cluster\addons\dns\coredns
https://github.com/kubernetes/kubernetes/archive/refs/tags/v1.27.9.zip

1.6.1 修改 CoreDNS 的 yaml 文件

（1）修改域名解析相关配置

image-20240108151730383

（2）指定CoreDNS的镜像地址以及资源限制（生产时资源建议调大）

image-20240108152909521

（3）修改集群的 DNS 地址

1
2
3
4
5
6
7
8
9
10
11
# 修改的地址需要与 pod 中域名地址相同
root@k8s-ha2-deploy-239:/etc/kubeasz# kubectl run net-test1 --image=alpine sleep 36000
root@k8s-ha2-deploy-239:/etc/kubeasz# kubectl get pod
NAME        READY   STATUS    RESTARTS   AGE
net-test1   1/1     Running   0          18s
root@k8s-ha2-deploy-239:/etc/kubeasz# kubectl exec -it net-test1 sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
/ # cat /etc/resolv.conf 
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.100.0.2
options ndots:5
image-20240108155322955

1.6.2 部署 CoreDNS 服务

（1）开始部署

1
2
3
4
5
6
7
8
9
10
11
root@k8s-ha2-deploy-239:~# kubectl apply -f coredns-v1.10.1.yaml 
serviceaccount/coredns created
clusterrole.rbac.authorization.k8s.io/system:coredns created
clusterrolebinding.rbac.authorization.k8s.io/system:coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created

# 如果遇到错误则需要重新分发
kubectl delete -f coredns-v1.10.1.yaml
kubectl apply -f coredns-v1.10.1.yaml

（2）测试域名解析服务

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
root@k8s-ha2-deploy-239:/etc/kubeasz# kubectl exec -it net-test1 sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
/ # ping www.baidu.com
PING www.baidu.com (183.2.172.42): 56 data bytes
64 bytes from 183.2.172.42: seq=0 ttl=48 time=23.531 ms
64 bytes from 183.2.172.42: seq=1 ttl=48 time=23.447 ms
^C
--- www.baidu.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 23.447/23.489/23.531 ms
/ # ping 223.6.6.6 -c 3
PING 223.6.6.6 (223.6.6.6): 56 data bytes
64 bytes from 223.6.6.6: seq=0 ttl=112 time=10.705 ms
64 bytes from 223.6.6.6: seq=1 ttl=112 time=10.654 ms
64 bytes from 223.6.6.6: seq=2 ttl=112 time=10.591 ms

--- 223.6.6.6 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 10.591/10.650/10.705 ms