一、基于 Redis7.2 构建 Redis Cluster

虽然在哨兵 sentinel 机制中，可以解决 Redis 高可用的问题，即当 master 故障后可以自动将 slave 提升为 master 从而可以保证 Redis 服务的正常使用，但是无法解决 Redis 单机写入的瓶颈问题。单机的 Redis 写入性能受限于单机的内存大小、并发数量、网卡速率等因素，因此 Redis 官方在 Redis 3.0版本之后推出了无中心架构的 Redis Cluster 机制，在无中心的 Redis 集群当中，其每个节点保存当前节点数据和整个集群状态，每个节点都和其它所有节点连接，特点如下：

1
2
3
4
5
6
7
1、所有Redis节点互联（PING机制）
2、集群中某个节点的失效，是整个集群中超过半数的节点监测都失效才算真正的失效
3、客户端可以不需要proxy即可直接连接Redis，应用程序需要写全部的Redis服务器IP
4、Redis Cluster把所有的Redis node映射到0~16383个槽位上（slot），读写需要到指定的Redis node上进行操作，因此有多少个Redis node相当于Redis并发扩展了多少倍
5、Redis Cluster预先分配16384个槽位，当需要再Redis集群中写入一个key-value的时候，会使用CRC16(key) mod 16384之后的值，然后决定将key写入到哪一个槽位从而决定写入哪一个Redis节点上，从而有效解决单机瓶颈
6、集群最大内存空间是master节点的内存累计空间
7、写入性能是master的累计写入总和

Redis Cluster 数据读写流程：https://developer.redis.com/operate/redis-at-scale/scalability/redis-cluster-and-client-libraries/

1.1 环境准备

系统环境

1
2
3
4
root@ubuntu18-server:~# cat /etc/issue
Ubuntu 18.04.6 LTS \n \l

Redis version: 7.2.0

节点分配

image-20240329163735950 
1
2
3
4
5
6
7
8
192.168.119.161
192.168.119.162
192.168.119.163
192.168.119.164
192.168.119.165
192.168.119.166
192.168.119.167       # 预留扩容
192.168.119.168       # 预留扩容

1.1.1 安装 Redis7.2.0 版本并开启集群功能

==使用脚本自动化安装，其余节点相同==

1
2
3
4
5
root@ubuntu18-server:~# tar xvf redis7InstallScript.tar.gz
root@ubuntu18-server:~# cd redis7InstallScript/
root@ubuntu18-server:~/redis7InstallScript# ls
redis-7.2.0_install.sh  redis-7.2.0.tar.gz  redis.conf
root@ubuntu18-server:~/redis7InstallScript# bash redis-7.2.0_install.sh
image-20240312145731860 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
# 开启Redis Cluster功能
root@ubuntu18-server:~# vim /apps/redis/etc/redis.conf
cluster-enabled yes
cluster-config-file nodes-6379.conf
masterauth 123456
requirepass 123456
root@ubuntu18-server:~# systemctl restart redis-server.service

# 查看集群功能是否开启
root@ubuntu18-server:~# redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> info cluster
# Cluster
cluster_enabled:1

1.1.2 集群命令使用

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
root@ubuntu18-server:~# redis-cli --cluster help
Cluster Manager Commands:
  create         host1:port1 ... hostN:portN # 创建集群并执行节点数
                 --cluster-replicas <arg>    # 指定master的副本数量
  check          <host:port> or <host> <port> - separated by either colon or space
                 --cluster-search-multiple-owners # 检查是否有槽位被分配给了多个节点
  info           <host:port> or <host> <port> - separated by either colon or space # 查看集群主机信息
  fix            <host:port> or <host> <port> - separated by either colon or space # 修复集群
                 --cluster-search-multiple-owners # 修复槽位被同时分配给多个节点的问题
                 --cluster-fix-with-unreachable-masters # 修复不可达主机
  reshard        <host:port> or <host> <port> - separated by either colon or space # 手动热迁移集群指定主机的slots数据到新的目的主机
                 --cluster-from <arg>
                 --cluster-to <arg>
                 --cluster-slots <arg>
                 --cluster-yes
                 --cluster-timeout <arg>
                 --cluster-pipeline <arg>
                 --cluster-replace
  rebalance      <host:port> or <host> <port> - separated by either colon or space # 如果各master的slot槽位差异很大，可以使用rebalance实现自动平衡集群中各主机的slot数量
                 --cluster-weight <node1=w1...nodeN=wN>
                 --cluster-use-empty-masters
                 --cluster-timeout <arg>
                 --cluster-simulate
                 --cluster-pipeline <arg>
                 --cluster-threshold <arg>
                 --cluster-replace
  add-node       new_host:new_port existing_host:existing_port # 添加节点到集群
                 --cluster-slave
                 --cluster-master-id <arg>
  del-node       host:port node_id # 从集群删除主机
  call           host:port command arg arg .. arg # 在集群的所有节点上执行命令
                 --cluster-only-masters
                 --cluster-only-replicas
  set-timeout    host:port milliseconds # 设置节点的超时时间
  import         host:port # 将外部redis服务器的数据导入当前集群
                 --cluster-from <arg>
                 --cluster-from-user <arg>
                 --cluster-from-pass <arg>
                 --cluster-from-askpass
                 --cluster-copy
                 --cluster-replace
  backup         host:port backup_directory # 备份集群数据
  help           

For check, fix, reshard, del-node, set-timeout, info, rebalance, call, import, backup you can specify the host and port of any working node in the cluster.

Cluster Manager Options:
  --cluster-yes  Automatic yes to cluster commands prompts

1.2 集群初始化

==在任意一节点可执行初始化操作==

1
root@ubuntu18-server:~# redis-cli -a 123456 --cluster create 192.168.119.161:6379 192.168.119.162:6379 192.168.119.163:6379 192.168.119.164:6379 192.168.119.165:6379 192.168.119.166:6379 --cluster-replicas 1
image-20240312163830580 image-20240312164210477

1.3 验证集群状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
root@ubuntu18-server:~# redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
# 主从状态信息
127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:1
slave0:ip=192.168.119.165,port=6379,state=online,offset=714,lag=1
master_failover_state:no-failover
master_replid:851739edde74e60519b830540c5299448a589c2d
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:728
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:536870912
repl_backlog_first_byte_offset:1
repl_backlog_histlen:728

# 集群状态信息
127.0.0.1:6379> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:523
cluster_stats_messages_pong_sent:545
cluster_stats_messages_sent:1068
cluster_stats_messages_ping_received:540
cluster_stats_messages_pong_received:523
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:1068
total_cluster_links_buffer_limit_exceeded:0

# 集群节点信息，Slave节点除了显示自己的ID信息外，Master的ID信息也会展示
127.0.0.1:6379> cluster nodes
fdee09e80ceda3bb7fc11c8eedf3833ac09931ec 192.168.119.165:6379@16379 slave 5fcc191932ea0660f0b9a636d34e22aa926824d2 0 1710233373696 1 connected
b7d52d2dd34812c34b37fa8ad08ac83cf22cf40f 192.168.119.163:6379@16379 master - 0 1710233372684 3 connected 10923-16383
da2b2b032b96e25a97bc41e99d12ff0f56009c52 192.168.119.162:6379@16379 master - 0 1710233371000 2 connected 5461-10922
4873b095054dc04fc0ae0975dfae2df2d1756d9d 192.168.119.166:6379@16379 slave da2b2b032b96e25a97bc41e99d12ff0f56009c52 0 1710233373000 2 connected
83db779dbfc60c79983fbe22427f870f2add700c 192.168.119.164:6379@16379 slave b7d52d2dd34812c34b37fa8ad08ac83cf22cf40f 0 1710233371663 3 connected
5fcc191932ea0660f0b9a636d34e22aa926824d2 192.168.119.161:6379@16379 myself,master - 0 1710233370000 1 connected 0-5460

1.4 基于python向集群写入数据

准备Python脚本文件并写入数据

image-20240312205104305

image-20240312211536685

1
2
3
4
5
6
7
8
9
10
# 查看每个节点分配了多少key（主从复制，故总共200个）
root@ubuntu18-server:~# redis-cli -a 123456 --cluster call 192.168.119.161:6379 DBSIZE
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Calling DBSIZE
192.168.119.161:6379: 32
192.168.119.162:6379: 32
192.168.119.163:6379: 36
192.168.119.166:6379: 32
192.168.119.165:6379: 32
192.168.119.164:6379: 36

1.5 Redis Cluster 集群节点维护

集群维护主要是集群监控，另外集群运行时间长久之后，难免由于硬件故障、网络规划、业务增长等原因对已有集群进行调整，比如增加 Redis node 节点、服务器过保更换、服务器因故障下线等。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# 显示集群信息
root@ubuntu18-server:~# redis-cli -a 123456 --no-auth-warning --cluster info 192.168.119.161:6379
192.168.119.161:6379 (5fcc1919...) -> 32 keys | 5461 slots | 1 slaves.
192.168.119.162:6379 (da2b2b03...) -> 32 keys | 5462 slots | 1 slaves.
192.168.119.163:6379 (b7d52d2d...) -> 36 keys | 5461 slots | 1 slaves.
[OK] 100 keys in 3 masters.
0.01 keys per slot on average.

# 检查集群
root@ubuntu18-server:~# redis-cli -a 123456 --no-auth-warning --cluster check 192.168.119.161:6379

# 修复集群
root@ubuntu18-server:~# redis-cli -a 123456 --no-auth-warning --cluster fix 192.168.119.161:6379

# 在所有节点批量执行命令
root@ubuntu18-server:~# redis-cli -a 123456 --no-auth-warning --cluster call 192.168.119.161:6379 DBSIZE

1.5.1 集群监控

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
# 服务是否正常响应
root@ubuntu18-server:~# redis-cli -h 192.168.119.162 -a 123456 -p 6379 --no-auth-warning ping
PONG

# 已用的内存字节数
root@ubuntu18-server:~# redis-cli -h 192.168.119.163 -a 123456 -p 6379 --no-auth-warning info | grep -w "used_memory"
used_memory:1820160

# 已经连接的客户端数量
root@ubuntu18-server:~# redis-cli -h 192.168.119.163 -a 123456 -p 6379 --no-auth-warning info | grep -w "connected_clients"
connected_clients:1

# 获取集群的状态
root@ubuntu18-server:~# redis-cli -h 192.168.119.163 -a 123456 -p 6379 --no-auth-warning cluster info | grep -w "cluster_state"
cluster_state:ok

# 获取已经分配的槽位
root@ubuntu18-server:~# redis-cli -h 192.168.119.163 -a 123456 -p 6379 --no-auth-warning cluster info | grep -w "cluster_slots_assigned"
cluster_slots_assigned:16384

# 获取状态是 ok 的槽位
root@ubuntu18-server:~# redis-cli -h 192.168.119.163 -a 123456 -p 6379 --no-auth-warning cluster info | grep -w "cluster_slots_ok"
cluster_slots_ok:16384

# 获取PFAIL状态的槽位的数量。PFAIL表示暂停与当前节点通信，但数据可能会恢复
root@ubuntu18-server:~# redis-cli -h 192.168.119.163 -a 123456 -p 6379 --no-auth-warning cluster info | grep -w "cluster_slots_pfail"
cluster_slots_pfail:0

# 获取已经失败的槽位
root@ubuntu18-server:~# redis-cli -h 192.168.119.163 -a 123456 -p 6379 --no-auth-warning cluster info | g
cluster_slots_fail:0

# 集群中主节点的数量
root@ubuntu18-server:~# redis-cli -h 192.168.119.163 -a 123456 -p 6379 --no-auth-warning cluster info | g
cluster_size:3

# 集群中所有节点数
root@ubuntu18-server:~# redis-cli -h 192.168.119.163 -a 123456 -p 6379 --no-auth-warning cluster info | grep -w "cluster_known_nodes"
cluster_known_nodes:6

# 当前节点访问数据命中次数
root@ubuntu18-server:~# redis-cli -h 192.168.119.163 -a 123456 -p 6379 --no-auth-warning  info | grep -w keyspace_hits
keyspace_hits:75

# 由于集群没有此数据，当前节点的未命中次数
root@ubuntu18-server:~# redis-cli -h 192.168.119.162 -a 123456 -p 6379 --no-auth-warning  info | grep -w keyspace_misses
keyspace_misses:2

1.5.2 导入外部Redis数据

RedisShark：https://github.com/tair-opensource/RedisShake

1.5.2.1 功能简介

1
2
3
rdb_reader: 支持读取解析本地rdb文件，向目的redis/redis cluster恢复数据
sync_reader: 支持对源redis 和目的redis 进行在线数据同步，需要支持 psync(redis 2.8开始支持)，psync 具备了数据全量同步和增量同步模式
scan_reader: 支持基于dump从源主机导出key，然后在目的主机使用RESTORE导入key，通过SCAN命令遍历源端数据库中的所有 key，不推荐

1.5.2.2 准备 redis 源主机

将主机ubuntu18-server7: 192.168.119.117 的数据导入到 Redis Cluster，源主机与 Redis Cluster 的 key 不要冲突

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# 为了方便观察，先删除我们的集群中的数据
root@ubuntu18-server3:~# redis-cli -a 123456 --cluster call 192.168.119.161:6379 flushall
root@ubuntu18-server3:~# redis-cli -a 123456 --no-auth-warning --cluster call 192.168.119.161:6379 dbsize
>>> Calling dbsize
192.168.119.161:6379: 0
192.168.119.162:6379: 0
192.168.119.166:6379: 0
192.168.119.163:6379: 0
192.168.119.164:6379: 0
192.168.119.165:6379: 0

# Redis 服务已安装完成
● redis-server.service - Redis data structure server
   Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2024-03-13 01:38:08 UTC; 4h 35min ago
     Docs: https://redis.io/documentation
 Main PID: 1005 (redis-server)
   Status: "Ready to accept connections"
    Tasks: 5 (limit: 4623)
   CGroup: /system.slice/redis-server.service
           └─1005 /apps/redis/bin/redis-server 0.0.0.0:6379

Mar 13 01:38:08 ubuntu18-server7 systemd[1]: Starting Redis data structure server...
Mar 13 01:38:08 ubuntu18-server7 systemd[1]: Started Redis data structure server.

# 写入数据
root@ubuntu18-server7:~# cat redis-client.sh 
#!/bin/bash

for i in `seq 1 100000`;do
  redis-cli -h 127.0.0.1 -a 123456 --no-auth-warning set key-${i} value-${i} &
#  echo "key-${i} value-${i} 写入完成"
done

echo "十万个key写入完成!"
root@ubuntu18-server7:~# bash redis-client.sh

1.5.2.3 部署 Redis-shake并同步数据

参考文档：
https://tair-opensource.github.io/RedisShake/zh/reader/scan_reader.html
https://github.com/tair-opensource/RedisShake/blob/v4/shake.toml

1
2
3
4
5
6
7
8
9
root@ubuntu18-server7:/usr/local/src# cd /usr/local/src/
root@ubuntu18-server7:/usr/local/src# wget https://github.com/tair-opensource/RedisShake/releases/download/v4.0.5/redis-shake-linux-amd64.tar.gz

root@ubuntu18-server7:/usr/local/src# tar xvf redis-shake-linux-amd64.tar.gz 
./redis-shake
./shake.toml

# 编辑配置文件
root@ubuntu18-server7:/usr/local/src# vim shake.toml

image-20240313145153963

1
root@ubuntu18-server7:/usr/local/src# ./redis-shake shake.toml

image-20240313150224485

1
2
3
4
5
6
7
8
9
# 写入一个数据
root@ubuntu18-server7:~# redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> set name Jerry
OK

# 观察同步状态，read_count 和 write_count 的数量加1
2024-03-13 07:03:22 INF read_count=[100001], read_ops=[0.00], write_count=[100001], write_ops=[0.00], syncing aof, diff=[0]
2024-03-13 07:03:27 INF read_count=[100001], read_ops=[0.00], write_count=[100001], write_ops=[0.00], syncing aof, diff=[0]

1.5.2.4 集群节点验证数据

1
2
3
4
5
6
7
8
9
# Master 节点数量一共100001个
root@ubuntu18-server3:~# redis-cli -a 123456 --no-auth-warning --cluster call 192.168.119.161:6379 dbsize
>>> Calling dbsize
192.168.119.161:6379: 33365
192.168.119.162:6379: 33280
192.168.119.166:6379: 33280
192.168.119.163:6379: 33356
192.168.119.164:6379: 33356
192.168.119.165:6379: 33365

1.5.3 Redis 横向扩容（添加服务器）

因业务发展需要，原本的三主三从集群架构无法满足现有业务的并发写入需求，因此添加两台服务器192.168.119.167、192.168.119.168，需要动态的添加到集群中并且不能影响业务的使用和数据丢失。

1.5.3.1 添加节点

1
2
3
4
5
6
# 开启集群功能
root@ubuntu18-server7:~# vim /apps/redis/etc/redis.conf
masterauth 123456
cluster-enabled yes
cluster-config-file nodes-6379.conf
root@ubuntu18-server7:~# systemctl restart redis-server.service
1
2
# 添加192.168.119.167至目标集群中
root@ubuntu18-server:~# redis-cli -a 123456 --no-auth-warning --cluster add-node 192.168.119.167:6379 192.168.119.161:6379
image-20240313160154710 
1
2
3
4
5
6
7
8
9
10
# 获取节点信息，用于添加从节点至主节点（新增的节点默认为Master）
root@ubuntu18-server:~# redis-cli -a 123456 --no-auth-warning
127.0.0.1:6379> CLUSTER NODES
5fcc191932ea0660f0b9a636d34e22aa926824d2 192.168.119.161:6379@16379 myself,master - 0 1710316998000 1 connected 0-5460
155755f369ad444dd54741f014ede241ad2986f2 192.168.119.167:6379@16379 master - 0 1710317000000 0 connected
da2b2b032b96e25a97bc41e99d12ff0f56009c52 192.168.119.162:6379@16379 master - 0 1710317001000 2 connected 5461-10922
4873b095054dc04fc0ae0975dfae2df2d1756d9d 192.168.119.166:6379@16379 slave da2b2b032b96e25a97bc41e99d12ff0f56009c52 0 1710317001829 2 connected
b7d52d2dd34812c34b37fa8ad08ac83cf22cf40f 192.168.119.163:6379@16379 master - 0 1710317002847 3 connected 10923-16383
83db779dbfc60c79983fbe22427f870f2add700c 192.168.119.164:6379@16379 slave b7d52d2dd34812c34b37fa8ad08ac83cf22cf40f 0 1710317000808 3 connected
fdee09e80ceda3bb7fc11c8eedf3833ac09931ec 192.168.119.165:6379@16379 slave 5fcc191932ea0660f0b9a636d34e22aa926824d2 0 1710317001000 1 connected
1
2
# 添加192.168.119.168为192.168.119.167的从节点
root@ubuntu18-server:~# redis-cli -a 123456 --cluster add-node 192.168.119.168:6379 192.168.119.161:6379 --cluster-slave --cluster-master-id 155755f369ad444dd54741f014ede241ad2986f2
image-20240313160728215 
1
2
# 查看当前集群信息
root@ubuntu18-server:~# redis-cli -a 123456 --no-auth-warning cluster nodes

image-20240313160945319

1.5.3.2 重新分配槽位

1
root@ubuntu18-server:~# redis-cli -a 123456 --no-auth-warning --cluster reshard 192.168.119.161:6379
image-20240313162722478

1.5.3.3 验证槽位分配情况

1
root@ubuntu18-server:~# redis-cli -a 123456 --no-auth-warning --cluster check 192.168.119.161:6379
image-20240313163144139

1.5.4 Redis 集群缩容（删除服务器）

主要涉及到服务器下架前的数据迁移、以及从集群删除服务器

1.5.4.1 迁移需要删除的Master槽位至其它Master

1
root@ubuntu18-server:~# redis-cli -a 123456 --no-auth-warning --cluster reshard 192.168.119.161:6379
image-20240313165613208 
1
2
3
4
5
6
7
8
9
10
# 验证集群主机状态（167和168都为Slave）
root@ubuntu18-server:~# redis-cli -a 123456 --no-auth-warning cluster nodes
9608fb089d98a384dd785516cf5747f92e7c291f 192.168.119.168:6379@16379 slave 5fcc191932ea0660f0b9a636d34e22aa926824d2 0 1710320306343 8 connected
5fcc191932ea0660f0b9a636d34e22aa926824d2 192.168.119.161:6379@16379 myself,master - 0 1710320302000 8 connected 0-6826 10923-12287
155755f369ad444dd54741f014ede241ad2986f2 192.168.119.167:6379@16379 slave 5fcc191932ea0660f0b9a636d34e22aa926824d2 0 1710320302000 8 connected
da2b2b032b96e25a97bc41e99d12ff0f56009c52 192.168.119.162:6379@16379 master - 0 1710320305323 2 connected 6827-10922
4873b095054dc04fc0ae0975dfae2df2d1756d9d 192.168.119.166:6379@16379 slave da2b2b032b96e25a97bc41e99d12ff0f56009c52 0 1710320304000 2 connected
b7d52d2dd34812c34b37fa8ad08ac83cf22cf40f 192.168.119.163:6379@16379 master - 0 1710320304306 3 connected 12288-16383
83db779dbfc60c79983fbe22427f870f2add700c 192.168.119.164:6379@16379 slave b7d52d2dd34812c34b37fa8ad08ac83cf22cf40f 0 1710320303282 3 connected
fdee09e80ceda3bb7fc11c8eedf3833ac09931ec 192.168.119.165:6379@16379 slave 5fcc191932ea0660f0b9a636d34e22aa926824d2 0 1710320304000 8 connected

1.5.4.2 从集群中删除节点

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
root@ubuntu18-server:~# redis-cli -a 123456 --no-auth-warning --cluster del-node 192.168.119.161:6379 9608fb089d98a384dd785516cf5747f92e7c291f
>>> Removing node 9608fb089d98a384dd785516cf5747f92e7c291f from cluster 192.168.119.161:6379
>>> Sending CLUSTER FORGET messages to the cluster...
>>> Sending CLUSTER RESET SOFT to the deleted node.

root@ubuntu18-server:~# redis-cli -a 123456 --no-auth-warning --cluster del-node 192.168.119.161:6379 155755f369ad444dd54741f014ede241ad2986f2
>>> Removing node 155755f369ad444dd54741f014ede241ad2986f2 from cluster 192.168.119.161:6379
>>> Sending CLUSTER FORGET messages to the cluster...
>>> Sending CLUSTER RESET SOFT to the deleted node.

root@ubuntu18-server:~# redis-cli -a 123456 --no-auth-warning --cluster check 192.168.119.161:6379
192.168.119.161:6379 (5fcc1919...) -> 50038 keys | 8192 slots | 1 slaves.
192.168.119.162:6379 (da2b2b03...) -> 24964 keys | 4096 slots | 1 slaves.
192.168.119.163:6379 (b7d52d2d...) -> 24999 keys | 4096 slots | 1 slaves.
[OK] 100001 keys in 3 masters.
6.10 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.119.161:6379)
M: 5fcc191932ea0660f0b9a636d34e22aa926824d2 192.168.119.161:6379
   slots:[0-6826],[10923-12287] (8192 slots) master
   1 additional replica(s)
M: da2b2b032b96e25a97bc41e99d12ff0f56009c52 192.168.119.162:6379
   slots:[6827-10922] (4096 slots) master
   1 additional replica(s)
S: 4873b095054dc04fc0ae0975dfae2df2d1756d9d 192.168.119.166:6379
   slots: (0 slots) slave
   replicates da2b2b032b96e25a97bc41e99d12ff0f56009c52
M: b7d52d2dd34812c34b37fa8ad08ac83cf22cf40f 192.168.119.163:6379
   slots:[12288-16383] (4096 slots) master
   1 additional replica(s)
S: 83db779dbfc60c79983fbe22427f870f2add700c 192.168.119.164:6379
   slots: (0 slots) slave
   replicates b7d52d2dd34812c34b37fa8ad08ac83cf22cf40f
S: fdee09e80ceda3bb7fc11c8eedf3833ac09931ec 192.168.119.165:6379
   slots: (0 slots) slave
   replicates 5fcc191932ea0660f0b9a636d34e22aa926824d2
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

1.5.4.3 重新平衡槽位

1
2
3
root@ubuntu18-server:~# redis-cli -a 123456 --no-auth-warning --cluster rebalance 192.168.119.161:6379

# 检查槽位分配情况
image-20240313170540258

1.5.5 验证 Master 宕机后的故障转移

当前集群环境

image-20240313174145458

1.5.5.1 关闭一个Master

1
2
# 关闭Master3这个节点
root@ubuntu18-server3:~# systemctl stop redis-server.service

1.5.5.2 查看集群主机状态

1
2
3
4
5
6
7
8
# 164 已经成为 Master
root@ubuntu18-server:~# redis-cli -a 123456 --no-auth-warning cluster nodes
5fcc191932ea0660f0b9a636d34e22aa926824d2 192.168.119.161:6379@16379 myself,master - 0 1710324014000 8 connected 2731-6826 10923-12287
da2b2b032b96e25a97bc41e99d12ff0f56009c52 192.168.119.162:6379@16379 master - 0 1710324015000 9 connected 0-1365 6827-10922
4873b095054dc04fc0ae0975dfae2df2d1756d9d 192.168.119.166:6379@16379 slave da2b2b032b96e25a97bc41e99d12ff0f56009c52 0 1710324015401 9 connected
b7d52d2dd34812c34b37fa8ad08ac83cf22cf40f 192.168.119.163:6379@16379 master,fail - 1710323975484 1710323971000 10 disconnected
83db779dbfc60c79983fbe22427f870f2add700c 192.168.119.164:6379@16379 master - 0 1710324016422 11 connected 1366-2730 12288-16383
fdee09e80ceda3bb7fc11c8eedf3833ac09931ec 192.168.119.165:6379@16379 slave 5fcc191932ea0660f0b9a636d34e22aa926824d2 0 1710324013355 8 connected

1.5.5.3 验证数据写入

1
2
3
4
5
# 查看164节点数据大小
root@ubuntu18-server:~# redis-cli -a 123456 -h 192.168.119.164 --no-auth-warning dbsize
(integer) 33329

# 往集群写入数据
image-20240313180459492 
1
2
3
# 查看数据写入情况
root@ubuntu18-server:~# redis-cli -a 123456 -h 192.168.119.164 --no-auth-warning dbsize
(integer) 33431

1.5.5.4 修复主机并重新加入集群

1
2
3
4
5
6
7
8
9
10
11
12
root@ubuntu18-server3:~# systemctl start redis-server.service

# 查看集群状态
root@ubuntu18-server:~# redis-cli -a 123456 --no-auth-warning cluster nodes
5fcc191932ea0660f0b9a636d34e22aa926824d2 192.168.119.161:6379@16379 myself,master - 0 1710324463000 8 connected 2731-6826 10923-12287
da2b2b032b96e25a97bc41e99d12ff0f56009c52 192.168.119.162:6379@16379 master - 0 1710324465797 9 connected 0-1365 6827-10922
4873b095054dc04fc0ae0975dfae2df2d1756d9d 192.168.119.166:6379@16379 slave da2b2b032b96e25a97bc41e99d12ff0f56009c52 0 1710324465000 9 connected
b7d52d2dd34812c34b37fa8ad08ac83cf22cf40f 192.168.119.163:6379@16379 slave 83db779dbfc60c79983fbe22427f870f2add700c 0 1710324466819 11 connected
83db779dbfc60c79983fbe22427f870f2add700c 192.168.119.164:6379@16379 master - 0 1710324464000 11 connected 1366-2730 12288-16383
fdee09e80ceda3bb7fc11c8eedf3833ac09931ec 192.168.119.165:6379@16379 slave 5fcc191932ea0660f0b9a636d34e22aa926824d2 0 1710324463755 8 connected

root@ubuntu18-server:~# redis-cli -a 123456 --no-auth-warning --cluster check 192.168.119.161:6379
image-20240313181004567