HMK's blog

保持思考|00后|等待


  • Home
  • Archive
  • Tags
  •   

© 2026 Hekang

Theme Typography by Makito

Proudly published with Hexo

Kubeasz-k8s1.35-longhorn

Posted at 2026-07-20 云原生 

Kubeasz 部署 K8s1.35 (存储:Longhorn)


一、环境规划

1.1 节点规划

角色 主机名 IP(示例) 配置 数量
Master node1 10.10.0.204 4C8G 50G SSD 1
Worker node2 10.10.0.201 4C8G 50G SSD 1
Worker node3 10.10.0.202 4C8G 50G SSD 1
Worker node4 10.10.0.203 8C16G 100G SSD 1

生产建议:Master 至少 3 节点保证高可用;Worker 根据业务负载规划;Longhorn 推荐每个 Worker 挂载一块独立数据盘(如 /dev/vdb),避免与系统盘争用 IO。

1.2 系统要求

项目 要求
OS Ubuntu 20.04/22.04 或 CentOS 7.9/8(推荐 Ubuntu 22.04)
Kernel ≥ 4.18(Longhorn 需要 iSCSI 支持,推荐 5.x)
网络 全节点互通,关闭 Swap
磁盘 Longhorn 节点需额外数据盘,格式化为 ext4/xfs
时间 所有节点 NTP 同步

二、部署前准备

2.1 基础环境初始化(所有节点)

1
# 配置安装节点与其他节点免密

2.2 iSCSI 客户端安装(Longhorn 必需,所有 Worker 节点)

1
2
3
4
5
6
# Ubuntu
apt install -y open-iscsi nfs-common

# 启动 iscsid
systemctl enable iscsid
systemctl start iscsid

三、Kubeasz 部署 K8s

3.1 安装 Kubeasz

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# 在部署机(可选独立部署机或 master-01)
export RELEASE=v3.6.9 # 检查最新版本: https://github.com/easz/kubeasz/releases

# 下载 kubeasz
curl -L https://github.com/easzlab/kubeasz/releases/download/${RELEASE}/ezdown -o /usr/local/bin/ezdown
chmod +x /usr/local/bin/ezdown


# Ubuntu 临时禁用ipv6
root@node1:~# # 临时禁用(重启网络后恢复)
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1

# 安装节点 配置和其他节点 免密
# 初始化目录
ezdown -D
#下载离线包
# ezdown -X cilium
# 下载离线镜像(可选,加速部署)
ezdown -X v1.28.2

3.2 配置集群

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# 容器化运行kubeasz
root@node1:~# ezdown -S
root@node1:~# cat ~/.bashrc
alias dk='docker exec -it kubeasz' # generated by kubeasz


# 一键安装,等价于执行docker exec -it kubeasz ezctl setup k8s-01 all
dk ezctl setup k8s-01 all

# 或者分步安装,具体使用 dk ezctl help setup 查看分步安装帮助信息
# dk ezctl setup k8s-01 01
# dk ezctl setup k8s-01 02
# dk ezctl setup k8s-01 03
# dk ezctl setup k8s-01 04


# 创建集群配置
root@node1:~# dk ezctl new k8s-prod
2026-07-17 08:47:33 [ezctl:145] DEBUG generate custom cluster files in /etc/kubeasz/clusters/k8s-prod
2026-07-17 08:47:33 [ezctl:151] DEBUG set versions
2026-07-17 08:47:33 [ezctl:181] DEBUG cluster k8s-prod: files successfully created.
2026-07-17 08:47:33 [ezctl:182] INFO next steps 1: to config '/etc/kubeasz/clusters/k8s-prod/hosts'
2026-07-17 08:47:33 [ezctl:183] INFO next steps 2: to config '/etc/kubeasz/clusters/k8s-prod/config.yml'

3.3 编辑集群配置文件

hosts 文件 (/etc/kubeasz/clusters/k8s-prod/hosts):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
root@node1:~# cat /etc/kubeasz/clusters/k8s-prod/hosts
# 'etcd' cluster should have odd member(s) (1,3,5,...)
[etcd]
10.10.0.204
# master node(s), set unique 'k8s_nodename' for each node
# CAUTION: 'k8s_nodename' must consist of lower case alphanumeric characters, '-' or '.',
# and must start and end with an alphanumeric character
[kube_master]
10.10.0.204 k8s_nodename='node1'
# work node(s), set unique 'k8s_nodename' for each node
# CAUTION: 'k8s_nodename' must consist of lower case alphanumeric characters, '-' or '.',
# and must start and end with an alphanumeric character
[kube_node]
10.10.0.202 k8s_nodename='node3'
10.10.0.201 k8s_nodename='node2'
10.10.0.203 k8s_nodename='node4'
10.10.0.204 k8s_nodename='node1'
# [optional] harbor server, a private docker registry
# 'NEW_INSTALL': 'true' to install a harbor server; 'false' to integrate with existed one
[harbor]
10.10.0.204 NEW_INSTALL=true

# [optional] loadbalance for accessing k8s from outside
[ex_lb]
#192.168.1.6 LB_ROLE=backup EX_APISERVER_VIP=192.168.1.250 EX_APISERVER_PORT=8443
#192.168.1.7 LB_ROLE=master EX_APISERVER_VIP=192.168.1.250 EX_APISERVER_PORT=8443

# [optional] ntp server for the cluster
[chrony]
10.10.0.204

[all:vars]
# --------- Main Variables ---------------
# Secure port for apiservers
SECURE_PORT="6443"

# Cluster container-runtime supported: docker, containerd
# if k8s version >= 1.24, docker is not supported
CONTAINER_RUNTIME="containerd"

# Network plugins supported: calico, flannel, kube-router, cilium, kube-ovn
CLUSTER_NETWORK="cilium"

# Service proxy mode of kube-proxy: 'iptables' or 'ipvs'
PROXY_MODE="ipvs"

# K8S Service CIDR, not overlap with node(host) networking
SERVICE_CIDR="10.68.0.0/16"

# Cluster CIDR (Pod CIDR), not overlap with node(host) networking
CLUSTER_CIDR="172.20.0.0/16"

# NodePort Range
NODE_PORT_RANGE="30000-32767"

# Cluster DNS Domain
CLUSTER_DNS_DOMAIN="cluster.local"

# -------- Additional Variables (don't change the default value right now) ---
# Binaries Directory
bin_dir="/opt/kube/bin"

# Deploy Directory (kubeasz workspace)
base_dir="/etc/kubeasz"

# Directory for a specific cluster
cluster_dir="{{ base_dir }}/clusters/k8s-prod"

# CA and other components cert/key Directory
ca_dir="/etc/kubernetes/ssl"

# Default 'k8s_nodename' is empty
k8s_nodename=''

# Default python interpreter
ansible_python_interpreter=/usr/bin/python3

config.yml (/etc/kubeasz/clusters/k8s-prod/config.yml):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
root@node1:~# cat /etc/kubeasz/clusters/k8s-prod/config.yml
############################
# prepare
############################
# 可选离线安装系统软件包 (offline|online)
INSTALL_SOURCE: "online"

# 可选进行系统安全加固 github.com/dev-sec/ansible-collection-hardening
# (deprecated) 未更新上游项目,未验证最新k8s集群安装,不建议启用
OS_HARDEN: false


############################
# role:deploy
############################
# default: ca will expire in 100 years
# default: certs issued by the ca will expire in 50 years
CA_EXPIRY: "876000h"
CERT_EXPIRY: "438000h"

# force to recreate CA and other certs, not suggested to set 'true'
CHANGE_CA: false

# kubeconfig 配置参数
CLUSTER_NAME: "cluster1"
CONTEXT_NAME: "context-{{ CLUSTER_NAME }}"

# k8s version
K8S_VER: "1.35.4"

# set unique 'k8s_nodename' for each node, if not set(default:'') ip add will be used
# CAUTION: 'k8s_nodename' must consist of lower case alphanumeric characters, '-' or '.',
# and must start and end with an alphanumeric character (e.g. 'example.com'),
# regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'
K8S_NODENAME: "{%- if k8s_nodename != '' -%} \
{{ k8s_nodename|replace('_', '-')|lower }} \
{%- else -%} \
k8s-{{ inventory_hostname|replace('.', '-') }} \
{%- endif -%}"

# use 'K8S_NODENAME' to set hostname
ENABLE_SETTING_HOSTNAME: true


############################
# role:etcd
############################
# 设置不同的wal目录,可以避免磁盘io竞争,提高性能
ETCD_DATA_DIR: "/var/lib/etcd"
ETCD_WAL_DIR: ""


############################
# role:runtime [containerd,docker]
############################
# [.]启用拉取加速镜像仓库
ENABLE_MIRROR_REGISTRY: true

# [.]添加信任的私有仓库
# 必须按照如下示例格式,协议头'http://'和'https://'不能省略
INSECURE_REG:
- "http://easzlab.io.local:5000"
- "https://reg.yourcompany.com"

# [.]基础容器镜像
SANDBOX_IMAGE: "easzlab.io.local:5000/easzlab/pause:3.10"

# [containerd] root 存储目录,默认:/var/lib/containerd
CONTAINERD_ROOT_DIR: "/var/lib/containerd"

# [containerd] state 存储目录,默认:/run/containerd
CONTAINERD_STATE_DIR: "/run/containerd"

# [containerd] config 目录,默认:/etc/containerd
CONTAINERD_CONFIG_DIR: "/etc/containerd"

# [containerd] systemd service 名称,默认:containerd.service
CONTAINERD_SERVICE_NAME: "containerd.service"

# [docker]容器存储目录
DOCKER_STORAGE_DIR: "/var/lib/docker"

# [docker]开启Restful API
DOCKER_ENABLE_REMOTE_API: false


############################
# role:kube-master
############################
# k8s 集群 master 节点证书配置,可以添加多个ip和域名(比如增加公网ip和域名)
MASTER_CERT_HOSTS:
- "10.1.1.1"
- "k8s.easzlab.io"
#- "www.test.com"

# node 节点上 pod 网段掩码长度(决定每个节点最多能分配的pod ip地址)
# 如果flannel 使用 --kube-subnet-mgr 参数,那么它将读取该设置为每个节点分配pod网段
# https://github.com/coreos/flannel/issues/847
NODE_CIDR_LEN: 24

# 是否启用集群audit功能
ENABLE_CLUSTER_AUDIT: false

############################
# role:kube-node
############################
# Kubelet 根目录
KUBELET_ROOT_DIR: "/var/lib/kubelet"

# node节点最大pod 数
MAX_PODS: 110

# 配置为kube组件(kubelet,kube-proxy,dockerd等)预留的资源量
# 数值设置详见templates/kubelet-config.yaml.j2
KUBE_RESERVED_ENABLED: "no"

# k8s 官方不建议草率开启 system-reserved, 除非你基于长期监控,了解系统的资源占用状况;
# 并且随着系统运行时间,需要适当增加资源预留,数值设置详见templates/kubelet-config.yaml.j2
# 系统预留设置基于 4c/8g 虚机,最小化安装系统服务,如果使用高性能物理机可以适当增加预留
# 另外,集群安装时候apiserver等资源占用会短时较大,建议至少预留1g内存
SYS_RESERVED_ENABLED: "no"


############################
# role:network [flannel,calico,cilium,kube-ovn,kube-router]
############################
# ------------------------------------------- flannel
# [flannel]设置flannel 后端"host-gw","vxlan"等
FLANNEL_BACKEND: "vxlan"
DIRECT_ROUTING: false

# [flannel]
flannel_ver: "v0.27.3"

# ------------------------------------------- calico
# 模式可选项有: [Always, CrossSubnet, Never],跨子网可以配置为Always与CrossSubnet
# CrossSubnet为隧道+BGP路由混合模式可以提升网络性能,同子网配置为Never即可.
# 公有云建议使用always比较省事,其他的话需要修改各自公有云的网络配置,具体可以参考各个公有云说明
CALICO_ENABLE_OVERLAY: "Always"

# [calico]设置 calico-node使用的host IP,bgp邻居通过该地址建立,可手工指定也可以自动发现
IP_AUTODETECTION_METHOD: "can-reach={{ groups['kube_master'][0] }}"

# [calico]设置calico 网络 backend: bird, vxlan, none
# 少数公有云(Azure)或者私有云不支持IPinIP封包,可以使用 vxlan 模式
CALICO_NETWORKING_BACKEND: "bird"

# [calico]设置calico 是否使用route reflectors
# 如果集群规模超过50个节点,建议启用该特性
CALICO_RR_ENABLED: false

# CALICO_RR_NODES 配置route reflectors的节点,如果未设置默认使用集群master节点
# CALICO_RR_NODES: ["192.168.1.1", "192.168.1.2"]
CALICO_RR_NODES: []

# [calico]更新支持calico 版本: ["3.19", "3.23"]
calico_ver: "v3.28.4"

# [calico]calico 主版本
calico_ver_main: "{{ calico_ver.split('.')[0] }}.{{ calico_ver.split('.')[1] }}"

# ------------------------------------------- cilium
# [cilium]镜像版本
cilium_ver: "1.17.4"
cilium_connectivity_check: false
cilium_hubble_enabled: false
cilium_hubble_ui_enabled: false

# ------------------------------------------- kube-ovn
# [kube-ovn]离线镜像tar包
kube_ovn_ver: "v1.11.5"

# ------------------------------------------- kube-router
# [kube-router]公有云上存在限制,一般需要始终开启 ipinip;自有环境可以设置为 "subnet"
OVERLAY_TYPE: "full"

# [kube-router]NetworkPolicy 支持开关
FIREWALL_ENABLE: true

# [kube-router]kube-router 镜像版本
kube_router_ver: "v1.5.4"


############################
# role:cluster-addon
############################
# coredns 自动安装
dns_install: "yes"
corednsVer: "1.12.4"
ENABLE_LOCAL_DNS_CACHE: true
dnsNodeCacheVer: "1.26.4"
# 设置 local dns cache 地址
LOCAL_DNS_CACHE: "169.254.20.10"

# metric server 自动安装
metricsserver_install: "yes"
metricsVer: "v0.8.0"

# dashboard 自动安装
dashboard_install: "no"
dashboardVer: "7.14.0"

# local-storage (local-path-provisioner) 自动安装
local_path_provisioner_install: "no"
local_path_provisioner_ver: "v0.0.31"
local_path_storage_class: "local-path"
# 设置默认本地存储路径
local_path_provisioner_dir: "/opt/local-path-provisioner"

# nfs-provisioner 自动安装
nfs_provisioner_install: "no"
nfs_provisioner_namespace: "kube-system"
nfs_provisioner_ver: "v4.0.2"
nfs_storage_class: "managed-nfs-storage"
nfs_server: "192.168.1.10"
nfs_path: "/data/nfs"

# openebs 自动安装
openebs_install: "no"
openebs_ver: "4.3.2"
openebs_namespace: "openebs"
openebs_hostpath: "/var/openebs/local"
openebs_hostpath_storage_class: "openebs-hostpath"
openebs_lvm_storage_class: "openebs-lvmpv"
openebs_lvm_vg: "vg_k8s"

# prometheus 自动安装
prom_install: "no"
prom_namespace: "monitor"
prom_storage_class: ""
prom_chart_ver: "75.7.0"

# network-check 自动安装
network_check_enabled: false
network_check_schedule: "*/5 * * * *"

# kubeblocks 自动安装
kubeblocks_ver: "1.0.1"
kubeblocks_install: "no"

# ingress-nginx 自动安装
# ingress-nginx 只会部署到node with 标签:ingress-controller/provider=ingress-nginx
ingress_nginx_install: "no"
ingress_nginx_namespace: "ingress-nginx"
ingress_nginx_ver: "4.13.0"
ingress_nginx_metrics_enabled: false # 需要先部署prometheus

# argocd 自动安装
argocd_install: "no"


############################
# role:harbor
############################
# harbor version,完整版本号
HARBOR_VER: "v2.12.4"
HARBOR_DOMAIN: "harbor.easzlab.io.local"
HARBOR_PATH: /var/data
HARBOR_TLS_PORT: 8443
HARBOR_REGISTRY: "{{ HARBOR_DOMAIN }}:{{ HARBOR_TLS_PORT }}"

# if set 'false', you need to put certs named harbor.pem and harbor-key.pem in directory 'down'
HARBOR_SELF_SIGNED_CERT: true

# install extra component
HARBOR_WITH_TRIVY: false

3.4 执行部署

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
# 部署所有
root@node1:~# dk ezctl setup k8s-prod all
ansible-playbook -i clusters/k8s-prod/hosts -e @clusters/k8s-prod/config.yml playbooks/90.setup.yml
*** Component Version *********************
*******************************************
* kubernetes: v1.35.4
* etcd: v3.6.7
* cilium: 1.17.4
*******************************************
2026-07-17 09:00:03 [ezctl:248] INFO cluster:k8s-prod setup step:all begins in 5s, press any key to abort:



#
# 步骤1:初始化环境
dk ezctl setup k8s-prod 01.prepare

# 步骤2:部署 etcd
dk ezctl setup k8s-prod 02.etcd

# 步骤3:部署容器运行时
dk ezctl setup k8s-prod 03.containerd

# 步骤4:部署 master 组件
dk ezctl setup k8s-prod 04.kube-master

# 步骤5:部署网络插件
dk ezctl setup k8s-prod 05.kube-node

# 步骤6:部署 worker 节点
dk ezctl setup k8s-prod 06.network

# 步骤7:部署核心服务
dk ezctl setup k8s-prod 07.cluster-addon

# 步骤8:部署负载均衡
dk ezctl setup k8s-prod 08.lb

# 步骤9:部署存储(Longhorn,见第四节)
# 手动执行

3.5 验证集群

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
root@node1:~# kubectl get nodes
NAME STATUS ROLES AGE VERSION
node1 Ready master 108m v1.35.4
node2 Ready node 107m v1.35.4
node3 Ready node 107m v1.35.4
node4 Ready node 107m v1.35.4
root@node1:~# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
etcd-0 Healthy ok
scheduler Healthy ok
root@node1:~# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
cilium-5kzff 1/1 Running 5 (43m ago) 107m
cilium-nr2sz 1/1 Running 5 (43m ago) 107m
cilium-operator-7888cb5457-76hdl 1/1 Running 1 (39m ago) 41m
cilium-rf2qq 1/1 Running 5 (43m ago) 107m
cilium-tbg7c 1/1 Running 5 (43m ago) 107m
coredns-5d577844c9-5rxs5 1/1 Running 0 41m
metrics-server-5f54f964bd-2ldbp 1/1 Running 0 6m30s
node-local-dns-7hrjp 1/1 Running 0 105m
node-local-dns-pgm65 1/1 Running 0 105m
node-local-dns-sttgr 1/1 Running 0 105m
node-local-dns-wxlc5 1/1 Running 0 105m


安装harbor仓库

需要提前把包放进去

1
2
3
4
root@node1:/etc/kubeasz/down# ls
harbor-offline-installer-v2.12.4.tgz
# dk ezctl setup k8s-prod 11

四、Longhorn 存储部署

4.1 准备数据盘(所有 Worker 节点)

1
2
3
4
5
6
7
8
9
10
11
12
13
# 查看数据盘(假设 /dev/vdb 为 Longhorn 专用数据盘)
lsblk

# 格式化数据盘
mkfs.ext4 -F /dev/sdb

# 挂载到 Longhorn 数据目录
mkdir -p /data/longhorn
mount /dev/sdb /data/longhorn
systemctl daemon-reload
# 写入 fstab 持久化
echo '/dev/sdb /data/longhorn ext4 defaults 0 2' >> /etc/fstab
systemctl daemon-reload

4.2 安装 Longhorn

安装Helm

1
root@node1:/etc/kubeasz/down# curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# 创建命名空间
kubectl create namespace longhorn-system

# 方式一:使用 Helm 安装(推荐)
helm repo add longhorn https://charts.longhorn.io
helm repo update

helm install longhorn longhorn/longhorn \
--namespace longhorn-system \
--create-namespace \
--version 1.8.1 \
--set defaultSettings.defaultDataPath=/data/longhorn \
--set defaultSettings.storageOverProvisioningPercentage=200 \
--set defaultSettings.storageMinimalAvailablePercentage=10 \
--set defaultSettings.replicaAutoBalance=best-effort \
--set persistence.defaultClass=true \
--set persistence.defaultClassReplicaCount=3 \
--set longhornManager.resources.requests.cpu=250m \
--set longhornManager.resources.requests.memory=512Mi \
--set longhornDriver.resources.requests.cpu=100m \
--set longhornDriver.resources.requests.memory=256Mi
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
# 提前下载镜像并导入
#./save-images.sh -i /root/longhorn.tar.gz

root@node1:~# cat longhorn-images.txt
longhornio/csi-attacher:v4.8.1
longhornio/csi-provisioner:v5.2.0
longhornio/csi-resizer:v1.13.2
longhornio/csi-snapshotter:v8.2.0
longhornio/csi-node-driver-registrar:v2.13.0
longhornio/livenessprobe:v2.15.0
longhornio/backing-image-manager:v1.8.1
longhornio/longhorn-engine:v1.8.1
longhornio/longhorn-instance-manager:v1.8.1
longhornio/longhorn-manager:v1.8.1
longhornio/longhorn-share-manager:v1.8.1
longhornio/longhorn-ui:v1.8.1
longhornio/longhorn-cli:v1.8.1
longhornio/support-bundle-kit:v0.0.52
root@node1:~# cat save-images.sh
#!/bin/bash
list="longhorn-images.txt"

POSITIONAL=()
while [[ $# -gt 0 ]]; do
key="$1"
case $key in
-i|--images)
images="$2"
shift # past argument
shift # past value
;;
-l|--image-list)
list="$2"
shift # past argument
shift # past value
;;
-p|--platform)
platform="$2"
shift # past argument
shift # past value
;;
-h|--help)
help="true"
shift
;;
*)
echo "Error! invalid flag: ${key}"
help="true"
break
;;
esac
done

usage () {
echo "USAGE: $0 [--image-list longhorn-images.txt] [--images longhorn-images.tar.gz] [--platform linux/amd64]"
echo " [-l|--images-list path] text file with list of images. 1 per line."
echo " [-p|--platform linux/arch] if using images-list path, pulls the image with the specified platform"
echo " [-i|--images path] tar.gz generated by docker save. If this flag is empty, the script does not export images to a tar.gz file"
echo " [-h|--help] Usage message"
}

if [[ $help ]]; then
usage
exit 0
fi

set -e -x

for i in $(cat ${list}); do
if [ -n "$platform" ]; then
docker pull ${i} --platform $platform
else
docker pull ${i}
fi
done

if [[ $images ]]; then
docker save $(cat ${list} | tr '\n' ' ') | gzip -c > ${images}
fi

gunzip -c /root/longhorn.tar.gz | /opt/kube/bin/containerd-bin/ctr -n k8s.io images import -

4.3 Longhorn 生产优化配置

部署完成后,在 Longhorn UI 或 CRD 中调整以下设置:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
# longhorn-setting.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: longhorn-default-setting
namespace: longhorn-system
data:
default-setting.yaml: |
# 默认副本数
default-replica-count: 3
# 默认数据路径
default-data-path: /data/longhorn
# 存储超配比例(200%)
storage-over-provisioning-percentage: 200
# 最小可用空间比例(低于10%停止调度)
storage-minimal-available-percentage: 10
# 副本自动均衡
replica-auto-balance: best-effort
# 节点排水策略
node-drain-policy: block-if-contains-last-replica
# 自动删除 PVC 时删除卷
deleting-confirmation-flag: true
# 快照数量限制
snapshot-count-max: 250
# 备份目标(生产建议配置 NFS/S3)
# backup-target: nfs://192.168.1.100:/backup/longhorn
# 自动 salvage
auto-salvage: true
# 自动删除无效副本
auto-delete-when-volume-is-purged: true
# 调度策略
replica-soft-anti-affinity: false
replica-zone-soft-anti-affinity: true
# 引擎并发
concurrent-automatic-engine-upgrade-per-node-limit: 3

4.4 创建 StorageClass[helm自动创建]

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
# longhorn-storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: longhorn
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: Retain # 生产建议 Retain,防止误删
volumeBindingMode: Immediate
parameters:
numberOfReplicas: "3"
staleReplicaTimeout: "30"
fromBackup: ""
fsType: "ext4"
dataLocality: "disabled"
# 数据本地性优化(可选,同节点优先)
# dataLocality: "best-effort"
unmapMarkSnapChainRemoved: "ignored"
---
# 高性能场景 StorageClass
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: longhorn-fast
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: Retain
volumeBindingMode: Immediate
parameters:
numberOfReplicas: "2"
staleReplicaTimeout: "30"
fsType: "ext4"
dataLocality: "best-effort"
1
kubectl apply -f longhorn-storageclass.yaml

4.5 配置备份目标(NFS 示例)

1
2
3
4
5
6
# 在 NFS 服务器上
# mkdir -p /backup/longhorn && chmod 777 /backup/longhorn

# Longhorn UI 中设置:
# Settings > General > Backup Target
# nfs://192.168.1.100:/backup/longhorn

五、验证与测试

5.1 验证 Longhorn

1
2
3
4
5
6
7
8
9
# 检查 Longhorn 组件
kubectl get pods -n longhorn-system

# 检查节点磁盘状态
kubectl get nodes -o json | jq '.items[].metadata.name'

# 访问 Longhorn UI(通过 kubectl proxy 或 Ingress)
kubectl port-forward -n longhorn-system svc/longhorn-frontend 8080:80
# 浏览器访问 http://localhost:8080

5.2 创建 PVC 测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
root@node1:~# cat nginx*
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nginx-longhorn-pvc
spec:
accessModes:
- ReadWriteOnce
storageClassName: longhorn
resources:
requests:
storage: 2Gi
----------------------------
apiVersion: v1
kind: Pod
metadata:
name: nginx-longhorn
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:latest
ports:
- containerPort: 80
volumeMounts:
- name: nginx-storage
mountPath: "/usr/share/nginx/html"
volumes:
- name: nginx-storage
persistentVolumeClaim:
claimName: nginx-longhorn-pvc


1
2
3
4
5
6
7
8
9
10
11
root@node1:~# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
longhorn (default) driver.longhorn.io Delete Immediate true 2d17h
longhorn-static driver.longhorn.io Delete Immediate true 2d17h
root@node1:~# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
nginx-longhorn-pvc Bound pvc-a75b143c-077a-45b5-8d65-a85a19de77b7 2Gi RWO longhorn <unset> 2d9h
root@node1:~# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
pvc-a75b143c-077a-45b5-8d65-a85a19de77b7 2Gi RWO Delete Bound default/nginx-longhorn-pvc longhorn <unset> 2d9h


Web UI

image-20260720092645683

六、生产加固建议

维度 措施
监控 部署 Prometheus + Grafana,导入 Longhorn Dashboard(ID: 17628)
告警 配置磁盘使用率 >80% 告警,副本异常告警
备份 设置定时 Backup CRD,关键数据每日备份到 NFS/S3
日志 收集 Longhorn Manager 日志到 ELK/Loki
安全 Longhorn UI 配置 Ingress + Basic Auth/TLS
升级 制定 Longhorn 滚动升级流程,升级前做好备份
灾备 跨可用区部署 Worker,利用 Longhorn Zone Anti-Affinity
资源隔离 为 Longhorn 组件设置 requests/limits,避免资源争抢

监控示例(Prometheus 告警规则)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# longhorn-alert-rules.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: longhorn-alerts
namespace: longhorn-system
spec:
groups:
- name: longhorn.rules
rules:
- alert: LonghornNodeDiskUsageHigh
expr: longhorn_node_storage_usage_bytes / longhorn_node_storage_capacity_bytes > 0.8
for: 5m
labels:
severity: warning
annotations:
summary: "Longhorn 节点磁盘使用率超过 80%"
- alert: LonghornVolumeDegraded
expr: longhorn_volume_robustness != "healthy"
for: 5m
labels:
severity: critical
annotations:
summary: "Longhorn 卷状态异常: {{ $labels.name }}"

七、部署Headlamp

Kubernetes Dashboard 项目已经归档,Headlamp 成为新的 Kubernetes UI 方向。

Headlamp 通过允许你从单个界面处理多个集群而无需切换工具或丢失上下文,扩展了这个视图。 这使得并排管理开发、暂存和生产环境变得更容易。

In-cluster

Install Headlamp in your cluster using Helm. Run the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
helm repo add headlamp https://kubernetes-sigs.github.io/headlamp/
helm install my-headlamp headlamp/headlamp --namespace kube-system

root@node1:~# helm repo add headlamp https://kubernetes-sigs.github.io/headlamp/
"headlamp" has been added to your repositories
root@node1:~# helm install my-headlamp headlamp/headlamp --namespace kube-system
NAME: my-headlamp
LAST DEPLOYED: Mon Jul 20 07:42:14 2026
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods --namespace kube-system -l "app.kubernetes.io/name=headlamp,app.kubernetes.io/instance=my-headlamp" -o jsonpath="{.items[0].metadata.name}")
export CONTAINER_PORT=$(kubectl get pod --namespace kube-system $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl --namespace kube-system port-forward $POD_NAME 8080:$CONTAINER_PORT
2. Get the token using
kubectl create token my-headlamp --namespace kube-system

1
2
3
4

root@node1:~# kubectl create token my-headlamp --namespace kube-system
eyJhbGciOiJSUzI1NiIsImtpZCI6InBWeEFWZDZ2bzg4Ql81aTRLQmx2S2YwUnd1VVBzaGdNTWc0UVhoM21aam8ifQ.eyJhdWQiOlsiYXBpIiwiaXN0aW8tY2EiXSwiZXhwIjoxNzg0NTM3MTEzLCJpYXQiOjE3ODQ1MzM1MTMsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2YyIsImp0aSI6Ijk2YzkxNDRkLTQ2YWEtNGI3NC1hODRlLTIxMmVjOWFlYzI4OSIsImt1YmVybmV0ZXMuaW8iOnsibmFtZXNwYWNlIjoia3ViZS1zeXN0ZW0iLCJzZXJ2aWNlYWNjb3VudCI6eyJuYW1lIjoibXktaGVhZGxhbXAiLCJ1aWQiOiI3MDlmYzExMi1mNDEwLTQyNGEtODIyMS02MTY1NjY3ZDg5ZTYifX0sIm5iZiI6MTc4NDUzMzUxMywic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOm15LWhlYWRsYW1wIn0.ANQASn7-QHk4JStAxxCIXXMFCa2yDj_GMsQl4n9eUTTo4ooi-t8mHnMUkO48u1nbBl-v4AaH2HE3EyzpDOllaED5Mb5yFWF18QDJljT44xDp-XA-N3mDWVzchObZ2wOkVIixpvBK7LAbNdUiBVwi6PosNSVtuIH4tNRQEeut4Cs7zX5bNEkOcDwIvdvN5FwSCvwKvIut2ivN_CAPNOJDEu7OqWDPWsTKMJRmTEbHlBR3i8UvDvL5i782CTc0K9JTcdUf2MMZZZqqfjulRiQBnJdGeAjYurqv8fiT-fybqcpJLls-3KshKAxYl-FJDBTrN1AuPeJme7W8wvupxXXk9Q

image-20260720155153488

Share 

 Previous post: magic Next post: kubevirt安装 

© 2026 Hekang

Theme Typography by Makito

Proudly published with Hexo